I have a C application whose one of the jobs is to call an executable file. That file has performance measurement routines inserted during compilation, at the level of intermediate code. It can measure time or L1/L2/L3 cache misses. In other words, I have modified the LLVM compiler to insert a call to that function and print the result to stdout for any compiled program.
Now, like I mentioned at the beginning, I would like to execute the program (with this result returned to stdout) from a separate C application and save that result. The way I'm doing it right now is:
void executeProgram(const char* filename, char* time) {
printf("Executing selected program %s...\n", filename);
char filePath[100] = "/home/michal/thesis/Drafts/output/";
strcat(filePath, filename);
FILE *fp;
fp = popen(filePath, "r");
char str[30];
if (fp == NULL) {
printf("Failed to run command\n" );
exit(1);
}
while (fgets(str, sizeof(str) - 1, fp) != NULL) {
strcat(time, str);
}
pclose(fp);
}
where filename is the name of the compiled executable to run. The result is saved to time string.
The problem is, that the results I'm getting are pretty different and unstable compared to those that are returned by simply running the executable 'by hand' from the command line (./test16). They look like:
231425
229958
230450
228534
230033
230566
231059
232016
230733
236017
213179
90515
229775
213351
229316
231642
230875
So they're mostly around 230000 us, with some occasional drops. The same executable, run from within the other application, produces:
97097
88706
91418
97970
97972
94597
95846
95139
91070
95918
107006
89988
90882
91986
90997
88824
129136
94976
102191
94400
95215
95061
92115
96319
114091
95230
114500
95533
102294
108473
105730
Note that it is the same executable that's being called. Yet the measured time it returns is different. The program that is being measured consists of a function call to a simple nested loop, accessing array elements. Here is the code:
#include "test.h"
#include <stdio.h>
float data[1000][1000] = {0};
void test(void)
{
int i0, i1;
int N = 80;
float mean[1000];
for (i0 = 0; i0 < N; i0++)
{
mean[i0] = 0.0;
for (i1 = 0; i1 < N; i1++) {
mean[i0] += data[i0][i1];
}
mean[i0] /= 1000;
}
}
I'm suspecting that there is something wrong in the way the program is invoked in the code, maybe the process should be forked or something? Any ideas?
You didnt specify where exactly your time measuring subroutines are inserted, so all I can really offer is guesswork.
The results seem to hint to the exact opposite - running the application from shell is slower, so I wouldn't worry about the way you're starting the process from the C code. My guess would be - when you run your program from shell, it's the terminal that's slowing you down. When you're running the process from your C code, you pipe the output back to your 'starter' application, which is already waiting for input on the pipe.
As a side note, consider switching from strcat to something safer, like strncat.
Related
I have a couple of questions on this piece of code, running on a jetson nano:
#include "stdio.h"
#include "unistd.h"
#include "stdlib.h"
float gputemp = 0;
float cputemp = 0;
int count = 0;
int main() {
char* cpu;
char* gpu;
cpu = (char*)malloc(sizeof(char)*6);
gpu = (char*)malloc(sizeof(char)*6);
while (1) {
FILE* fcputemp = fopen("/sys/devices/virtual/thermal/thermal_zone1/temp", "r");
FILE* fgputemp = fopen("/sys/devices/virtual/thermal/thermal_zone2/temp","r");
if (!fcputemp || !fgputemp ) {
printf("Something went wrong\n");
exit(EXIT_FAILURE);
}
cputemp = atoi(fgets(cpu, 6, fcputemp))/1000;
gputemp = atoi(fgets(gpu, 6, fgputemp))/1000;
printf("\rCpu : %.2f, Gpu : %.2f. Elapsed time : %d", cputemp, gputemp, count);
fflush(stdout);
fclose(fcputemp);
fclose(fgputemp);
count++;
sleep(1);
}
}
Here I have to open, get the temperatures, and then close the file each loop iteration in order to get valid data (and dont segfault).
My concern here is the number of (expensive) kernel switches needed to do this.
I know that premature optimization is evil, but there is another way (or maybe the RIGHT way) to do that, opening the file only once?
And why the sensor interface (the file) cant update itself if I have it open?
P.S: Yes, I know, I didnt free cpu nor gpu variables, this is only "demo" code (just watch how i measure the time passed lol)
I'm not sure you can do this opening the files once and once only. You could try rewinding, but sysfs isn't a "real" filesystem and those aren't real files. If you rewind you might get the same data over and over, especially when using buffered calls like fopen().
The open operation is what prepares that data for reading. Since this is all managed by the kernel it should have very little overhead, and no actual disk activity. Consider that programs like top read thousands of these every second and it's no big deal.
I'm working on a C program (Ubuntu 14.04) that does basically:
Opens a 1GB file
Reads it by buffer of 1MB
Looks for some objects in the buffer
Computes the MD5 signature of each object found
My program take 10 secondes the first time to achieve this, and then only 1 seconde the next times (even if I work on a second copy of initial file).
I know that this has something to do with caching, does my program work on cached data after the first time ? or directly show cached results without doing any computation ?
int main(int argc, char** argv) {
unsigned char buffer[BUFFER_SIZE];
int i, number, count = 0;
int start, end = 0;
FILE *file;
file = fopen("/dump/ram.lime", "r");
if (file != NULL) {
while ((number = fread(buffer, 1, BUFFER_SIZE, file)) > 0) {
for (i = 0; i < number; i++) {
find_object(buffer, &start, &end);
md5_compute(&buffer[start], end - start);
}
}
} else {
printf("errno %d \n", errno);
}
printf("count = %d \n", count);
return (EXIT_SUCCESS);
}
Because the second time, most of your program code and most of the file data are already sitting in the page cache (and the kernel won't need any disk I/O to get them into RAM)
You'll likely to observe similar speedup if you run any other program (like cat or wc) on that big file which reads it sequentially before running your own code.
See also posix_fadvise(2), sync(2) & the Linux specific readahead(2) & http://www.linuxatemyram.com/ ; use the free(1) command before, during, and after running your program to measure memory. Read also about proc(5), since /proc/ contains a lot of useful pseudo-files describing the kernel state of your machine or your process.
Use also time(1), perhaps as /usr/bin/time -v, to benchmark several times your program. See also time(7) & getrusage(2) ...
I am trying to find fasta sequences of 5 ids/name as provided by user from a big fasta file (containing 80000 fasta sequences) using an external header file kseq.h as in: http://lh3lh3.users.sourceforge.net/kseq.shtml. When I run the program in a for loop, I have to open/close the big fasta file again and again (commented in the code) which makes the computation time slow. On the contrary, if I open/close only once outside the loop, the program stops if it encounters an entry which is not present in the big fasta file I.e. it reaches end of the file. Can anyone suggest how to get all the sequences without losing computational time. The code is:
#include <zlib.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include "ext_libraries/kseq.h"
KSEQ_INIT(gzFile, gzread)
int main(int argc, char *argv[])
{
char gwidd_ids[100];
kseq_t *seq;
int i=0, nFields=0, row=0, col=0;
int size=1000, flag1=0, l=0, index0=0;
printf("Opening file %s\n", argv[1]);
char **gi_ids=(char **)malloc(sizeof(char *)*size);
for(i=0;i<size;i++)
{
gi_ids[i]=(char *)malloc(sizeof(char)*50);
}
FILE *fp_inp = fopen(argv[1], "r");
while(fscanf(fp_inp, "%s", gwidd_ids) == 1)
{
printf("%s\n", gwidd_ids);
strcpy(gi_ids[index0], gwidd_ids);
index0++;
}
fclose(fp_inp);
FILE *f0 = fopen("xxx.txt", "w");
FILE *f1 = fopen("yyy.txt", "w");
FILE *f2 = fopen("zzz", "w");
FILE *instream = NULL;
instream = fopen("fasta_seq_uniprot.txt", "r");
gzFile fpf = gzdopen(fileno(instream), "r");
for(col=0;col<index0;col++)
{
flag1=0;
// FILE *instream = NULL;
// instream = fopen("fasta_seq_nr_uniprot.txt", "r");
// gzFile fpf = gzdopen(fileno(instream), "r");
kseq_t *seq = kseq_init(fpf);
while((kseq_read(seq)) >= 0 && flag1 == 0)
{
if(strcasecmp(gi_ids[col], seq->name.s) == 0)
{
fprintf(f1, ">%s\n", gi_ids[col]);
fprintf(f2, ">%s\n%s\n", seq->name.s, seq->seq.s);
flag1 = 1;
}
}
if(flag1 == 0)
{
fprintf(f0, "%s\n", gi_ids[col]);
}
kseq_destroy(seq);
// gzclose(fpf);
}
gzclose(fpf);
fclose(f0);
fclose(f1);
fclose(f2);
for(i=0;i<size;i++)
{
free(gi_ids[i]);
}
free(gi_ids);
return 0;
}
A few examples of inputfile (fasta_seq_uniprot.txt) is:
P21306
MSAWRKAGISYAAYLNVAAQAIRSSLKTELQTASVLNRSQTDAFYTQYKNGTAASEPTPITK
P38077
MLSRIVSNNATRSVMCHQAQVGILYKTNPVRTYATLKEVEMRLKSIKNIEKITKTMKIVASTRLSKAEKAKISAKKMD
-----------
-----------
The user entry file is
P37592\n
Q8IUX1\n
B3GNT2\n
Q81U58\n
P70453\n
Your problem appears a bit different than you suppose. That the program stops after trying to retrieve a sequence that is not present in the data file is a consequence of the fact that it never rewinds the input. Therefore, even for a query list containing only sequences that are present in the data file, if the requested sequence IDs are not in the same relative order as the data file then the program will fail to find some of the sequences (it will pass them by when looking for an earlier-listed sequence, never to return).
Furthermore, I think it likely that the time savings you observe comes from making only a single pass through the file, instead of a (partial) pass for each requested sequence, not so much from opening it only once. Opening and closing a file is a bit expensive, but nowhere near as expensive as reading tens or hundreds of kilobytes from it.
To answer your question directly, I think you need to take these steps:
Move the kseq_init(seq) call to just before the loop.
Move the kseq_destroy(seq) call to just after the loop.
Put in a call to kseq_rewind(seq) as the last statement in the loop.
That should make your program right again, but it is likely to kill pretty much all your time savings, because you will return to scanning the file from the beginning for each requested sequence.
The library you are using appears to support only sequential access. Therefore, the most efficient way to do the job both right and fast would be to invert the logic: read sequences one at a time in an outer loop, testing each one as you go to see whether it matches any of the requested ones.
Supposing that the list of requested sequences will contain only a few entries, like your example, you probably don't need to do any better testing for matches than just using an inner loop to test each requested sequence id vs. the then-current sequence. If the query lists may be a lot longer, though, then you could consider putting them in a hash table or sorting them into the same order as the data file to make it possible to test more efficiently for matches.
Hello everyone I'm new to C but I've recently been getting a weird segfault error with my fopen.
FILE* thefile = fopen(argv[1],"r");
The problem I've been having is that this code works on other smaller text files, but when I try with a file around 400MB it will give a sefault error. I've even tried hardcoding the filename but that doesn't work either. Could there be a problem in the rest of the code causing the segfault on this line?(doubt it but would like to know if its possible. It's just really odd that no errors come up for a small text file, but a large text file does get errors.
Thanks!
EDIT* didn't want to bog this down with too much but heres my code
int main(int argc, char *argv[])
{
if(argc != 3)
{
printf("[ERROR] Invalid number of arguments. Please pass 2 arguments, input_bound_file (column 1:probe, columne 2,...: samples) and desired_output_file_name");
exit(2);
}
int i,j;
rankAvg= g_hash_table_new(g_direct_hash, g_direct_equal);
rankCnt= g_hash_table_new(g_direct_hash, g_direct_equal);
table = g_hash_table_new_full (g_direct_hash, g_direct_equal, NULL, g_free);
getCounts(argv[1]);
printf("NC=: %i nR =: %i",nC,nR);
double srcMat[nR][nC];
int rankMat[nR][nC];
double normMat[nR][nC];
int sorts[nR][nC];
char line[100];
FILE* thefile = fopen(argv[1],"r");
printf("%s\n", strerror(errno));
FILE* output = fopen(argv[2],"w");
char* rownames[100];
i=0;j = 1;
int processedProbeNumber = 0;
int previousStamp = 0;
fgets(line,sizeof(line),thefile); //read file
while(fgets(line,sizeof(line),thefile) != NULL)
{
cleanSpace(line); //creates only one space between entries
char dest[100];
int len = strlen(line);
for(i = 0; i < len; i++)
{
if(line[i] == ' ') //read in rownames
{
rownames[j] = strncpy(dest, line, i);
dest[i] = '\0';
break;
}
}
char* token = strtok(line, " ");
token = strtok(NULL, " ");
i=1;
while(token!=NULL) //put words into array
{
rankMat[j][i]= abs(atof(token));
srcMat[j][i] = abs(atof(token));
token = strtok(NULL, " ");
i++;
}
// set the first column as a row id
j++;
processedProbeNumber++;
if( (processedProbeNumber-previousStamp) >= 10000)
{
previousStamp = processedProbeNumber;
printf("\tnumber of loaded lines = %i",processedProbeNumber);
}
}
printf("\ttotal number of loaded lines = %i \n",processedProbeNumber);
fclose(thefile);
How do you know that fopen is seg faulting? If you're simply sprinkling printf in the code, there's a chance the standard output isn't sent to the console before the error occurs. Obviously, if you're using a debugger you will know exactly where the segfault occured.
Looking at your code, nR and nC aren't defined so I don't know how big rankMat and srcMat are, but two thoughts crossed my mind while looking at your code:
You don't check i and j to ensure that they don't exceed nR and nC
If nR and nC are sufficiently large, that may mean you're using a very large amount of memory on the stack (srcMat, rankMat, normMat, and sorts are all huge). I don't know what environemnt you're running in, but some systems my not be able to handle huge stacks (Linux, Windows, etc. should be fine, but I do a lot of embedded work). I normally allocate very large structures in the heap (using malloc).
Generally files 2GB (2**31) or larger are the ones you can expect to get this on. This is because you are starting to run out of space in a 32-bit integer for things like file indices, and one bit is typically taken up for directions in relative offsets.
Supposedly on Linux you can get around this issue by using the following macro defintion:
#define _FILE_OFFSET_BITS 64
Some systems also provide a separate API call for large file opens (eg: fopen64() in MKS).
400Mb should not be considered a "large file" nowadays. I would reserve this for files larger than, say, 2Gb.
Also, just opening a file is very unlikely to give a segfault. WOuld you show us the code that access the file? I suspect some other factor is at play here.
UPDATE
I still can't tell exactly what's happening here. There are strange things that could be legitimate: you discard the first line and also the first token of each line.
You also assign to all the rownames[j] (except the first one) the address of dest which is a variable that has a block scope and whose associated memory is most likely to be reused outside that block. I hope you don't rely on rownames[j] to be any meaningful (but then why you have them?) and you never try to access them.
C99 allows you to mix variable declarations with actual instructions but I would suggest a little bit of cleaning to make the code clearer (also a better indentation would help).
From the symptoms I would look for some memory corruption somewhere. On small files (and hence less tokens) it may go unnoticed, but with larger files (and many more token) it fires a segfault.
I'm making a load balancer (a very simple one). It looks at how long the user has been idle, and the load on the system to determine if a process can run, and it goes through processes in a round-robin fashion.
All of the data needed to control the processes are stored in a text file.
The file might look like this:
PID=4390 IDLE=0.000000 BUSY=2.000000 USER=2.000000
PID=4397 IDLE=3.000000 BUSY=1.500000 USER=4.000000
PID=4405 IDLE=0.000000 BUSY=2.000000 USER=2.000000
PID=4412 IDLE=0.000000 BUSY=2.000000 USER=2.000000
PID=4420 IDLE=3.000000 BUSY=1.500000 USER=4.000000
This is a university assignment, however parsing the text file isn't supposed to be a big part of it, which means I can use whatever way is the quickest for me to implement.
Entries in this file will be added and removed as processes finish or are added under control.
Any ideas on how to parse this?
Thanks.
Here is a code that will parse your file, and also account for the fact that your file might be unavailable (that is, fopen might fail), or being written while you read it (that is, fscanf might fail). Note that infinite loop, which you might not want to use (that's more pseudo-code than actual code to be copy-pasted in your project, I didn't try to run it). Note also that it might be quite slow given the duration of the sleep there: you might want to use a more advanced approach, that's more sort of a hack.
int pid;
float idle, busy, user;
FILE* fid;
fpos_t pos;
int pos_init = 0;
while (1)
{
// try to open the file
if ((fid = fopen("myfile.txt","rw+")) == NULL)
{
sleep(1); // sleep for a little while, and try again
continue;
}
// reset position in file (if initialized)
if (pos_init)
fsetpos (pFile,&pos);
// read as many line as you can
while (!feof(fid))
{
if (fscanf(fid,"PID=%d IDLE=%f BUSY=%f USER=%f",&pid, &idle, &busy, &user))
{
// found a line that does match this pattern: try again later, the file might be currently written
break;
}
// add here your code processing data
fgetpos (pFile,&pos); // remember current position
pos_init = 1; // position has been initialized
}
fclose(fid);
}
As far as just parsing is concerned, something like this in a loop:
int pid;
float idle, busy, user;
if(fscanf(inputStream, "PID=%d IDLE=%f BUSY=%f USER=%f", %pid, &idle, &busy, &user)!=4)
{
/* handle the error */
}
But as #Blrfl pointed out, the big problem is to avoid mixups when your application is reading the file and the others are writing to it. To solve this problem you should use a lock or something like that; see e.g. the flock syscall.
Use fscanf in a loop. Here's a GNU C tutorial on using fscanf.
/* fscanf example */
#include <stdio.h>
typedef struct lbCfgData {
int pid;
double idle;
double busy;
double user;
} lbCfgData_t ;
int main ()
{
// PID=4390 IDLE=0.000000 BUSY=2.000000 USER=2.000000
lbCfgData_t cfgData[128];
FILE *f;
f = fopen ("myfile.txt","rw+");
for ( int i = 0;
i != 128 // Make sure we don't overflow the array
&& fscanf(f, "PID=%u IDLE=%f BUSY=%f USER=%f", &cfgData[i].pid,
&cfgData[i].idle, &cfgData[i].busy, cfgData[i].user ) != EOF;
i++
);
fclose (f);
return 0;
}