I'm writing several benchmark programs in C to achieve the following tasks:
The speed with which one can read from a network disk. Print the seconds needed to read 8192 bytes.
The speed with which one can read from the local directory /tmp on your local machine. Print the seconds needed to read 8192 bytes.
The speed with which one can read from the disk page cache. Print the seconds needed to read 8192 bytes.
The speed with which one can write to a network disk. Print the seconds needed to write 8192 bytes.
The speed with which one can write to the local directory /tmp on your local machine. Print the seconds needed to write 8192 bytes.
The goal here is to measure just the time that doing the file read or write takes (using read and write to avoid any buffering time from fread)
My general approach for 1 and 2 is to create a file of 8192 bytes and write that to disk (whether it be the local directory or the network disk) and then call sleep(10) to wait for the page cache to flush so that I'm measuring the time of the actual I/O, not the cache I/O. Then I measure the time it takes to do an empty for loop several thousand times, then the time it takes to read 8192 bytes in then subtract the two, divided over the average of all iterations. My code for that looks like:
struct timespec emptyLoop1, emptyLoop2;
clock_gettime(CLOCK_REALTIME, &emptyLoop1);
for(i = 0, j = 0; i < ITERATIONS; i++) {
j+=i*i;
}
clock_gettime(CLOCK_REALTIME, &emptyLoop2);
char readbuf[NUM_BYTES];
struct timespec beforeRead, afterRead;
clock_gettime(CLOCK_REALTIME, &beforeRead);
for(i = 0, j = 0; i < ITERATIONS; i++){
j+=i*i;
read(fd, readbuf, NUM_BYTES);
}
Is that sufficient for accurately measuring the times of reading from those locations?
Next, I'm confused as to how to read from the page cache. Where does that exist on disk and how do I access it? Finally, there's some tricks for 4 and 5 which are apparently much harder than they seem but I'm not sure what I'm missing.
Following is my file reading function, enabling choice of using or not using the memory based cache. If writing files first, similar open statements are needed. Note that direct I/O cannot be used over a LAN and caching can be unpredictable. More details and access to source codes and execution files can be found in http://www.roylongbottom.org.uk/linux_disk_usb_lan_benchmarks.htm.
int readFile(int use, int dsize)
{
int p;
if (useCache)
{
handle = open(testFile, O_RDONLY);
}
else
{
handle = open(testFile, O_RDONLY | O_DIRECT);
}
if (handle == -1)
{
printf (" Cannot open data file for reading\n\n");
fprintf (outfile, " Cannot open data file for reading\n\n");
fclose(outfile);
printf(" Press Enter\n");
g = getchar();
return 0;
}
for (p=0; p<use; p++)
{
if (read(handle, dataIn, dsize) == -1)
{
printf (" Error reading file\n\n");
fprintf (outfile, " Error reading file\n\n");
fclose(outfile);
close(handle);
printf(" Press Enter\n");
g = getchar();
return 0;
}
}
close(handle);
return 1;
}
Related
I have a Zynq SoC which runs a Linux system and would like to time the speed at which I can write to its SD card in C.
I have tried out clock() and clock_gettime() where for the latter, I have done the following:
#define BILLION 1000000000.0
#define BUFFER_SIZE 2097152
...
for (int i=0; i<100; i++) {
memcpy(buf, channel->dma + offset, BUFFER_SIZE);
/* Open file for output */
FILE *f = fopen(path, "w");
/* Start timer */
clock_gettime(CLOCK_MONOTONIC, &start);
/* Write data from cpu ram to sd card*/
fwrite(buf, 1, BUFFER_SIZE, f);
fclose(f);
/* Stop timer and report result */
clock_gettime(CLOCK_MONOTONIC, &end);
elapsed_time = (end.tv_sec - start.tv_sec) + (end.tv_nsec - start.tv_nsec) / BILLION;
printf("%lf seconds\n", elapsed_time);
}
The times that I'm getting are around 0.019 seconds for 2097152 bytes which I believe is wrong because the speed of writing to the SD card is quoted as 10 MB/s and can't possibly be the 110 MB/s that I appear to be getting. When I, instead put the timing code outside the loop and, basically, time how long it takes to write the whole 100*BUFFER_SIZE, I get a more reasonable 18 MB/s.
My question is, is it incorrect to try to time a single fwrite operation? Does clock_gettime() not give me the adequate accuracy/precision for that? I would like to have the most accurate value possible for how quickly I can write to the disk because it is a very important parameter in the design of the system and so this discrepancy I'm getting is rather disconcerting.
The Linux Kernel often caches read write access to disk storage. That is, it returns from the write call and does the actual writing in the background. It does that transparently, so that if you would read the same file, just after writing, you would get the results you wrote, even if they are not yet transferred and written completely to disk.
To force a complete write you can call the fsync function. It blocks until all file IO for a specific file has completed. In your case a call to
#include <unistd.h>
...
fwrite(...);
fsync(fileno(f));
should suffice.
Edit
as #pmg mentioned in the comments, there is also be buffering at stream level. Although it probably is not that large of a buffer, you can force the stream buffer to be written with a call to fflush() before the fsync.
I'm working on a C program (Ubuntu 14.04) that does basically:
Opens a 1GB file
Reads it by buffer of 1MB
Looks for some objects in the buffer
Computes the MD5 signature of each object found
My program take 10 secondes the first time to achieve this, and then only 1 seconde the next times (even if I work on a second copy of initial file).
I know that this has something to do with caching, does my program work on cached data after the first time ? or directly show cached results without doing any computation ?
int main(int argc, char** argv) {
unsigned char buffer[BUFFER_SIZE];
int i, number, count = 0;
int start, end = 0;
FILE *file;
file = fopen("/dump/ram.lime", "r");
if (file != NULL) {
while ((number = fread(buffer, 1, BUFFER_SIZE, file)) > 0) {
for (i = 0; i < number; i++) {
find_object(buffer, &start, &end);
md5_compute(&buffer[start], end - start);
}
}
} else {
printf("errno %d \n", errno);
}
printf("count = %d \n", count);
return (EXIT_SUCCESS);
}
Because the second time, most of your program code and most of the file data are already sitting in the page cache (and the kernel won't need any disk I/O to get them into RAM)
You'll likely to observe similar speedup if you run any other program (like cat or wc) on that big file which reads it sequentially before running your own code.
See also posix_fadvise(2), sync(2) & the Linux specific readahead(2) & http://www.linuxatemyram.com/ ; use the free(1) command before, during, and after running your program to measure memory. Read also about proc(5), since /proc/ contains a lot of useful pseudo-files describing the kernel state of your machine or your process.
Use also time(1), perhaps as /usr/bin/time -v, to benchmark several times your program. See also time(7) & getrusage(2) ...
I am trying to read a file, read in how many bytes it contains and then round it up to its nearest GB and then double the file size. However, is there is way to read the file and then some do all this stuff back into the same file?
Here is what I have so far, but it creates a new file with the new contents but I'm not sure if my logic is correct
Also, do you create a constant like BYTE with #define?
So far as a test case I just used byte as an int and make it equal to 50
#include<stdio.h>
#include<stdlib.h>
#include<string.h>
#include <time.h>
// #define BYTE 50
int main()
{
FILE *fp1, *fp2;
int ch1;
clock_t elapsed;
char fname1[40], fname2[40];
char a;
printf("Enter name of the file:");
fgets(fname1, 40, stdin);
while ( fname1[strlen(fname1) - 1] == '\n')
{
fname1[strlen(fname1) -1] = '\0';
}
fp1 = fopen(fname1, "r");
if ( fp1 == NULL )
{
printf("Cannot open %s for reading\n", fname1 );
exit(1);
}
printf("This program will round up the current file into highest GB, and then double it");
elapsed = clock(); // get starting time
ch1 = getc(fp1); // read a value from each file
int num = 50;
int bytes = 0;
while(1) // keep reading while values are equal or not equal; only end if it reaches the end of one of the files
{
ch1 = getc(fp1);
bytes++;
if (ch1 == EOF) // if either file reaches the end, then its over!
{
break; // if either value is EOF
}
}
// 1,000,000,000 bytes in a GB
int nextInt = bytes%num;
// example: 2.0GB 2,000,000,000 - 1.3GB 1,300,000,000 = 7,000,000,000 OR same thing as 2,000,000,000%1,300,000,000 = 700,000,000
int counter = 0;
printf("Enter name of the file you would like to create:");
fgets(fname2, 40, stdin);
while ( fname2[strlen(fname2) - 1] == '\n')
{
fname2[strlen(fname2) -1] = '\0';
}
fp2 = fopen(fname2, "w");
if ( fp1 == NULL )
{
printf("Cannot open %s for reading\n", fname2);
exit(1);
}
if(fp2 == NULL)
{
puts("Not able to open this file");
fclose(fp1);
exit(1);
}
while(counter != nextInt)
{
a = fgetc(fp1);
fputc(a, fp2);
counter++;
}
fclose(fp1); // close files
fclose(fp2);
printf("Total number of bytes in the file %u: ", bytes);
printf("Round up the next GB %d: ", nextInt);
elapsed = clock() - elapsed; // elapsed time
printf("That took %.4f seconds\n", (float)elapsed/CLOCKS_PER_SEC);
return 0;
}
You increment bytes before you check for EOF, so you have an off-by-one error.
However, reading a file byte by byte is a slow way of finding its size. Using standard C, you may be able to use ftell() — if you're on a 64-bit Unix-like machine. Otherwise, you're working too close to the values that will fit in 32-bit values. Using a plain int for bytes is going to run into trouble.
Alternatively, and better, you stat() or fstat() to get the exact size directly.
When it comes to doubling the size of the file, you could simply seek to the new end position and write a byte at that position. However, that does not allocate all the disk space (on a Unix machine); it will be a sparse file.
On rewrite, you need to know how your system will handle two open file streams on a single file. On Unix-like systems, you can open the original file once for reading and once for writing in append mode. You could then read large chunks (64 KiB, 256 KiB?) of data at a time from the read file descriptor and write that to the write descriptor. However, you need to keep track of how much data to write because the read won't encounter EOF.
Your code is going to write a lot of 0xFF bytes to the tail of the file on most systems (where EOF is recorded as -1).
Note that there are Gibibytes GiB (230 = 1,073,741,824 bytes) and Gigabytes GB (officially 109 = 1,000,000,000 bytes, but not infrequently used to mean GiB). See Wikipedia on Binary prefix, etc.
You're working way too hard. I'll assume your OS is Windows or Linux.
On Windows, _stat will get the exact length of a file. In Linux it's stat. Both will do this from file system information, so it's almost instantaneous.
On Windows, _chsize will extend the file to any number of bytes. On Linux it's ftruncate. The OS will be writing zeros to the extension, so it will be a fast write indeed.
In all cases it's simple to find the documentation by searching.
The code will be straight-line (no loops), about 10 lines.
Rounding up to the next GB is simply done with
#define GIGA ((size_t)1 << 30)
size_t new_size = (old_size + GIGA - 1) & ~(GIGA - 1);
I have subtitles /srt files with data being generated by the second. I am trying to extract the data from them and save them to text files
in a while(1) loop, its a simple text read and write operation.
But i have found out using fgets to traverse through the file is causing this simple operation to have very high cpu usage ( top command ref). more the number of fgets higher the
cpu usage.
The code bit:
int convert_to_txt(FILE *f1, FILE *f2) {
static int str_len = 0;
char cap_nxt_tmp[128];
while (fgets(cap_nxt_tmp, 128, f1) != NULL) {
fprintf(f2, "%s\n", cap_nxt_tmp);
}
return 0;
}
int main(int argc, char* argv[]) {
FILE *inputFilePtr;
FILE *outputFilePtr;
char inputPath[1024] = "storage/sample.srt";
char outputPath[1024] = "storage/sample.txt";
while (1) {
outputFilePtr = fopen(outputPath, "w");
inputFilePtr = fopen(inputPath, "r");
if (inputFilePtr == NULL || outputFilePtr == NULL) {
perror("Error");
}
convert_to_txt(inputFilePtr, outputFilePtr);
fclose(inputFilePtr);
fclose(outputFilePtr);
// theres a break on an end condition lets say least run time is an hour.
}
return 0;
}
i cant understand whats wrong in the way i am using fgets to read the srt/captions file ( 1-2 mb per file), a simple text file i/o/read operation should not consume too much cpu usage.
this basic file i/o in a while(1) of a language like python shows only a 2-3% cpu usage.
but this in c is showing about 30% cpu usage.
How can i reduce the cpu usage without using sleep . is there any less cpu alternative to fgets?
I don't see anything wrong or strange here.
Your main while(1) loop never stops, which means you're converting your files over and over again. Files get cached by the OS, so you're not really accessing your physical drives. The program therefore spends almost all the CPU time doing strlen(cap_nxt_tmp), and you observe very high CPU load.
Currently I'm working on a small program that reads large files and sorts them. After some benchmarking I stumbled upon a weird performance issue. When the input file got to large the writing of the output file took longer than the actual sorting. So I went deeper into the code and finally realized that the fputs-function might be the problem. So I wrote this little benchmarking programm.
#include "stdio.h"
#include "ctime"
int main()
{
int i;
const int linecount = 50000000;
//Test Line with 184 byte
const char* dummyline = "THIS IS A LONG TEST LINE JUST TO SHOW THAT THE WRITER IS GUILTY OF GETTING SLOW AFTER A CERTAIN AMOUNT OF DATA THAT HAS BEEN WRITTEN. hkgjhkdsfjhgk jhksjdhfkjh skdjfhk jshdkfjhksjdhf\r\n";
clock_t start = clock();
clock_t last = start;
FILE* fp1 = fopen("D:\\largeTestFile.txt", "w");
for(i=0; i<linecount; i++){
fputs(dummyline, fp1);
if(i%100000==0){
printf("%i Lines written.\r", i);
if(i%1000000 == 0){
clock_t ms = clock()-last;
printf("Writting of %i Lines took %i ms\n", i, ms);
last = clock();
}
}
}
printf("%i Lines written.\n", i);
fclose(fp1);
clock_t ms = clock()-start;
printf("Writting of %i Lines took %i ms\n", i, ms);
}
When you execute the programm, you can see a clear drop of performance after about 14 to 15 mio lines which is about 2.5GB of data. The writing takes about 3 times as long as before. The threshold of 2GB indicate a 64bit issue, but I haven't found anything about that in the web. I also tested if there is a difference between binary and character-mode (e.g. "wb" and "w"), but there is none. I also tried to preallocate the filesize (to avoid file fragmentation) by seeking to the expected end and writing a zerobyte, but that had also little to no effect.
I'm running a Windows 7 64bit machine but I've tested it on a Windows Server 2008 64bit R1 machine as well. Currently I'm testing on a NTFS filesystem with more than 200GB of free space. My system has 16GB of RAM so that shouldn't be a problem either. The testprogram only uses about 700KB. The page faults, which I suspected earlier, are also very low (~400 page faults during whole runtime).
I know that for such large data the fwrite()-function would suite the task better, but at the moment I'm interested if there is another workaround and why this is happening. Any help would be highly appreciated.
The main reason for all this is a Windows disk cache. Then your program eats all RAM for it, then swapping begins, and thus, slowdowns. To fight these you need to:
1) Open file in commit mode using c flag:
FILE* fp1 = fopen("D:\\largeTestFile.txt", "wc");
2) Periodically write buffer to disk using flush function:
if(i%1000000 == 0)
{
// write content to disk
fflush(fp1);
clock_t ms = clock()-last;
printf("Writting of %i Lines took %i ms\n", i, ms);
last = clock();
}
This way you will use reasonable amount of disk cache. Speed will be basically limited by the speed of your hard drive.