I am currently working on a project that involves transferring a very large file (about 6GB) from one Linux Server to another. The servers run on Debian Squeeze.
In order to achieve my main goal, I initially send the file's name and size to the destination machine, and I create an empty file for storing the data blocks that I progressively receive from the source machine.
My problem is that the creation of a 6GB file takes too long in my server. To make it more clear I use the following C routine in order to create the new file:
void create_file(char* f_name, long long f_size) {
char* bs, *of, *s_f_size, *count;
if((pid = fork()) < 0) {
perror("fork() failed.");
return;
}
if(pid == 0) {
//Call execl
of = (char*) malloc(sizeof(char)*(strlen("of=") + strlen(f_name) + 1));
s_f_size = (char*) malloc(sizeof(char)*32);
sprintf(s_f_size, "%lld", file_size);
count = (char*) malloc(sizeof(char)*(strlen("count=") + strlen(s_f_size) + 1));
strcpy(of, "of=");
strcat(of, f_name);
strcpy(count, "count=");
strcat(count, s_f_size);
ret = execl("/bin/dd", "dd", "if=/dev/zero", of, "bs=1", count, (char*) 0);
if(ret < 0) {
perror("execl() failed");
free(s_f_size);
free(of);
free(count);
return;
}else {
free(s_f_size);
free(of);
free(count);
return;
}
}else {
status = 0;
wpid = wait(&status);
}
}
I used the Linux dd command because I thought that it would be the quickest way to create an empty 6GB file. However, it takes about 15 minutes to complete. Is there a way to create the empty file faster? What am I doing wrong?
Thank you for your time.
Sincerely,
Nick
In addition to what Joachim Pileborg suggested, you can also use posix_fallocate() to pre-allocate space for your file.
First creat the file, then lseek to the wanted end, and write a dummy byte. Very quick way to create an arbitrary large but sparse file.
If you don't want the file to be sparse, then find out the block size of the drive (can be found out using stat on most POSIX platforms). Create a buffer of that size, and write it to the file until the wanted size.
If the stat structure doesn't have the st_blksize member, then most filesystems have a blocksize of 4 or 8 kB. You can probably make this buffer larger, but not too large. Experiment and benchmark!
If you're using kernel v2.6.31+ and if filesystem supports it, consider using fallocate:
fallocate -l 6GB hugefile
It preallocates blocks to a file.
Creating large files takes long because there's a lot on the file system the OS has to do. Only in case of sparse files this can be skipped (see Joachim Pileborg's answer for that). A sparse file is a file containing "holes" (large chunks of zero bytes). Such a file does not use as much space as it is large. Creating such a file beforehand will produce the file with the correct size very fast.
In case you want to reserve the disk space to avoid running out of it before the transmission is complete, a sparse file won't do. You will have to write at least one byte into each block then to avoid the holes of a sparse file. I'm not sure this will be faster than simply dump zeros into the file until it has the desired size, as you already do.
i remember, i've used open system call to create a empty file. Then dump data to file.
In case of partial data write, keep seek the position and dump from there. If file exists use the file to overwrite the data into it.
With respect to performance this approach was quite good.
Related
I'm trying to process some rather big binary files (up to 20gb per file). To do so, I want to read them in chunks, process and write to the new file.
To read the file, I use:
fp = fopen(filename,"rb"); //open binary file to read
fseek(fp, start_of_chunk, SEEK_SET); //set cursor to current chunk
fread(data,size_of_chunk,1,fp); //read one chunk of data
fclose(fp);
I then allocate some arrays using calloc and process the data.
To write it back, I use:
fout = fopen(file_out,"ab"); //open output binary file in append mode
fwrite(processed_data,size_of_processed_data,1,fout);
fclose(fout);
At the end, I use free() on each of the allocated variables (biggest one is the read data chunk (~400 MB), smallest one is about 1kB) and start all over with reading the next chunk.
The output works fine but the process keeps up eating memory like there's no tomorrow. An 800 MB test file eats up to 6 GB of RAM, rising steadily. Reading the whole file at once uses pretty much the same amount of memory as reading the chunks. Using free() between the loops releases not even 5% of the used memory even though those variables contain 99% of the data.
As I am pretty new to C programming, is there anything I might be missing? I read in another thread that the OS (Windows 10 x64 in my case, same problem on Windows 7 x64) might be too slow to release memory. Reading bigger chunks did not help in my case. All the file handles are closed after reading/writing and all allocated arrays are freed at the end.
Edit: I am allocating multiple times within the loop. I'm using
free(data);
data = (uint16_t*)calloc(number_of_elements,sizeof(uint16_t));
before each call to read data. To process a certain part of the chunk, I'm using
data_part = (uint16_t*)calloc(number_of_elements,sizeof(uint16_t));
memmove(data_part,pointer_to_part_of_chunk, size_of_data_part);
Edit 2: I changed some stuff, thanks to the comments. Still, memory load keeps rising steadily (tho slower than before). Here's the code once again:
//initiate variables
//allocate arrays
data = (uint16_t*)calloc(number_of_elements,sizeof(uint16_t));
array1 = (uint16_t*)calloc(number_of_elements1,sizeof(uint16_t));
array2 = (uint16_t*)calloc(number_of_elements2,sizeof(uint16_t)); //and so on
//some precalculations
//start of the loop
while (not_end_of_the_file){
fp = fopen(filename,"rb"); //open binary file to read
fseek(fp, start_of_chunk, SEEK_SET); //set cursor to current chunk
fread(data,size_of_chunk,1,fp); //read one chunk of data
fclose(fp);
//calculate stuff
start_of_chunk = start_of_chunk + increment;
for (i=0;i<I;i++){
memmove(array1,pointer_to_part_of_data, size_of_array1);
if (statement){
subfunction1(array1);
}
else{
subfunction2(array1);
}; //nothing more than some for loops, if statements and arithmetic operations
//NO further allocations here, all buffers will be reused
};
//write result
fout = fopen(file_out,"ab"); //open output binary file in append mode
fwrite(processed_data,size_of_processed_data,1,fout);
fclose(fout);
};
There's some allocating happening in the subfunctions but that memory should be freed after the subfunction is done, shouldn't it?
free(data);
data = (uint16_t*)calloc(number_of_elements,sizeof(uint16_t));
Are you sure the value of data in the free line is the value returned from the previous call to calloc? If it's NULL when you call free, that is a no-op.
The idea behind this program is to simply access the ram and download the data from it to a txt file.
Later Ill convert the txt file to jpeg and hopefully it will be readable .
However when I try and read from the RAM using NEW[] it takes waaaaaay to long to actually copy all the values into the file?
Isnt it suppose to be really fast? I mean I save pictures everyday and it doesn't even take a second?
Is there some other method I can use to dump memory to a file?
#include <stdio.h>
#include <stdlib.h>
#include <hw/pci.h>
#include <hw/inout.h>
#include <sys/mman.h>
main()
{
FILE *fp;
fp = fopen ("test.txt","w+d");
int NumberOfPciCards = 3;
struct pci_dev_info info[NumberOfPciCards];
void *PciDeviceHandler1,*PciDeviceHandler2,*PciDeviceHandler3;
uint32_t *Buffer;
int *BusNumb; //int Buffer;
uint32_t counter =0;
int i;
int r;
int y;
volatile uint32_t *NEW,*NEW2;
uintptr_t iobase;
volatile uint32_t *regbase;
NEW = (uint32_t *)malloc(sizeof(uint32_t));
NEW2 = (uint32_t *)malloc(sizeof(uint32_t));
Buffer = (uint32_t *)malloc(sizeof(uint32_t));
BusNumb = (int*)malloc(sizeof(int));
printf ("\n 1");
for (r=0;r<NumberOfPciCards;r++)
{
memset(&info[r], 0, sizeof(info[r]));
}
printf ("\n 2");
//Here the attach takes place.
for (r=0;r<NumberOfPciCards;r++)
{
(pci_attach(r) < 0) ? FuncPrint(1,r) : FuncPrint(0,r);
}
printf ("\n 3");
info[0].VendorId = 0x8086; //Wont be using this one
info[0].DeviceId = 0x3582; //Or this one
info[1].VendorId = 0x10B5; //WIll only be using this one PLX 9054 chip
info[1].DeviceId = 0x9054; //Also PLX 9054
info[2].VendorId = 0x8086; //Not used
info[2].DeviceId = 0x24cb; //Not used
printf ("\n 4");
//I attached the device and give it a handler and set some setting.
if ((PciDeviceHandler1 = pci_attach_device(0,PCI_SHARE|PCI_INIT_ALL, 0, &info[1])) == 0)
{
perror("pci_attach_device fail");
exit(EXIT_FAILURE);
}
for (i = 0; i < 6; i++)
//This just prints out some details of the card.
{
if (info[1].BaseAddressSize[i] > 0)
printf("Aperture %d: "
"Base 0x%llx Length %d bytes Type %s\n", i,
PCI_IS_MEM(info[1].CpuBaseAddress[i]) ? PCI_MEM_ADDR(info[1].CpuBaseAddress[i]) : PCI_IO_ADDR(info[1].CpuBaseAddress[i]),
info[1].BaseAddressSize[i],PCI_IS_MEM(info[1].CpuBaseAddress[i]) ? "MEM" : "IO");
}
printf("\nEnd of Device random info dump---\n");
printf("\nNEWs Address : %d\n",*(int*)NEW);
//Not sure if this is a legitimate way of memory allocation but I cant see to read the ram any other way.
NEW = mmap_device_memory(NULL, info[1].BaseAddressSize[3],PROT_READ|PROT_WRITE|PROT_NOCACHE, 0,info[1].CpuBaseAddress[3]);
//Here is where things are starting to get messy and REALLY long to just run through all the ram and dump it.
//Is there some other way I can dump the data in the ram into a file?
while (counter!=info[1].BaseAddressSize[3])
{
fprintf(fp, "%x",NEW[counter]);
counter++;
}
fclose(fp);
printf("0x%x",*Buffer);
}
A few issues that I can see:
You are writing blocks of 4 bytes - that's quite inefficient. The stream buffering in the C library may help with that to a degree, but using larger blocks would still be more efficient.
Even worse, you are writing out the memory dump in hexadecimal notation, rather than the bytes themselves. That conversion is very CPU-intensive, not to mention that the size of the output is essentially doubled. You would be better off writing raw binary data using e.g. fwrite().
Depending on the specifics of your system (is this on QNX?), reading from I/O-mapped memory may be slower than reading directly from physical memory, especially if your PCI device has to act as a relay. What exactly is it that you are doing?
In any case I would suggest using a profiler to actually find out where your program is spending most of its time. Even a rudimentary system monitor would allow you to determine if your program is CPU-bound or I/O-bound.
As it is, "waaaaaay to long" is hardly a valid measurement. How much data is being copied? How long does it take? Where is the output file located?
P.S.: I also have some concerns w.r.t. what you are trying to do, but that is slightly off-topic for this question...
For fastest speed: write the data in binary form and use the open() / write() / close() API-s. Since your data is already available in a contiguous block of (virtual) memory it is a waste to copy it to a temporary buffer (used by the fwrite(), fprintf(), etc. API-s).
The code using write() will be similar to:
int fd = open("filename.bin", O_RDWR|O_CREAT, S_IRWXU);
write(fd, (void*)NEW, 4*info[1].BaseAddressSize[3]);
close(fd);
You will need to add error handling and make sure that the buffer size is specified correctly.
To reiterate, you get the speed-up from:
avoiding the conversion from binary to ASCII (as pointed out by others above)
avoiding many calls to libc
reducing the number of system-calls (from inside libc)
eliminating the overhead of copying data to a temporary buffer inside the fwrite()/fprintf() and related functions (buffering would be useful if your data arrived in small chunks, including the case of converting to ASCII in 4 byte units)
I intentionally ignore commenting on other parts of your code as it is apparently not intended to be production quality yet and your question is focused on how to speed up writing data to a file.
I was wondering if there was any significant performance increase in using sys/stat.h versus fseek() and and ftell()?
Choosing between fstat() and the fseek()/ftell() combination, there isn't going to be much difference. The single function call should be slightly quicker than the double function call, but the difference won't be great.
Choosing between stat() and the combination isn't a very fair comparison. For the combination calls, the hard work was done when the file was opened, so the inode information is readily available. The stat() call has to parse the file path and then report what it finds. It should almost always be slower - unless you recently opened the file anyway so the kernel has most of the information cached. Even so, the pathname lookup required by stat() is likely to make it slower than the combination.
If you're not sure, try it!
I just coded this test. I generated 10,000 files of 2KB each, and iterated over all of them, asking for their file size.
Results on my machine by measuring with the "time" command and doing an average of 10 runs:
fseek/fclose version: 0.22 secs
stat version: 0.06 secs
So, the winner (at least on my machine): stat!
Here's the test code:
#include <stdio.h>
#include <sys/stat.h>
#if 0
size_t getFileSize(const char * filename)
{
struct stat st;
stat(filename, &st);
return st.st_size;
}
#else
size_t getFileSize(const char * filename)
{
FILE * fd=fopen(filename, "rb");
if(!fd)
printf("ERROR on file %s\n", filename);
fseek(fd, 0, SEEK_END);
size_t size = ftell(fd);
fclose(fd);
return size;
}
#endif
int main()
{
char buf[256];
int i, n;
for(i=0; i<10000; ++i)
{
sprintf(buf, "file_%d", i);
if(getFileSize(buf)!= 2048)
printf("WRONG!\n");
}
return 0;
}
Logically, one would assume that fseek() when prompted to seek to the end of the file uses stat to know how far to seek, or rather, where the end of the file is.
This would make fseek slower than using the facilities directly, and it also requires you to fopen the file in the first place.
Still, any performance difference is likely to be negligible, and if you need to open the file for some reason anyway, fseek/ftell likely improves the readability of your code significantly.
For stat.h you mainly want to use it to tell the stats of the file. Like if you want to tell if it's a file or a directory, etc.
However, if you want to do manipulations with the file, then you'll probably want to use ftell() and fseek(). That is you're actually doing manipulations on the file stream itself.
So in terms of performance, it's really what you need.
Hope it helps :) Cheers!
Depending on the circumstances, stat() can be hundred of times faster then seek()/tell(). I am currently toying around with sshfs/FUSE and getting the file size of a few thousand files with seek()/tell() takes well over a minute, doing it with stat() takes a second. So the difference is pretty huge when working over sshfs/FUSE.
I want to use /dev/random or /dev/urandom in C. How can I do it? I don't know how can I handle them in C, if someone knows please tell me how. Thank you.
In general, it's a better idea to avoid opening files to get random data, because of how many points of failure there are in the procedure.
On recent Linux distributions, the getrandom system call can be used to get crypto-secure random numbers, and it cannot fail if GRND_RANDOM is not specified as a flag and the read amount is at most 256 bytes.
As of October 2017, OpenBSD, Darwin and Linux (with -lbsd) now all have an implementation of arc4random that is crypto-secure and that cannot fail. That makes it a very attractive option:
char myRandomData[50];
arc4random_buf(myRandomData, sizeof myRandomData); // done!
Otherwise, you can use the random devices as if they were files. You read from them and you get random data. I'm using open/read here, but fopen/fread would work just as well.
int randomData = open("/dev/urandom", O_RDONLY);
if (randomData < 0)
{
// something went wrong
}
else
{
char myRandomData[50];
ssize_t result = read(randomData, myRandomData, sizeof myRandomData);
if (result < 0)
{
// something went wrong
}
}
You may read many more random bytes before closing the file descriptor. /dev/urandom never blocks and always fills in as many bytes as you've requested, unless the system call is interrupted by a signal. It is considered cryptographically secure and should be your go-to random device.
/dev/random is more finicky. On most platforms, it can return fewer bytes than you've asked for and it can block if not enough bytes are available. This makes the error handling story more complex:
int randomData = open("/dev/random", O_RDONLY);
if (randomData < 0)
{
// something went wrong
}
else
{
char myRandomData[50];
size_t randomDataLen = 0;
while (randomDataLen < sizeof myRandomData)
{
ssize_t result = read(randomData, myRandomData + randomDataLen, (sizeof myRandomData) - randomDataLen);
if (result < 0)
{
// something went wrong
}
randomDataLen += result;
}
close(randomData);
}
There are other accurate answers above. I needed to use a FILE* stream, though. Here's what I did...
int byte_count = 64;
char data[64];
FILE *fp;
fp = fopen("/dev/urandom", "r");
fread(&data, 1, byte_count, fp);
fclose(fp);
Just open the file for reading and then read data. In C++11 you may wish to use std::random_device which provides cross-platform access to such devices.
Zneak is 100% correct. Its also very common to read a buffer of random numbers that is slightly larger than what you'll need on startup. You can then populate an array in memory, or write them to your own file for later re-use.
A typical implementation of the above:
typedef struct prandom {
struct prandom *prev;
int64_t number;
struct prandom *next;
} prandom_t;
This becomes more or less like a tape that just advances which can be magically replenished by another thread as needed. There are a lot of services that provide large file dumps of nothing but random numbers that are generated with much stronger generators such as:
Radioactive decay
Optical behavior (photons hitting a semi transparent mirror)
Atmospheric noise (not as strong as the above)
Farms of intoxicated monkeys typing on keyboards and moving mice (kidding)
Don't use 'pre-packaged' entropy for cryptographic seeds, in case that doesn't go without saying. Those sets are fine for simulations, not fine at all for generating keys and such.
Not being concerned with quality, if you need a lot of numbers for something like a monte carlo simulation, it's much better to have them available in a way that will not cause read() to block.
However, remember, the randomness of a number is as deterministic as the complexity involved in generating it. /dev/random and /dev/urandom are convenient, but not as strong as using a HRNG (or downloading a large dump from a HRNG). Also worth noting that /dev/random refills via entropy, so it can block for quite a while depending on circumstances.
zneak's answer covers it simply, however the reality is more complicated than that. For example, you need to consider whether /dev/{u}random really is the random number device in the first place. Such a scenario may occur if your machine has been compromised and the devices replaced with symlinks to /dev/zero or a sparse file. If this happens, the random stream is now completely predictable.
The simplest way (at least on Linux and FreeBSD) is to perform an ioctl call on the device that will only succeed if the device is a random generator:
int data;
int result = ioctl(fd, RNDGETENTCNT, &data);
// Upon success data now contains amount of entropy available in bits
If this is performed before the first read of the random device, then there's a fair bet that you've got the random device. So #zneak's answer can better be extended to be:
int randomData = open("/dev/random", O_RDONLY);
int entropy;
int result = ioctl(randomData, RNDGETENTCNT, &entropy);
if (!result) {
// Error - /dev/random isn't actually a random device
return;
}
if (entropy < sizeof(int) * 8) {
// Error - there's not enough bits of entropy in the random device to fill the buffer
return;
}
int myRandomInteger;
size_t randomDataLen = 0;
while (randomDataLen < sizeof myRandomInteger)
{
ssize_t result = read(randomData, ((char*)&myRandomInteger) + randomDataLen, (sizeof myRandomInteger) - randomDataLen);
if (result < 0)
{
// error, unable to read /dev/random
}
randomDataLen += result;
}
close(randomData);
The Insane Coding blog covered this, and other pitfalls not so long ago; I strongly recommend reading the entire article. I have to give credit to their where this solution was pulled from.
Edited to add (2014-07-25)...
Co-incidentally, I read last night that as part of the LibReSSL effort, Linux appears to be getting a GetRandom() syscall. As at time of writing, there's no word of when it will be available in a kernel general release. However this would be the preferred interface to get cryptographically secure random data as it removes all pitfalls that access via files provides. See also the LibReSSL possible implementation.
I'm taking a networking class at school and am using C/GDB for the first time. Our assignment is to make a webserver that communicates with a client browser. I am well underway and can open files and send them to the client. Everything goes great till I open a very large file and then I seg fault. I'm not a pro at C/GDB so I'm sorry if that is causing me to ask silly questions and not be able to see the solution myself but when I looked at the dumped core I see my seg fault comes here:
if (-1 == (openfd = open(path, O_RDONLY)))
Specifically we are tasked with opening the file and the sending it to the client browser. My Algorithm goes:
Open/Error catch
Read the file into a buffer/Error catch
Send the file
We were also tasked with making sure that the server doesn't crash when SENDING very large files. But my problem seems to be with opening them. I can send all my smaller files just fine. The file in question is 29.5MB.
The whole algorithm is:
ssize_t send_file(int conn, char *path, int len, int blksize, char *mime) {
int openfd; // File descriptor for file we open at path
int temp; // Counter for the size of the file that we send
char buffer[len]; // Buffer to read the file we are opening that is len big
// Open the file
if (-1 == (openfd = open(path, O_RDONLY))) {
send_head(conn, "", 400, strlen(ERROR_400));
(void) send(conn, ERROR_400, strlen(ERROR_400), 0);
logwrite(stdout, CANT_OPEN);
return -1;
}
// Read from file
if (-1 == read(openfd, buffer, len)) {
send_head(conn, "", 400, strlen(ERROR_400));
(void) send(conn, ERROR_400, strlen(ERROR_400), 0);
logwrite(stdout, CANT_OPEN);
return -1;
}
(void) close(openfd);
// Send the buffer now
logwrite(stdout, SUC_REQ);
send_head(conn, mime, 200, len);
send(conn, &buffer[0], len, 0);
return len;
}
I dunno if it is just a fact that a I am Unix/C novice. Sorry if it is. =( But you're help is much appreciated.
It's possible I'm just misunderstanding what you meant in your question, but I feel I should point out that in general, it's a bad idea to try to read the entire file at once, in case you deal with something that's just too big for your memory to handle.
It's smarter to allocate a buffer of a specific size, say 8192 bytes (well, that's what I tend to do a lot, anyway), and just always read and send that much, as much as necessary, until your read() operation returns 0 (and no errno set) for end of stream.
I suspect you have a stackoverflow (I should get bonus points for using that term on this site).
The problem is you are allocating the buffer for the entire file on the stack all at once. For larger files, this buffer is larger than the stack, and the next time you try to call a function (and thus put some parameters for it on the stack) the program crashes.
The crash appears at the open line because allocating the buffer on the stack doesn't actually write any memory, it just changes the stack pointer. When your call to open tries tow rite the parameters to the stack, the top of the stack is now overflown and this causes a crash.
The solution is as Platinum Azure or dreamlax suggest, read in the file little bits at a time or allocate your buffer on the heap will malloc or new.
Rather than using a variable length array, perhaps try allocated the memory using malloc.
char *buffer = malloc (len);
...
free (buffer);
I just did some simple tests on my system, and when I use variable length arrays of a big size (like the size you're having trouble with), I also get a SEGFAULT.
You're allocating the buffer on the stack, and it's way too big.
When you allocate storage on the stack, all the compiler does is decrease the stack pointer enough to make that much room (this keeps stack variable allocation to constant time). It does not try to touch any of this stacked memory. Then, when you call open(), it tries to put the parameters on the stack and discovers it has overflowed the stack and dies.
You need to either operate on the file in chunks, memory-map it (mmap()), or malloc() storage.
Also, path should be declared const char*.