Related
I'm not an expert in C and I'm looking for some advice to to make my program more robust and reliable. Just to give some context: I've written a program to do some scientific computation that takes quite a long time (about 20h) that I'm executing on a large university HPC linux cluster using a SLRUM scheduling system and NFS mounted file systems. What seems to happen is that some time during the 20h the connection to the file system goes stale (on the entire machine; independent of my program) and the first attempt to open & write a file takes a really long time and that results in a segfault cored dumped error that I have so far not been able to precisely track down. Below is a minimal file that at least conceptually reproduces the error: The program starts, opens a file and everything works. The program does some long computation (simulated by sleep()), tries to open & write to the same file again, and it fails. What are some conventions to make my code more robust and reliably write my results to file without crashing?
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char **argv) {
// Declare variables
FILE *outfile;
char outname[150] = "result.csv";
// Open file for writing
printf("CHECKING if output file '%s' is writable?", outname);
outfile=fopen(outname, "w");
if (outfile == NULL) {
perror("Failed: ");
exit(EXIT_FAILURE);
}
fclose(outfile);
printf(" PASSED.\n");
// Do some computation that takes really long (around 19h)
sleep(3);
// Open file again and Write results
printf("Writing results to %s ...", outname);
outfile=fopen(outname, "w");
if (outfile == NULL) {
perror("Failed writing in tabulate_vector_new: ");
exit(EXIT_FAILURE);
}
fprintf( outfile, "This is the important result.\n");
fclose(outfile);
printf(" DONE.\n");
return 0;
}
It seems odd that your program would segfault due to an NFS issue. I would expect it to hang indefinitely, not crash. That having been said, I would suggest forking a new process to check whether the NFS mount is working. That way, your important code won't be directly involved in testing the problematic file system. Something like the following approach may be useful:
pid_t pid = fork();
if (pid == -1)
{
// error, failed to fork(). should probably give up now. something is really wrong.
}
else if (pid > 0)
{
// if the child exits, it has successfully interacted with the NFS file system
wait(NULL);
// proceed with attempting to write important data
}
else
{
// we are the child; fork df in order to test the NFS file system
execlp("df", "df", "/mnt", (char *)NULL)
// the child has been replaced by df, which will try to statfs(2) /mnt for us
}
The general concept here is that we utilize the df command to check whether the NFS file system (which I assume is at /mnt) is working. If it's temporarily not working, df should hang until it starts working again, and then exit, returning control to your program. If you suspect df might hang forever, you could enhance my example by using alarm(2) to wait a certain period of time, probably at least a few minutes, after which you could retry running df. Note that this could result in zombie df processes sticking around.
In the end, the correct solution is to try to get a more reliable NFS server, but until you can do that, I hope this is helpful.
This is my program:
#include <stdio.h>
int main() {
FILE *logh;
logh = fopen("/home/user1/data.txt", "a+");
if (logh == NULL)
{
printf("error creating file \n");
return -1;
}
// write some data to the log handle and check if it gets written..
int result = fprintf(logh, "this is some test data \n");
if (result > 0)
printf("write successful \n");
else
printf("couldn't write the data to filesystem \n");
while (1) {
};
fclose(logh);
return 0;
}
When i run this program, i see that the file is getting created but it does not contain any data. what i understand i that there is data caching in memory before the data is actually written to the filesystem to avoid multiple IOs to increase performance. and I also know that i can call fsync/fdatasync inside the program to force a sync. but can i force the sync from outside without having to change the program?
I tried running sync command from Linux shell but it does not make the data to appear on the file. :(
Please help if anybody knows any alternative to do the same.
One useful information: I was researching some more on this and finally found this, to remove internal buffering altogether, the FILE mode can be set to _IONBF using int setvbuf(FILE *stream, char *buf, int mode, size_t size)
The IO functions usingFILE pointers cache the data to be written in an internal buffer within the program's memory until they decide to perform a system call to 'really' write it (which is for normal files usually when the size of the data cached reaches BUFSIZ).
Until then, there is no way to force writing from outside the progam.
The problem is that your program does not close the file because of your while statement. Remove these lines:
while (1) {
};
If the intent is to wait forever, then close the file with fclose before executing the while statement.
What I am trying to do
So, I have been trying to access keyboard input in Linux. Specifically, I need to be able to access modifier key presses without other keys being pressed. Furthermore, I want to be able to do this without an X system running.
So, in short, my requirements are these:
Works on Linux
Does not need X11
Can retrieve modifier key press without any other keys being pressed
This includes the following keys:
Shift
Control
Alt
All I need is a simple 0 = not pressed, 1 = currently pressed to let me know if
the key is being held down when the keyboard is checked
My computer setup
My normal Linux machine is on a truck towards my new apartment; so, I only have a Macbook Air to work with right now. Therefore, I am running Linux in a VM to test this out.
Virtual Machine in VirtualBox
OS: Linux Mint 16
Desktop Environment: XFCE
Everything below was done in this environment. I've tried both with X running and in one of the other ttys.
My Thoughts
I'll alter this if someone can correct me.
I've done a fair bit of reading to realize that higher-level libraries do not provide this kind of functionality. Modifier keys are used with other keys to provide an alternate key code. Accessing the modifier keys themselves through a high-level library in Linux isn't as easy. Or, rather, I haven't found a high-level API for this on Linux.
I thought libtermkey would be the answer, but it doesn't seem to support the Shift modifier key any better than normal keystroke retrieval. I'm also not sure if it works without X.
While working with libtermkey (before I realized it didn't get shift in cases like Shift-Return), I was planning to write a daemon that would run to gather keyboard events. Running copies of the daemon program would simply pipe requests for keyboard data and receive keyboard data in response. I could use this setup to have something always running in the background, in case I cannot check key code statuses at specific times (have to be receive key codes as they happen).
Below are my two attempts to write a program that can read from the Linux keyboard device. I've also included my small check to make sure I had the right device.
Attempt #1
I have tried to access the keyboard device directly, but am encountering issues. I have tried the suggestion here that is in another Stack Overflow thread. It gave me a segmentation fault; so, I changed it from fopen to open:
// ...
int fd;
fd = open("/dev/input/by-path/platform-i8042-serio-0-event-kbd", O_RDONLY);
char key_map[KEY_MAX/8 + 1];
memset(key_map, 0, sizeof(key_map));
ioctl(fd, EVIOCGKEY(sizeof key_map), key_map);
// ...
While there was no segmentation fault, there was no indicator of any key press (not just modifier keys). I tested this using:
./foo && echo "TRUE" || echo "FALSE"
I've used that to test for successful return codes from commands quite a lot; so, I know that's fine. I've also outputted the key (always 0) and mask (0100) to check. It just doesn't seem to detect anything.
Attempt #2
From here, I thought I'd try a slightly different approach. I wanted to figure out what I was doing wrong. Following this page providing a snippet demonstrating printing out key codes, I bundled that into a program:
#include <stdio.h>
#include <stdint.h>
#include <string.h>
#include <fcntl.h>
#include <linux/input.h>
int main(int argc, char** argv) {
uint8_t keys[128];
int fd;
fd = open("/dev/input/by-path/platform-i8042-serio-event-kbd", O_RDONLY);
for (;;) {
memset(keys, 0, 128);
ioctl (fd, EVIOCGKEY(sizeof keys), keys);
int i, j;
for (i = 0; i < sizeof keys; i++)
for (j = 0; j < 8; j++)
if (keys[i] & (1 << j))
printf ("key code %d\n", (i*8) + j);
}
return 0;
}
Previously, I had the size to 16 bytes instead of 128 bytes. I should honestly spend a bit more time understanding ioctl and EVIOCGKEY. I just know that it supposedly maps bits to specific keys to indicate presses, or something like that (correct me if I'm wrong, please!).
I also didn't have a loop initially and would just hold down various keys to see if a key code appeared. I received nothing; so, I thought a loop might make the check easier to test in case a missed something.
How I know the input device is the right one
I tested it by running cat on the input device. Specifically:
$ sudo cat /dev/input/by-path/platform-i8042-serio-0-event-kbd
Garbage ASCII was sent to my terminal on key press and release events starting with the return (enter) key when I began the output using cat. I also know that this seems to work fine with modifier keys like shift, control, function, and even Apple's command key on my Macbook running a Linux VM. Output appeared when a key was pressed, began to appear rapidly from subsequent signals sent by holding the key down, and outputted more data when a key was released.
So, while my approach may not be the right one (I'm willing to hear any alternative), the device seems to provide what I need.
Furthermore, I know that this device is just a link pointing to /dev/input/event2 from running:
$ ls -l /dev/input/by-path/platform-i8042-serio-0-event-kbd
I've tried both programs above with /dev/input/event2 and received no data. Running cat on /dev/input/event2 provided the same output as with the link.
Open the input device,
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <linux/input.h>
#include <string.h>
#include <stdio.h>
static const char *const evval[3] = {
"RELEASED",
"PRESSED ",
"REPEATED"
};
int main(void)
{
const char *dev = "/dev/input/by-path/platform-i8042-serio-0-event-kbd";
struct input_event ev;
ssize_t n;
int fd;
fd = open(dev, O_RDONLY);
if (fd == -1) {
fprintf(stderr, "Cannot open %s: %s.\n", dev, strerror(errno));
return EXIT_FAILURE;
}
and then read keyboard events from the device:
while (1) {
n = read(fd, &ev, sizeof ev);
if (n == (ssize_t)-1) {
if (errno == EINTR)
continue;
else
break;
} else
if (n != sizeof ev) {
errno = EIO;
break;
}
The above snippet breaks out from the loop if any error occurs, or if the userspace receives only a partial event structure (which should not happen, but might in some future/buggy kernels). You might wish to use a more robust read loop; I personally would be satisfied by replacing the last break with continue, so that partial event structures are ignored.
You can then examine the ev event structure to see what occurred, and finish the program:
if (ev.type == EV_KEY && ev.value >= 0 && ev.value <= 2)
printf("%s 0x%04x (%d)\n", evval[ev.value], (int)ev.code, (int)ev.code);
}
fflush(stdout);
fprintf(stderr, "%s.\n", strerror(errno));
return EXIT_FAILURE;
}
For a keypress,
ev.time: time of the event (struct timeval type)
ev.type: EV_KEY
ev.code: KEY_*, key identifier; see complete list in /usr/include/linux/input.h
ev.value: 0 if key release, 1 if key press, 2 if autorepeat keypress
See Documentation/input/input.txt in the Linux kernel sources for further details.
The named constants in /usr/include/linux/input.h are quite stable, because it is a kernel-userspace interface, and the kernel developers try very hard to maintain compatibility. (That is, you can expect there to be new codes every now and then, but existing codes rarely change.)
I have a C application which generates a lot of output and for which speed is critical. The program is basically a loop over a large (8-12GB) binary input file which must be read sequentially. In each iteration the read bytes are processed and output is generated and written to multiple files, but never to multiple files at the same time. So if you are at the point where output is generated and there are 4 output files you write to either file 0 or 1 or 2, or 3. At the end of the iteration I now write the output using fwrite(), thereby waiting for the write operation to finish. The total number of output operations is large, up to 4 million per file, and output size of files ranges from 100mb to 3.5GB. The program runs on a basic multicore processor.
I want to write output in a separate thread and I know this can be done with
Asyncronous I/O
Creating threads
I/O completion ports
I have 2 type of questions, namely conceptual and code specific.
Conceptual Question
What would be the best approach. Note that the application should be portable to Linux, however, I don't see how that would be very important for my choice for 1-3, since I would write a wrapper around anything kernel/API specific. For me the most important criteria is speed. I have read that option 1 is not that likely to increase the performance of the program and that the kernel in any case creates new threads for the i/o operation, so then why not use option (2) immediately with the advantage that it seems easier to program (also since I did not succeed with option (1), see code issues below).
Note that I read https://stackoverflow.com/questions/3689759/how-can-i-run-a-specific-function-of-thread-asynchronously-in-c-c, but I dont see a motivation on what to use based on the nature of the application. So I hope somebody could provide me with some advice what would be best in my situation. Also from the book "Windows System Programming" by Johnson M. Hart, I know that the recommendation is using threads, mainly because of the simplicity. However, will it also be fastest?
Code Question
This question involves the attempts I made so far to make asynchronous I/O work. I understand that its a big piece of code so that its not that easy to look into. In any case I would really appreciate any attempt.
To decrease execution time I try to write the output by means of a new thread using WINAPI via CreateFile() with FILE_FLAGGED_OVERLAP with an overlapped structure. I have created a sample program in which I try to get this to work. However, I encountered 2 problems:
The file is only opened in overlapped mode when I delete an already existing file (I have tried using CreateFile in different modes (CREATE_ALWAYS, CREATE_NEW, OPEN_EXISTING), but this does not help).
Only the first WriteFile is executed asynchronously. The remainder of WriteFile commands is synchronous. For this problem I already consulted http://support.microsoft.com/kb/156932. It seems that the problem I have is related to the fact that "any write operation to a file that extends its length will be synchronous". I've already tried to solve this by increasing file size/valid data size (commented region in code). However, I still do not get it to work. I'm aware of the fact that it could be the case that to get most out of asynchronous io i should CreateFile with FILE_FLAG_NO_BUFFERING, however I cannot get this to work as well.
Please note that the program creates a file of about 120mb in the path of execution. Also note that print statements "not ok" are not desireable, I would like to see "can do work in background" appear on my screen... What goes wrong here?
#include <windows.h>
#include <math.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#define ASYNC // remove this definition to run synchronously (i.e. using fwrite)
#ifdef ASYNC
struct _OVERLAPPED *pOverlapped;
HANDLE *pEventH;
HANDLE *pFile;
#else
FILE *pFile;
#endif
#define DIM_X 100
#define DIM_Y 150000
#define _PRINTERROR(msgs)\
{printf("file: %s, line: %d, %s",__FILE__,__LINE__,msgs);\
fflush(stdout);\
return 0;} \
#define _PRINTF(msgs)\
{printf(msgs);\
fflush(stdout);} \
#define _START_TIMER \
time_t time1,time2; \
clock_t clock1; \
time(&time1); \
printf("start time: %s",ctime(&time1)); \
fflush(stdout);
#define _END_TIMER\
time(&time2);\
clock1 = clock();\
printf("end time: %s",ctime(&time2));\
printf("elapsed processor time: %.2f\n",(((float)clock1)/CLOCKS_PER_SEC));\
fflush(stdout);
double aio_dat[DIM_Y] = {0};
double do_compute(double A,double B, int arr_len);
int main()
{
_START_TIMER;
const char *pName = "test1.bin";
DWORD dwBytesToWrite;
BOOL bErrorFlag = FALSE;
int j=0;
int i=0;
int fOverlapped=0;
#ifdef ASYNC
// create / open the file
pFile=CreateFile(pName,
GENERIC_WRITE, // open for writing
0, // share write access
NULL, // default security
CREATE_ALWAYS, // create new/overwrite existing
FILE_FLAG_OVERLAPPED, // | FILE_FLAG_NO_BUFFERING, // overlapped file
NULL); // no attr. template
// check whether file opening was ok
if(pFile==INVALID_HANDLE_VALUE){
printf("%x\n",GetLastError());
_PRINTERROR("file not opened properly\n");
}
// make the overlapped structure
pOverlapped = calloc(1,sizeof(struct _OVERLAPPED));
pOverlapped->Offset = 0;
pOverlapped->OffsetHigh = 0;
// put event handle in overlapped structure
if(!(pOverlapped->hEvent = CreateEvent(NULL,TRUE,FALSE,NULL))){
printf("%x\n",GetLastError());
_PRINTERROR("error in createevent\n");
}
#else
pFile = fopen(pName,"wb");
#endif
// create some output
for(j=0;j<DIM_Y;j++){
aio_dat[j] = do_compute(i, j, DIM_X);
}
// determine how many bytes should be written
dwBytesToWrite = (DWORD)sizeof(aio_dat);
for(i=0;i<DIM_X;i++){ // do this DIM_X times
#ifdef ASYNC
//if(i>0){
//SetFilePointer(pFile,dwBytesToWrite,NULL,FILE_CURRENT);
//if(!(SetEndOfFile(pFile))){
// printf("%i\n",pFile);
// _PRINTERROR("error in set end of file\n");
//}
//SetFilePointer(pFile,-dwBytesToWrite,NULL,FILE_CURRENT);
//}
// write the bytes
if(!(bErrorFlag = WriteFile(pFile,aio_dat,dwBytesToWrite,NULL,pOverlapped))){
// check whether io pending or some other error
if(GetLastError()!=ERROR_IO_PENDING){
printf("lasterror: %x\n",GetLastError());
_PRINTERROR("error while writing file\n");
}
else{
fOverlapped=1;
}
}
else{
// if you get here output got immediately written; bad!
fOverlapped=0;
}
if(fOverlapped){
// do background, this msgs is what I want to see
for(j=0;j<DIM_Y;j++){
aio_dat[j] = do_compute(i, j, DIM_X);
}
for(j=0;j<DIM_Y;j++){
aio_dat[j] = do_compute(i, j, DIM_X);
}
_PRINTF("can do work in background\n");
}
else{
// not overlapped, this message is bad
_PRINTF("not ok\n");
}
// wait to continue
if((WaitForSingleObject(pOverlapped->hEvent,INFINITE))!=WAIT_OBJECT_0){
_PRINTERROR("waiting did not succeed\n");
}
// reset event structure
if(!(ResetEvent(pOverlapped->hEvent))){
printf("%x\n",GetLastError());
_PRINTERROR("error in resetevent\n");
}
pOverlapped->Offset+=dwBytesToWrite;
#else
fwrite(aio_dat,sizeof(double),DIM_Y,pFile);
for(j=0;j<DIM_Y;j++){
aio_dat[j] = do_compute(i, j, DIM_X);
}
for(j=0;j<DIM_Y;j++){
aio_dat[j] = do_compute(i, j, DIM_X);
}
#endif
}
#ifdef ASYNC
CloseHandle(pFile);
free(pOverlapped);
#else
fclose(pFile);
#endif
_END_TIMER;
return 1;
}
double do_compute(double A,double B, int arr_len)
{
int i;
double res = 0;
double *xA = malloc(arr_len * sizeof(double));
double *xB = malloc(arr_len * sizeof(double));
if ( !xA || !xB )
abort();
for (i = 0; i < arr_len; i++) {
xA[i] = sin(A);
xB[i] = cos(B);
res = res + xA[i]*xA[i];
}
free(xA);
free(xB);
return res;
}
Useful links
http://software.intel.com/sites/products/documentation/studio/composer/en-us/2011/compiler_c/cref_cls/common/cppref_asynchioC_aio_read_write_eg.htm
http://www.ibm.com/developerworks/linux/library/l-async/?ca=dgr-lnxw02aUsingPOISIXAIOAPI
http://www.flounder.com/asynchexplorer.htm#Asynchronous%20I/O
I know this is a big question and I would like to thank everybody in advance who takes the trouble reading it and perhaps even respond!
You should be able to get this to work using the OVERLAPPED structure.
You're on the right track: the system is preventing you from writing asynchronously because every WriteFile extends the size of the file. However, you're doing the file size extension wrong. Simply calling SetFileSize will not actually reserve space in the MFT. Use the SetFileValidData function. This will allocate clusters for your file (note that they will contain whatever garbage the disk had there) and you should be able to execute WriteFile and your computation in parallel.
I would stay away from FILE_FLAG_NO_BUFFERING. You're after more performance with parallelism I presume? Don't prevent the cache from doing its job.
Another option that you did not consider is a memory mapped file. Those are available on Windows and Linux. There is a handy Boost abstraction that you could use.
With a memory mapped file, every thread in your process could write its output to the file on its own time, assuming that the record sizes are known and each thread has its own output area.
The operating system will take care of writing the mapped pages to disk when needed or when it gets around to it or when you close the file. Maybe when you close the file. Now that I think about it, some operating systems may require that you call msync to guarantee it.
I don't see why you would want to write asynchronously. Doing things in parallel does not make them faster in all cases. If you write two file at the same time to the same disk, it will almost always be a lot faster. If that is the case, just write them one after another.
If you have some fancy drive like SSD or a virtual RAM drive, parallel writing could be faster. You have to create an file with at full size and then do your parallel magic.
Asynchronous writing is nice, but is done by any OS anyway. The potential gain for you is that you can do other things than writing to disk like displaying a progress bar. This is where multi-threading can help you.
So imho you should use serial writing or parallel writing to multiple disks.
hth
This looks like a simple question, but I didn't find anything similar here.
Since there is no file copy function in C, we have to implement file copying ourselves, but I don't like reinventing the wheel even for trivial stuff like that, so I'd like to ask the cloud:
What code would you recommend for file copying using fopen()/fread()/fwrite()?
What code would you recommend for file copying using open()/read()/write()?
This code should be portable (windows/mac/linux/bsd/qnx/younameit), stable, time tested, fast, memory efficient and etc. Getting into specific system's internals to squeeze some more performance is welcomed (like getting filesystem cluster size).
This seems like a trivial question but, for example, source code for CP command isn't 10 lines of C code.
This is the function I use when I need to copy from one file to another - with test harness:
/*
#(#)File: $RCSfile: fcopy.c,v $
#(#)Version: $Revision: 1.11 $
#(#)Last changed: $Date: 2008/02/11 07:28:06 $
#(#)Purpose: Copy the rest of file1 to file2
#(#)Author: J Leffler
#(#)Modified: 1991,1997,2000,2003,2005,2008
*/
/*TABSTOP=4*/
#include "jlss.h"
#include "stderr.h"
#ifndef lint
/* Prevent over-aggressive optimizers from eliminating ID string */
const char jlss_id_fcopy_c[] = "#(#)$Id: fcopy.c,v 1.11 2008/02/11 07:28:06 jleffler Exp $";
#endif /* lint */
void fcopy(FILE *f1, FILE *f2)
{
char buffer[BUFSIZ];
size_t n;
while ((n = fread(buffer, sizeof(char), sizeof(buffer), f1)) > 0)
{
if (fwrite(buffer, sizeof(char), n, f2) != n)
err_syserr("write failed\n");
}
}
#ifdef TEST
int main(int argc, char **argv)
{
FILE *fp1;
FILE *fp2;
err_setarg0(argv[0]);
if (argc != 3)
err_usage("from to");
if ((fp1 = fopen(argv[1], "rb")) == 0)
err_syserr("cannot open file %s for reading\n", argv[1]);
if ((fp2 = fopen(argv[2], "wb")) == 0)
err_syserr("cannot open file %s for writing\n", argv[2]);
fcopy(fp1, fp2);
return(0);
}
#endif /* TEST */
Clearly, this version uses file pointers from standard I/O and not file descriptors, but it is reasonably efficient and about as portable as it can be.
Well, except the error function - that's peculiar to me. As long as you handle errors cleanly, you should be OK. The "jlss.h" header declares fcopy(); the "stderr.h" header declares err_syserr() amongst many other similar error reporting functions. A simple version of the function follows - the real one adds the program name and does some other stuff.
#include "stderr.h"
#include <stdarg.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
void err_syserr(const char *fmt, ...)
{
int errnum = errno;
va_list args;
va_start(args, fmt);
vfprintf(stderr, fmt, args);
va_end(args);
if (errnum != 0)
fprintf(stderr, "(%d: %s)\n", errnum, strerror(errnum));
exit(1);
}
The code above may be treated as having a modern BSD license or GPL v3 at your choice.
As far as the actual I/O goes, the code I've written a million times in various guises for copying data from one stream to another goes something like this. It returns 0 on success, or -1 with errno set on error (in which case any number of bytes might have been copied).
Note that for copying regular files, you can skip the EAGAIN stuff, since regular files are always blocking I/O. But inevitably if you write this code, someone will use it on other types of file descriptors, so consider it a freebie.
There's a file-specific optimisation that GNU cp does, which I haven't bothered with here, that for long blocks of 0 bytes instead of writing you just extend the output file by seeking off the end.
void block(int fd, int event) {
pollfd topoll;
topoll.fd = fd;
topoll.events = event;
poll(&topoll, 1, -1);
// no need to check errors - if the stream is bust then the
// next read/write will tell us
}
int copy_data_buffer(int fdin, int fdout, void *buf, size_t bufsize) {
for(;;) {
void *pos;
// read data to buffer
ssize_t bytestowrite = read(fdin, buf, bufsize);
if (bytestowrite == 0) break; // end of input
if (bytestowrite == -1) {
if (errno == EINTR) continue; // signal handled
if (errno == EAGAIN) {
block(fdin, POLLIN);
continue;
}
return -1; // error
}
// write data from buffer
pos = buf;
while (bytestowrite > 0) {
ssize_t bytes_written = write(fdout, pos, bytestowrite);
if (bytes_written == -1) {
if (errno == EINTR) continue; // signal handled
if (errno == EAGAIN) {
block(fdout, POLLOUT);
continue;
}
return -1; // error
}
bytestowrite -= bytes_written;
pos += bytes_written;
}
}
return 0; // success
}
// Default value. I think it will get close to maximum speed on most
// systems, short of using mmap etc. But porters / integrators
// might want to set it smaller, if the system is very memory
// constrained and they don't want this routine to starve
// concurrent ops of memory. And they might want to set it larger
// if I'm completely wrong and larger buffers improve performance.
// It's worth trying several MB at least once, although with huge
// allocations you have to watch for the linux
// "crash on access instead of returning 0" behaviour for failed malloc.
#ifndef FILECOPY_BUFFER_SIZE
#define FILECOPY_BUFFER_SIZE (64*1024)
#endif
int copy_data(int fdin, int fdout) {
// optional exercise for reader: take the file size as a parameter,
// and don't use a buffer any bigger than that. This prevents
// memory-hogging if FILECOPY_BUFFER_SIZE is very large and the file
// is small.
for (size_t bufsize = FILECOPY_BUFFER_SIZE; bufsize >= 256; bufsize /= 2) {
void *buffer = malloc(bufsize);
if (buffer != NULL) {
int result = copy_data_buffer(fdin, fdout, buffer, bufsize);
free(buffer);
return result;
}
}
// could use a stack buffer here instead of failing, if desired.
// 128 bytes ought to fit on any stack worth having, but again
// this could be made configurable.
return -1; // errno is ENOMEM
}
To open the input file:
int fdin = open(infile, O_RDONLY|O_BINARY, 0);
if (fdin == -1) return -1;
Opening the output file is tricksy. As a basis, you want:
int fdout = open(outfile, O_WRONLY|O_BINARY|O_CREAT|O_TRUNC, 0x1ff);
if (fdout == -1) {
close(fdin);
return -1;
}
But there are confounding factors:
you need to special-case when the files are the same, and I can't remember how to do that portably.
if the output filename is a directory, you might want to copy the file into the directory.
if the output file already exists (open with O_EXCL to determine this and check for EEXIST on error), you might want to do something different, as cp -i does.
you might want the permissions of the output file to reflect those of the input file.
you might want other platform-specific meta-data to be copied.
you may or may not wish to unlink the output file on error.
Obviously the answers to all these questions could be "do the same as cp". In which case the answer to the original question is "ignore everything I or anyone else has said, and use the source of cp".
Btw, getting the filesystem's cluster size is next to useless. You'll almost always see speed increasing with buffer size long after you've passed the size of a disk block.
the size of each read need to be a multiple of 512 ( sector size ) 4096 is a good one
Here is a very easy and clear example: Copy a file. Since it is written in ANSI-C without any particular function calls I think this one would be pretty much portable.
Depending on what you mean by copying a file, it is certainly far from trivial. If you mean copying the content only, then there is almost nothing to do. But generally, you need to copy the metadata of the file, and that's surely platform dependent. I don't know of any C library which does what you want in a portable manner. Just handling the filename by itself is no trivial matter if you care about portability.
In C++, there is the file library in boost
One thing I found when implementing my own file copy, and it seems obvious but it's not: I/O's are slow. You can pretty much time your copy's speed by how many of them you do. So clearly you need to do as few of them as possible.
The best results I found were when I got myself a ginourmous buffer, read the entire source file into it in one I/O, then wrote the entire buffer back out of it in one I/O. If I even had to do it in 10 batches, it got way slow. Trying to read and write out each byte, like a naieve coder might try first, was just painful.
The accepted answer written by Steve Jessop does not answer to the first part of the quession, Jonathan Leffler do it, but do it wrong: code should be written as
while ((n = fread(buffer, 1, sizeof(buffer), f1)) > 0)
if (fwrite(buffer, n, 1, f2) != 1)
/* we got write error here */
/* test ferror(f1) for a read errors */
Explanation:
sizeof(char) = 1 by definition, always: it does not matter how many bits in it, 8 (in most cases), 9, 11 or 32 (on some DSP, for example) — size of char is one. Note, it is not an error here, but an extra code.
The fwrite function writes upto nmemb (second argument) elements of specified size (third argument), it does not required to write exactly nmemb elements. To fix this you must write the rest of the data readed or just write one element of size n — let fwrite do all his work. (This item is in question, should fwrite write all data or not, but in my version short writes impossible until error occurs.)
You should test for a read errors too: just test ferror(f1) at the end of loop.
Note, you probably need to disable buffering on both input and output files to prevent triple buffering: first on read to f1 buffer, second in our code, third on write to f2 buffer:
setvbuf(f1, NULL, _IONBF, 0);
setvbuf(f2, NULL, _IONBF, 0);
(Internal buffers should, probably, be of size BUFSIZ.)