Split file into n smaller files in C - c

Question

I'm making a program to split a file into N smaller parts
of (almost) equal sizes. So here's my code:
FILE * fp = fopen(file,"r");
long aux;
long cursor = 0;
long blockSize = 1024000; //supose each smaller file will have 1 MB
long bytesLimit = blockSize;
for( i = 0 ; i < n ; i++) {
FILE * fp_aux = fopen( outputs[i] , "w"); //outputs is an array of temporary file names
while(cursor < bytesLimit) { //here occurs the infinite loop
fscanf(fp,"%lu\n",&aux);
fprintf(fp_aux,"%lu\n",aux);
cursor = ftell(fp);
}
fclose(fp_aux);
bytesLimit = bytesLimit + blockSize;
}
//here add some more logic to get the remaining content left in the main file
The code works if I want to split the file into two or three parts, but when I try to split it into 10 parts, fscanf locks on reading the same number and stays on an infinite loop there.
My input file has the format "%lu\n" like below:
1231231
4341342
4564565
...

Answer 1

If splitting a file is the focus, then simplify your method. Because your post indicates you are working with a text file, the assumption is that it contains words with punctuation, numbers, linefeeds etc. With this type of content, it can be parsed into lines using fgets()/fputs(). This will allow you to read lines from one large file, tracking accumulated size as you go, and writing lines to several smaller files...
Some simple steps:
1) determine file size of file to be split
2) Set desired small file size.
3) open large file
4) Use fgets/fputs in a loop, opening and closing files to split contents, using accumulated size as split point.
5) Clean up. (fclose files etc.)
Here is an example that will illustrate these steps. This splits a large text file by size, regardless of text content. (I used a text file with 130K of volume and split it into segments of 5k
#define SEGMENT 5000 //approximate target size of small file
long file_size(char *name);//function definition below
int main(void)
{
int segments=0, i, len, accum;
FILE *fp1, *fp2;
long sizeFile = file_size(largeFileName);
segments = sizeFile/SEGMENT + 1;//ensure end of file
char filename[260]={"c:\\play\\smallFileName_"};//base name for small files.
char largeFileName[]={"c:\\play\\largeFileName.txt"};//change to your path
char smallFileName[260];
char line[1080];
fp1 = fopen(largeFileName, "r");
if(fp1)
{
for(i=0;i<segments;i++)
{
accum = 0;
sprintf(smallFileName, "%s%d.txt", filename, i);
fp2 = fopen(smallFileName, "w");
if(fp2)
{
while(fgets(line, 1080, fp1) && accum <= SEGMENT)
{
accum += strlen(line);//track size of growing file
fputs(line, fp2);
}
fclose(fp2);
}
}
fclose(fp1);
}
return 0;
}
long file_size(char *name)
{
FILE *fp = fopen(name, "rb"); //must be binary read to get bytes
long size=-1;
if(fp)
{
fseek (fp, 0, SEEK_END);
size = ftell(fp)+1;
fclose(fp);
}
return size;
}

Answer 2

if you have bad data in the file that isn't a long unsigned int format then the fscanf will read it, the file pointer for the fp file object won't change. Then the program sets the fp file pointer back to the start of that read and it will do it again
To prevent this you need to check the return value of the fscanf to see that it has an appropriate value ( probably 1 )

Answer 3

If you want to split a file into several parts with a specified maximum file size of each part, why do you use fscanf(..), ftell(..) and fprintf(..)?
This is not the fastest way to achieve your goal...
I recommend doing it in this way:
Open input file
As long as there is input data (!feof(..))
Open output file (if not already open)
Read block of input data (fread)
Write block of data to output file (fwrite)
track number of bytes written and close output file if maximum file size is reached
Go back to step 2.
Clean up
If doing so the split files will not exceed a specific maximum file size. Additionally you avoid usage of slow file I/O functions like fprintf.
A possible implementation would look like this:
/*
** splitFile
** Splits an existing input file into multiple output files with a specified
** maximum file size.
**
** Return Value:
** Number of created result files, or 0 in case of bad input data or a negative
** value in case of an error during file splitting.
*/
int splitFile(char *fileIn, size_t maxSize)
{
int result = 0;
FILE *fIn;
FILE *fOut;
char buffer[1024 * 16];
size_t size;
size_t read;
size_t written;
if ((fileIn != NULL) && (maxSize > 0))
{
fIn = fopen(fileIn, "rb");
if (fIn != NULL)
{
fOut = NULL;
result = 1; /* we have at least one part */
while (!feof(fIn))
{
/* initialize (next) output file if no output file opened */
if (fOut == NULL)
{
sprintf(buffer, "%s.%03d", fileIn, result);
fOut = fopen(buffer, "wb");
if (fOut == NULL)
{
result *= -1;
break;
}
size = 0;
}
/* calculate size of data to be read from input file in order to not exceed maxSize */
read = sizeof(buffer);
if ((size + read) > maxSize)
{
read = maxSize - size;
}
/* read data from input file */
read = fread(buffer, 1, read, fIn);
if (read == 0)
{
result *= -1;
break;
}
/* write data to output file */
written = fwrite(buffer, 1, read, fOut);
if (written != read)
{
result *= -1;
break;
}
/* update size counter of current output file */
size += written;
if (size >= maxSize) /* next split? */
{
fclose(fOut);
fOut = NULL;
result++;
}
}
/* clean up */
if (fOut != NULL)
{
fclose(fOut);
}
fclose(fIn);
}
}
return (result);
}
The above code split a test file with a size of 126803945 bytes into 121 1MB parts in about 500ms.
Note that the size of buffer (here: 16KB) affects the speed a file is split. The bigger the buffer the faster a huge file is split. If you want to use really large buffers (>1MB or so) you have to allocate (and free) the buffer on each call (or use a static buffer if you do not need reentrant code).