Read from a file line by line block by block in C - c

Question

I need to read 4000 lines from a file, do some manipulations with them, then read next 4000 lines, do some manipulations with them, then read next 4000 lines and so on till the end of some BIG file. How to do this correctly?
Here is my basic code which is just reading all lines but not block by block from a file:
#include <stdio.h>
#include <string.h>
#define bufSize 1024
int main(int argc, char *argv[])
{
FILE* fp;
char buf[bufSize];
if (argc != 2)
{
fprintf(stderr, "Usage: %s <soure-file>\n", argv[0]);
return 1;
}
if ((fp = fopen(argv[1], "r")) == NULL)
{ /* Open source file. */
perror("fopen source-file");
return 1;
}
while (fgets(buf, sizeof(buf), fp) != NULL)
{
buf[strlen(buf) - 1] = '\0'; /* eat the newline fgets() stores */
printf("%s\n", buf);
}
fclose(fp);
return 0;
}

Answer 1

What is the problem?
You would need to use 2 loops. The outer one would repeat reading chunks of lines until EOF.
The code could look like the following sketch:
...
while (1)
{
/* read <= 4000 lines and process */
}
The inner would read the lines and stores them away:
size_t lines = 0; /** next index to be used with lineBuffer
(and number of lines already stored)*/
char *lineBuffer[4000];
char buf[bufSize];
while (lines < 4000 && fgets(buf, sizeof(buf), fp) != NULL)
{
buf[strlen(buf) - 1] = '\0';
lineBuffer[lines] = malloc(strlen(buf);
strcpy(lineBuffer[lines], buf);
lines++;
}
if (lines == 0)
{
break; /* we are done*/
}
/* do processing on data */
/* deallocate dynamic memory */
for (int i = lines - 1; lines>=0; i--)
{
free(lineBuffer[i]);
}
lines = 0;
Of course you could use static memory allocation using
char lineBuffer[4000][bufSize];
instead of mallocing. That would save doing alloc/dealloc sequences, or do allocation on top-level (outside the loops) using calloc(400*bufSize);
But this is left to personal preferences given the fact that a total of about 4MB of memory is being used with current settings.
With respect to performance:
having statically allocated memory might give some speed benefit.
You might also try increasing buffer size used by stdio (by a setbuffer() variant to read larger chunks from file.
Whether this will have any noticable effect is subject to some performance measurings you should take (if an issue in the first place)

Answer 2

Ok,then allocate a two dimensional array on the heap and declare a variable n of type size_t to keep track of the number of lines.Try the following:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define bufSize 1024
int main(int argc, char *argv[])
{
FILE* fp;
if (argc != 2)
{
fprintf(stderr, "Usage: %s <soure-file>\n", argv[0]);
return 1;
}
if ((fp = fopen(argv[1], "r")) == NULL)
{ /* Open source file. */
perror("fopen source-file");
return 1;
}
size_t n = 0;
char(*buf)[bufSize] = malloc(bufSize * 4000);
if (!buf) {
fprintf(stderr, "Error - Failed to allocate memory.\n");
fclose(fp);
return 1;
}
while (1)
{
if (fgets(buf[n], bufSize, fp) != NULL) {
n++;
if (n == 4000) {
/* do something */
// ...
n = 0;
}
}
else {
// do something with leftover
break;
}
}
free(buf);
fclose(fp);
return 0;
}