C - insert lines from very big file into array - c

Question

First of all, I'm quite new with C, and I know this is a very repeated question, however, I could not find anything that could help me with my problem.
Here is my code:
It takes a text file and stores each line in an array.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main()
{
FILE *file;
file = fopen("test.txt", "r");
char buffer[600];
char *lines[10000];
int i = 0;
while(fgets(buffer, sizeof(buffer), file))
{
lines[i] = malloc(sizeof(buffer));
strcpy(lines[i], buffer);
i++;
free(lines[i]);
}
fclose(file);
return 1;
}
This works fine for small text files.
However it doesn't with large ones (even setting buffer and lines with much bigger numbers). Actually, if I increment buffer[] and *lines[] like 1000000 bytes, it doesn't give anything (if I understood well, it gives undefined behaviour). And I need to get this work with a 100.000 lines file with variable length lines,
So, how could I declare a very large array so I can pass each line? Since, as I exposed, it doesn't work with a large file.
Any help is appreciated!

Answer 1

char *lines[10000]; is just an array of pointers to the lines, not the array (memory) that is going to store the actual lines.
malloc is allocating a chunk of memory for each line, you are suppose to call free only when you are done using this chunk.
If you remove the free your solution would work, but you need to remember to free at some other point.

Answer 2

And I need to get this work with a 100.000 lines file with variable length lines,
So, how could I declare a very large array so I can pass each line?
This line
char *lines[10000];
gives you a variable with automatic storage duration - often called a local variable.
On most systems such a variable are located on a stack and most systems have a fixed limit for the size of the stack and thereby also a limit for the size of such a local variable.
So if you change the code to
char *lines[1000000];
to be able to handle larger files, it is likely that the variable use too much memory on the stack, i.e. you have a stack overflow.
A simple solution is to allocate the variable dynamically. Like:
char **lines = malloc(1000000 * sizeof *lines);
This will allocate 1000000 char-pointers and you can use lines as if it's an array - for instance like:
lines[i] = malloc(sizeof(buffer));
For something like this I'll also recommend that you take a look at realloc so that you can adjust the size of memory as needed.
Besides that your use of free seems strange and it's for sure wrong as you increment i between the malloc and the free.

Answer 3

You can allocate any space just as big as you need. So you will get rid of the fixed and limited numbers.
I have "massaged" your example in this way. The only thing I didn't is a first round through the file to obtain the longest line. So I kept the fixed buffer length.
Allocate only as many pointer to the lines as you need. For this you define a pointer to pointers to char.
Allocate only as many characters for each line as you need. This is done most conveniently with the function strdup(). If your library doesn't have it (it is not standard) you can replace it with the right combination of strlen(), malloc(), and strcpy(). How to do this is left as an exercise for you. ;-)
Handle allocation errors, especially if you plan to read huge files.
Free the allocated memories blocks, the sequence for the lines is not important. But lines has to be kept until all lines[*] are freed.
This is the code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main()
{
FILE *file;
file = fopen("test.txt", "r");
char buffer[600];
char **lines = NULL;
int i = 0;
while (fgets(buffer, sizeof(buffer), file))
{
lines = realloc(lines, (i + 1) * sizeof (char*));
if (lines == NULL)
{
// any error handling you like...
return EXIT_FAILURE;
}
lines[i] = strdup(buffer);
if (lines[i] == NULL)
{
// any error handling you like...
return EXIT_FAILURE;
}
i++;
}
fclose(file);
// work with the lines
for (int j = 0; j < i; ++j)
{
free(lines[j]);
}
free(lines);
return EXIT_SUCCESS;
}
Some notes:
Because of the realloc() on each line the run time of you program will scale bad for files with a giant number of lines. To improve this you might like to use some better algorithm, for example by allocating in steps of growing numbers. But this is a completely different issue.
You don't need to free allocated memory yourself at all if you need the memory until the end of the program. The C runtime will then free the memory automatically.