C read big file into char* array too slow - c

Question

I'd like to read a big file while the first character of a line isn't " ".
But the code I have written is very slow. How can I speed up the routine?
Is there a better solution instead of getline?
void readString(const char *fn)
{
FILE *fp;
char *vString;
struct stat fdstat;
int stat_res;
stat_res = stat(fn, &fdstat);
fp = fopen(fn, "r+b");
if (fp && !stat_res)
{
vString = (char *)calloc(fdstat.st_size + 1, sizeof(char));
int dataEnd = 1;
size_t len = 0;
int emptyLine = 1;
char **linePtr = malloc(sizeof(char*));
*linePtr = NULL;
while(dataEnd)
{
// Check every line
getline(linePtr, &len, fp);
// When data ends, the line begins with space (" ")
if(*linePtr[0] == 0x20)
emptyLine = 0;
// If line begins with space, stop writing
if(emptyLine)
strcat(vString, *linePtr);
else
dataEnd = 0;
}
strcat(vString, "\0");
free(linePtr);
linePtr = NULL;
}
}
int main(int argc, char **argv){
readString(argv[1]);
return EXIT_SUCCESS;
}

Answer 1

How can I speed up the routine?
The most suspicious aspect of your program performance-wise is the strcat(). On each call, it needs to scan the whole destination string from the beginning to find the place to append the source string. As a result, if your file's lines have length bounded by a constant (even a large one), then your approach's performance scales with the square of the file length.
The asymptotic complexity analysis doesn't necessarily tell the whole story, though. The I/O part of your code scales linearly with file length, and since I/O is much more expensive than in-memory data manipulation, that will dominate your performance for small enough files. If you're in that regime then you're probably not going to do much better than you already do. In that event, though, you might still do a bit better by reading the whole file at once via fread(), and then scanning it for end-of-data via strstr():
size_t nread = fread(vString, 1, fdstat.st_size, fp);
// Handle nread != fdstat.st_size ...
// terminate the buffer as a string
vString[nread] = '\0';
// truncate the string after the end-of-data:
char *eod = strstr(vString, "\n ");
if (eod) {
// terminator found - truncate the string after the newline
eod[1] = '\0';
} // else no terminator found
That scales linearly, so it addresses your asymptotic complexity problem, too, but if the data of interest will often be much shorter than the file, then it will leave you in those cases doing a lot more costly I/O than you need to do. In that event, one alternative would be to read in chunks, as #laissez_faire suggested. Another would be to tweak your original algorithm to track the end of vString so as to use strcpy() instead of strcat() to append each new line. The key part of that version would look something like this:
char *linePtr = NULL;
size_t nread = 0;
size_t len = 0;
*vString = '\0'; // In case the first line is end-of-data
for (char *end = vString; ; end += nread) {
// Check every line
nread = getline(&linePtr, &len, fp);
if (nread < 0) {
// handle eof or error ...
}
// When data ends, the line begins with space (" ")
if (*linePtr == ' ') {
break;
}
strcpy(end, *linePtr);
}
free(linePtr);
Additionally, note that
you do not need to initially zero-fill the memory allocated for *vString, as you're just going to overwrite those zeroes with the data of real interest (and then ignore the rest of the buffer).
You should not cast the return value of malloc()-family functions, including calloc().

Answer 2

Have you tried to read the file using fread and read a bigger chunk of data in each step and then parse the data after reading it? Something like:
#include <stdio.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <stdlib.h>
char *readString(const char *fn)
{
FILE *fp;
char *vString;
struct stat fdstat;
int stat_res;
stat_res = stat(fn, &fdstat);
fp = fopen(fn, "r+b");
if (fp && !stat_res) {
vString = (char *) calloc(fdstat.st_size + 1, sizeof(char));
int newline = 1;
int index = 0;
while (index < fdstat.st_size) {
int len =
fdstat.st_size - index >
4096 ? 4096 : fdstat.st_size - index;
char *buffer = (char *) malloc(len);
int read_len = fread(buffer, 1, len, fp);
int i;
if (newline) {
if (read_len > 0 && buffer[0] == ' ') {
return vString;
}
newline = 0;
}
for (i = 0; i < read_len; ++i) {
if (buffer[i] == '\n') {
if (i + 1 < read_len && buffer[i + 1] == ' ') {
memcpy(vString + index, buffer, i + 1);
return vString;
}
newline = 1;
}
}
memcpy(vString + index, buffer, read_len);
index += read_len;
}
}
return vString;
}
int main(int argc, char **argv)
{
char *str = readString(argv[1]);
printf("%s", str);
free(str);
return EXIT_SUCCESS;
}