CSV file reading issue: with large amont of data - c

Question

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
char* getfield(char* line, int num) {
char* tok = line;
char* result;
if (line)
{
do
{
if (!--num)
{
tok = strchr(line, ',');
if (tok == NULL)
{
tok = &line[strlen(line)];
}
size_t fieldlen = tok - line;
if (fieldlen)
{
result = (char*)malloc(fieldlen+1);
result[fieldlen] = '\0';
strncpy(result, line, fieldlen);
return result;
}
else
{
break;
}
}
tok = strchr(line, ',');
line = tok + 1;
} while (tok);
}
result = (char*)malloc(2);
strcpy(result, "0");
return result;
}
int main()
{
FILE* stream = fopen("data.csv", "r");
char line[1024];
char *pstr;int num1,num2,num3;
char* value1,value2,value3;
while (fgets(line, 1024, stream))
{
char* tmp = strdup(line);
value1=getfield(tmp, 1);
value2=getfield(tmp, 2);
value3=getfield(tmp, 3);
num1 =strtol(value1,&pstr,10);
num2 =strtol(value2,&pstr,10);
num3 =strtol(value3,&pstr,10)
free(value1);
free(value2);
free(value3);
printf("Fields 1,2,3 would be 1=%d 2=%d 3=%d\n", num1,num2,num3);
// NOTE strtok clobbers tmp
free(tmp);
}
}
above is my C code to read the file....
:::: data.csv ::::
10,34,30
10,33,
23,45,23
25,,45
above is my file..
here my issue is I can call the function with "num" field. so that for reading of every line I suppose to call the function 3 times.. !! so the performance is too low for the large data files.. can someone help me that I can call the function at once and It will return an array.. than I can easily store and print (e.g. for the first line array[0]=10,array[1]=34,array[2]=30 )

Answer 1

You could speed it up by creating a fast split function that will destroy your line (not to mention the many lurking segmentation faults and memory leaks; this code has NO error checking or freeing of resources):
#include <stdio.h>
#include <stdlib.h>
char **split(char *line, char sep, int fields) {
char **r = (char **)malloc(fields * sizeof(char*));
int lptr = 0, fptr = 0;
r[fptr++] = line;
while (line[lptr]) {
if (line[lptr] == sep) {
line[lptr] = '\0';
r[fptr] = &(line[lptr+1]);
fptr++;
}
lptr++;
}
return r;
}
int main(int argc, char **argv) {
char line[] = "some,info,in a line";
char **fields = split(line, ',', 3);
printf("0:%s 1:%s 2:%s\n", fields[0], fields[1], fields[2]);
}
result:
0:some 1:info 2:in a line

Answer 2

I haven't run timing test on your code, but I'll bet a nickel that the problems is using malloc(). That is SLOW.
What Bart means is that a char[] array can contain multiple strings, back-to-back. If you scan through the array as a single string once, changing all ',' characters to '\0', your last line would look like:
{ '2', '5', 0, 0, '4', '5', 0, ? rest of buffer }
^ ^ ^ !
The ^ carets below mark the positions where you'd record pointers to three strings. As you can see, they are equivalent to separate strings of "25", "", "45" in separate arrays. The ! below marks the 0 that ended the original string. Nothing beyond that has any meaning.
All this depends on being able to modify the original string in-place, probably rendering it useless for any further processing (like printing out the offending line if an invalid field is detected). However, you are already copying the original buffer for local use, so that shouldn't be a problem. I'd get rid of the malloc for that copy buffer too, by the way.
Code might look like:
while (fgets(line, 1024, stream))
{
char tmp[sizeof line]; /* this will save a malloc()/free() pair */
char *tok, *fence, *pstr;
char ch, *cp1=line, *cp2=tmp;
while (0 != (ch = *cp1++))
*cp2++ = (ch == ',') ? 0 : ch;
fence = cp2; /* remember end of string */
*fence = 0; /* and terminate final string */
tok = tmp; /* point to first token */
num1 =strtol(tok, &pstr, 10);
if (tok < fence) tok += strlen(tok) + 1;
num2 =strtol(tok,&pstr,10);
if (tok < fence) tok += strlen(tok) + 1;
num3 =strtol(tok,&pstr,10);
printf("Fields 1,2,3 would be 1=%d 2=%d 3=%d\n", num1,num2,num3);
}
Obviously you don't need a 1K buffer to handle three values, so there will be a loop to pull out the values. The if statement after the first two strtol() calls is your replacement for getfield(), which isn't needed any more.
After this is working, look at data validation. Nothing in this (or in the original) will detect invalid numbers.