Obtaining zero-length string from strtok() - c

Question

I have a CSV file containing data such as
value;name;test;etc
which I'm trying to split by using strtok(string, ";"). However, this file can contain zero-length data, like this:
value;;test;etc
which strtok() skips. Is there a way I can avoid strtok from skipping zero-length data like this?

Answer 1

A possible alternative is to use the BSD function strsep() instead of strtok(), if available.
From the man page:
The strsep() function is intended as a replacement for the strtok()
function. While the strtok() function should be preferred for
portability reasons (it conforms to ISO/IEC 9899:1990 ("ISO C90"))
it is unable to handle empty fields, i.e., detect fields delimited by
two adjacent delimiter characters, or to be used for more than a
single string at a time. The strsep() function first appeared in
4.4BSD.
A simple example (also copied from that man page):
char *token, *string, *tofree;
tofree = string = strdup("value;;test;etc");
while ((token = strsep(&string, ";")) != NULL)
printf("token=%s\n", token);
free(tofree);
Output:
token=value
token=
token=test
token=etc
so empty fields are handled correctly.
Of course, as others already said, none of these simple tokenizer functions handles
delimiter inside quotation marks correctly, so if that is an issue, you should use
a proper CSV parsing library.

Answer 2

There is no way to make strtok() not behave this way. From man page:
A sequence of two or more contiguous delimiter bytes in the parsed
string is considered to be a single delimiter. Delimiter bytes at the
start or end of the string are ignored. Put another way: the tokens
returned by strtok() are always nonempty strings.
But what you can do is check the amount of '\0' characters before the token, since strtok() replaces all encountered tokens with '\0'. That way you'll know how many tokens were skipped. Source info:
This end of the token is automatically replaced by a null-character,
and the beginning of the token is returned by the function.
And a code sample to show what I mean.
char* aStr = ...;
char* ptr = NULL;
ptr = strtok (...);
char* back = ptr;
int count = -1;
do {
back--;
if (back <= aStr) break; // to protect against reads before aStr
count++;
} while (*back = '\0');
(written without ide or testing, may be an invalid implementation, but the idea stands).

Answer 3

No you can't.
From "man strtok":
A sequence of two or more contiguous delimiter characters in the
parsed string is considered to be a single delimiter. Delimiter
characters at the start or end of the string are ignored. Put
another way: the tokens returned by strtok() are always nonempty
strings.
You could also run into problems if your data contains the delimiter inside quotes or any other "escape".
I think the best solution is to get a CSV parsing library or write your own parsing function.

Answer 4

From recent experience, it looks like strtok() does not necessarily replace all delimiters with the end of string characters, but rather replaces the first delimiter it finds with an end of string character and skips the following delimiters but leaves them in place.
This means that in the nominal case (no zero-length strings before delimiters), every call to strtok() after the first call to strtok() will return a pointer to a string that begins after a \0 character.
In the case where strtok() reads zero-length strings between delimiters, strtok() will return a pointer to a string that begins after a delimiter character that has not been replaced with \0.
Here is my solution for finding out whether strtok() has skipped a zero-length string between delimiters.
// Previous code is needed to point strtok to a string and start ingesting from it.
char * field_string = strtok(NULL, ',');
// Note that this can't be done after the first call to strtok for a given buffer, since the previous character would be outside of the string's memory space.
if (*(field_string-1) == '\0') {
// no delimiters were skipped
} else {
// one or more delimiters were skipped
}