splitting string and counting tokens in c - c

Question

I have a text file that contains multiple strings that are different lengths that I need to split into tokens.
Is it best to use strtok to split these strings and how can I count the tokens?
Example of strings from the file
Emma Stone#1169876#COMP242#COMP333#COMP336#COMP133#COMP231
Emma Watson#1169875#COMP336#COMP2421#COMP231#COMP338#CCOMP3351
Kevin Hart#1146542#COMP142#COMP242#COMP231#COMP336#COMP331#COMP334
George Clooney#1164561#COMP336#COMP2421#COMP231#COMP338#CCOMP3351
Matt Damon#1118764#COMP439#COMP4232#COMP422#COMP311#COMP338
Johnny Depp#1019876#COMP311#COMP242#COMP233#COMP3431#COMP333#COMP432

Answer 1

Generally, using strtok is a good solution to the problem:
#include <stdio.h>
#include <string.h>
int main( void )
{
char line[] =
"Emma Stone#1169876#COMP242#COMP333#COMP336#COMP133#COMP231";
char *p;
int num_tokens = 0;
p = strtok( line, "#" );
while ( p != NULL )
{
num_tokens++;
printf( "Token #%d: %s\n", num_tokens, p );
p = strtok( NULL, "#" );
}
}
This program has the following output:
Token #1: Emma Stone
Token #2: 1169876
Token #3: COMP242
Token #4: COMP333
Token #5: COMP336
Token #6: COMP133
Token #7: COMP231
However, one disadvantage of using strtok is that it is destructive in the sense that it modifies the string, by replacing the # delimiters with terminating null characters. If you do not want this, then you can use strchr instead:
#include <stdio.h>
#include <string.h>
int main( void )
{
const char *const line =
"Emma Stone#1169876#COMP242#COMP333#COMP336#COMP133#COMP231";
const char *p = line, *q;
int num_tokens = 1;
while ( ( q = strchr( p, '#' ) ) != NULL )
{
printf( "Token #%d: %.*s\n", num_tokens, q-p, p );
num_tokens++;
p = q + 1;
}
printf( "Token #%d: %s\n", num_tokens, p );
}
This program has identical output to the first program:
Token #1: Emma Stone
Token #2: 1169876
Token #3: COMP242
Token #4: COMP333
Token #5: COMP336
Token #6: COMP133
Token #7: COMP231
Another disadvantage with strtok is that it is not reentrant or thread-safe, whereas strchr is. However, some platforms provide a function strtok_r, which does not have these disadvantages. But that function does still has the disadvantage of being destructive.

Answer 2

Yes, you should use strtok to split these strings.
On
how can I count the tokens
You can simply add a counter inside while and increment it by one in each iteration to get the total number of tokens.
#include <stdio.h>
#include <string.h>
int main(void) {
char string[] = "Hello world this is a simple string";
char *token = strtok(string, " ");
int count = 0;
while (token != NULL) {
count++;
token = strtok(NULL, " ");
}
printf("Total number of tokens = %d", count);
return 0;
}

Answer 3

strtok() is rarely the right tool for anything. In this case, it is unclear whether a sequence of ## is equivalent to a single # and whether a # appearing at the beginning or end of line is to be ignored...
strtok() makes strong assumptions for these cases that may not be the expected behavior.
Furthermore, strtok() modifies its string argument and uses a hidden static state that makes it unsafe in multithreaded programs and prone to programming errors in nested use cases. strtok_r(), where available, solves these issues but the semantics are still somewhat counterintuitive.
For your purpose, you must define precisely what is a token and a separator. If empty tokens are allowed, strtok() is definitely not a solution.

Answer 4

You can also write your own function to handle this quite trivial split:
char **split(char *str, char **argv, size_t *argc, const char delim)
{
*argc = 0;
if(*str && *str)
{
argv[0] = str;
*argc = 1;
while(*str)
{
if(*str == delim)
{
*str = 0;
str++;
if(*str)
{
argv[*argc] = str;
*argc += 1;
continue;
}
}
str++;
}
}
return argv;
}
int main(void)
{
char *argv[10];
size_t argc;
char str[] = "Emma Stone#1169876#COMP242#COMP333#COMP336#COMP133#COMP231";
split(str, argv, &argc, '#');
printf("Numner of substrings: %zu\n", argc);
for(size_t i = 0; i < argc; i++)
printf("token [%2zu] = `%s`\n", i, argv[i]);
}
https://godbolt.org/z/b1aarnfWs
Remarks: same as strtok it requires str to me modifiable. str will be modified.