Fastest algorithm for splitting a string in C? - c

Question

I'm writing a CLI input parser and I want to know what is the fastest algorithm to split a string into tokens.
Rules:
A space means end of the token.
Any char can be escaped with a backslash which means that I take it as is without any specal meaning. (currently only used for escaping a space)
Here is the code I am currently using:
#define POINTER_ARRAY_STEP 10
#define STRING_STEP 255
char **parse_line(char *line)
{
char **array;
size_t array_len;
size_t array_index;
size_t token_len;
size_t token_index;
array_len = POINTER_ARRAY_STEP;
array = malloc((array_len + 1) * sizeof(char*)); /* +1 to leave space for NULL */
array_index = 0;
token_len = STRING_STEP;
array[array_index] = malloc(token_len + 1);
token_index = 0;
for (; *line; ++line) {
if (array_index == array_len) {
array_len += POINTER_ARRAY_STEP;
array = realloc(array, (array_len + 1) * sizeof(char*));
}
if (token_index == token_len) {
token_len += STRING_STEP;
array[array_index] = realloc(array[array_index], token_len + 1);
}
if (*line == '\\') {
array[array_index][token_index++] = *++line;
continue;
}
if (*line == ' ') {
array[array_index][token_index] = 0;
array[++array_index] = malloc(token_len + 1);
token_index = 0;
continue;
}
array[array_index][token_index++] = *line;
}
/* Null terminate the last array element */
array[array_index][token_index] = 0;
/* Null terminate the array */
array[array_index + 1] = NULL;
return array;
}

Answer 1

Your method is both unsafe and inefficient: you do not check for memory allocation failure and you call realloc() way too many times.
Here is another approach:
make a first pass to count the number of tokens and escapes,
allocate the pointer array and a buffer for the tokens
make a second pass, copying the characters into the buffer, splitting the tokens and making the pointer array point to the tokens.
return the array pointer.
The memory can later be freed by calling free() on both the pointer array and its first element.
Here is the code:
#include <stdlib.h>
char **parse_line(const char *line) {
size_t len, i, j, k, items = 1, escapes = 0;
char **array;
char *buf;
for (len = 0; line[len]; len++) {
if (line[len] == '\\') {
escapes++;
if (line[++len] == '\0')
break;
} else
if (line[len] == ' ') {
items++;
}
}
if (len == escapes) {
/* special case empty line */
array = malloc(sizeof(*array));
if (array != NULL) {
array[0] = NULL;
}
return array;
}
array = malloc((items + 1) * sizeof(*array));
buf = malloc(len + 1 - escapes);
if (array == NULL || buf == NULL) {
free(array);
free(buf);
return NULL;
}
items[0] = buf;
k = 1;
for (i = j = 0; i < len; i++) {
if (line[i] == '\\') {
if (++i == len)
break;
buf[j++] = line[i];
} else
if (line[i] == ' ') {
buf[j++] = '\0';
items[k++] = buf + j;
} else {
buf[j++] = line[i];
}
}
buf[j] = '\0';
items[k] = NULL;
return items;
}

Answer 2

Suggested fastest algorithm
Make only 2 passes. Once to find token count and length of buffer needed.
2nd time to chop up duplicated line.
The array of tokens point to a single allocated memory, so when done, tokens[0] needs to be free'd as well as tokens.
As OP asked for a faster algorithm, below is $pseudo code$/code.
char **parse_line(const char *line) {
size_t token_count = 1;
const char *p = line;
$Let `p` each element in `line`$
$if `*p` is a separator, increment `token_count`
$else `*p` is an escape not followed by a \0$
$advance `p`$
}
}
$ `p`, at the end of the string, so allocate enough for a $
$ duplicate +1 based on the difference of `p` and the start. $
$ Check allocation success $
$ Copy into `char *line_buffer` $
// The token count is known, get enough pointers + 1 and check
char **tokens = malloc(sizeof *tokens * (token_count + 1));
// More complex code could perform only 1 allocation for `tokens` and `line_buffer`
$ Let `q` be first element in line_buffer
$ Let `d` be the same (destination pointer)
$ set begin_token flag $
size_t token_index = 0;
for (;;) {
$ if begin_token flag set $ {
$ clear begin_token $
tokens[token_index] = q;
d = q;
}
$ if `*q` separator (space or \0) $ {
*d = '\0';
token_index++;
$ if *q at end of string $ break;
$ set begin_token flag $
$else {
if `*q` is an escape not followed by a \0$
$advance q$
}
$copy *q to *d, advance both pointers.
}
}
$set last tokens[] to NULL$
return tokens;
}