Writing a get next token function - c

Question

Given a C-string: how would I be able to write a function that will get the next token in the string, and a function that will peek the next token and return that without using global variables?
What I'm trying to do is have a static variable that will hold the string, and when called, it would just increment a pointer, and it will reset that static variable throwing out the token that has been retrieved. The problem is: how would I be able to differentiate between the first call (when it will actually store the string) and the other calls, when I am just retrieving it?
Any thoughts on this?
EDIT:
Here's what I have now that "works" but I want to make sure that it should actually work and its not just a coincidence of a pointer being null:
char next_token(char *line) {
static char *p;
if (p == NULL)
p = line;
else {
char next_token = p[0];
p++;
return next_token;
}
}

Answer 1

The code in your edit is wrong. You are handling the NULL case incorrectly.
I initially answered in terms of emulating strtok which seemed to be what you wanted, but you have clarified that you want single characters.
The if-condition should be:
if (line != NULL) p = line;
And you presumably remove the else so that code executes every time... Unless you don't want a result on the first call (you should at least return a value though).
You call like this:
char token = next_token(line);
while( 0 != (token = next_token(NULL)) ) {
// etc
}

Answer 2

typedef struct {
char* raw;
// whatever you need to keep track
} parser_t
void parser_init(parser_t* p, char* s)
{
// init your parser
}
bool parser_get_token(parser_t* p, char* token)
{
// return the token in "token" or return a bool error ( or an enum of various errors)
}
bool parser_peek_token(parser_t* p, char* token)
{
// same deal, but don't update where you are...
}

Answer 3

You have a couple of choices. One would be to use an interface roughly like strtok does, where passing a non-null pointer initializes the static variable, and passing a null pointer retrieves a token. This, however, is fairly ugly, clumsy, error-prone, and problematic in the presence of multithreading.
Another possibility would be to use a file-level static variable with separate functions (both in that file) to initialize the static variable, and to retrieve the next token from the string. This is marginally cleaner, but still have most of the same problems.
A third would be to make it act (for one example) like a file -- the user calls parse_open (for example), passing in the string to parse. You return an opaque handle to them. They then pass that back to (say) get_token each time they want another token.

Answer 4

Basically, there are three ways of a function to pass information back to its caller:
via a global variable
via the return value
via a pointer argument
And, similarly there are ways for the function to maintain state between calls:
via a global or (function-)static variable
by supplying it as a function parameter and returning it after every call
via a pointer argument.
A nice coding convention for a lexer/tokeniser is to use the return value to communicate the number of characters consumed. (and maybe use an extra pointer variable to pass the parser state to and fro calls)
This is wakkerbot's parser:
STATIC size_t tokenize(char *string, int *sp);
Usage:
STATIC void make_words(char * src, struct sentence * target)
{
size_t len, pos, chunk;
STRING word ;
int state = 0; /* FIXME: this could be made static to allow for multi-line strings */
target->size = 0;
len = strlen(src);
if (!len) return;
for(pos=0; pos < len ; ) {
chunk = tokenize(src+pos, &state);
if (!chunk) { /* maybe we should reset state here ... */ pos++; }
if (chunk > STRLEN_MAX) {
warn( "Make_words", "Truncated too long string(%u) at %s\n", (unsigned) chunk, src+pos);
chunk = STRLEN_MAX;
}
word.length = chunk;
word.word = src+pos;
if (word_is_usable(word)) add_word_to_sentence(target, word);
if (pos+chunk >= len) break;
pos += chunk;
}
...
}