Tokenize String by using pointer - c

Question

I'm trying to tokenize a sting and here is my attempt.
char new_str[1024];
void tokenize_init(const char str[]){//copy the string into global section
strcpy(new_str,str);
}
int i = 0;
char *tokenize_next() {
const int len = strlen(new_str);
for(; i <= len; i++) {
if ( i == len) {
return NULL;
}
if ((new_str[i] >= 'a' && new_str[i] <= 'z') ||
(new_str[i] >= 'A' && new_str[i] <= 'Z')) {
continue;
}else {
new_str[i] = '\0';
i = i + 1;
return new_str;
}
}
return NULL;
}
//main function
int main(void) {
char sentence[] = "This is a good-sentence for_testing 1 neat function.";
printf("%s\n", sentence);
tokenize_init(sentence);
for (char *nt = tokenize_next();
nt != NULL;
nt = tokenize_next())
printf("%s\n",nt);
}
However, it just print out the first word of the sentence(which is "This") and then stop. Can someone tell me why? My guess is my new_str is not persisent and when the main function recall tokenize_next() the new_str become just the first word of the sentence. Thanks in advance.

Answer 1

The reason that it only prints out "This" is because you iterate to the first non-letter character which happens to be a space, and you replace this with a null terminating character at this line:
new_str[i] = '\0';
After that, it doesn't matter what you do to the rest of the string, it will only print up to that point. The next time tokenize_next is called the length of the string is no longer what you think it is because it is only counting the word "This" and since "i" has already reached that amount the function returns and so does every successive call to it:
if ( i == len)
{
return NULL;
}
To fix the function you would need to somehow update your pointer to look past that character on the next iteration.
However, this is quite kludgy. You are much better off using one of the mentioned functions such as strtok or strsep
UPDATE:
If you cannot use those functions then a redesign of your function would be ideal, however, per your request, try the following modifications:
#include <string.h>
#include <cstdio>
char new_str[1024];
char* str_accessor;
void tokenize_init(const char str[]){//copy the string into global section
strcpy(new_str,str);
str_accessor = new_str;
}
int i = 0;
char* tokenize_next(void) {
const int len = strlen(str_accessor);
for(i = 0; i <= len; i++) {
if ( i == len) {
return NULL;
}
if ((str_accessor[i] >= 'a' && str_accessor[i] <= 'z') ||
(str_accessor[i] >= 'A' && str_accessor[i] <= 'Z')) {
continue;
}
else {
str_accessor[i] = '\0';
char* output = str_accessor;
str_accessor = str_accessor + i + 1;
if (strlen(output) <= 0)
{
str_accessor++;
continue;
}
return output;
}
}
return NULL;
}
//main function
int main(void) {
char sentence[] = "This is a good-sentence for_testing 1 neater function.";
printf("%s\n", sentence);
tokenize_init(sentence);
for (char *nt = tokenize_next(); nt != NULL; nt = tokenize_next())
printf("%s\n",nt);
}