How does strtok() split the string into tokens in C? - c

Please explain to me the working of strtok() function. The manual says it breaks the string into tokens. I am unable to understand from the manual what it actually does.
I added watches on str and *pch to check its working when the first while loop occurred, the contents of str were only "this". How did the output shown below printed on the screen?
/* strtok example */
#include <stdio.h>
#include <string.h>
int main ()
{
char str[] ="- This, a sample string.";
char * pch;
printf ("Splitting string \"%s\" into tokens:\n",str);
pch = strtok (str," ,.-");
while (pch != NULL)
{
printf ("%s\n",pch);
pch = strtok (NULL, " ,.-");
}
return 0;
}
Output:
Splitting string "- This, a sample string." into tokens:
This
a
sample
string

the strtok runtime function works like this
the first time you call strtok you provide a string that you want to tokenize
char s[] = "this is a string";
in the above string space seems to be a good delimiter between words so lets use that:
char* p = strtok(s, " ");
what happens now is that 's' is searched until the space character is found, the first token is returned ('this') and p points to that token (string)
in order to get next token and to continue with the same string NULL is passed as first
argument since strtok maintains a static pointer to your previous passed string:
p = strtok(NULL," ");
p now points to 'is'
and so on until no more spaces can be found, then the last string is returned as the last token 'string'.
more conveniently you could write it like this instead to print out all tokens:
for (char *p = strtok(s," "); p != NULL; p = strtok(NULL, " "))
{
puts(p);
}
EDIT:
If you want to store the returned values from strtok you need to copy the token to another buffer e.g. strdup(p); since the original string (pointed to by the static pointer inside strtok) is modified between iterations in order to return the token.

strtok() divides the string into tokens. i.e. starting from any one of the delimiter to next one would be your one token. In your case, the starting token will be from "-" and end with next space " ". Then next token will start from " " and end with ",". Here you get "This" as output. Similarly the rest of the string gets split into tokens from space to space and finally ending the last token on "."

strtok maintains a static, internal reference pointing to the next available token in the string; if you pass it a NULL pointer, it will work from that internal reference.
This is the reason strtok isn't re-entrant; as soon as you pass it a new pointer, that old internal reference gets clobbered.

strtok doesn't change the parameter itself (str). It stores that pointer (in a local static variable). It can then change what that parameter points to in subsequent calls without having the parameter passed back. (And it can advance that pointer it has kept however it needs to perform its operations.)
From the POSIX strtok page:
This function uses static storage to keep track of the current string position between calls.
There is a thread-safe variant (strtok_r) that doesn't do this type of magic.

strtok will tokenize a string i.e. convert it into a series of substrings.
It does that by searching for delimiters that separate these tokens (or substrings). And you specify the delimiters. In your case, you want ' ' or ',' or '.' or '-' to be the delimiter.
The programming model to extract these tokens is that you hand strtok your main string and the set of delimiters. Then you call it repeatedly, and each time strtok will return the next token it finds. Till it reaches the end of the main string, when it returns a null. Another rule is that you pass the string in only the first time, and NULL for the subsequent times. This is a way to tell strtok if you are starting a new session of tokenizing with a new string, or you are retrieving tokens from a previous tokenizing session. Note that strtok remembers its state for the tokenizing session. And for this reason it is not reentrant or thread safe (you should be using strtok_r instead). Another thing to know is that it actually modifies the original string. It writes '\0' for teh delimiters that it finds.
One way to invoke strtok, succintly, is as follows:
char str[] = "this, is the string - I want to parse";
char delim[] = " ,-";
char* token;
for (token = strtok(str, delim); token; token = strtok(NULL, delim))
{
printf("token=%s\n", token);
}
Result:
this
is
the
string
I
want
to
parse

The first time you call it, you provide the string to tokenize to strtok. And then, to get the following tokens, you just give NULL to that function, as long as it returns a non NULL pointer.
The strtok function records the string you first provided when you call it. (Which is really dangerous for multi-thread applications)

strtok modifies its input string. It places null characters ('\0') in it so that it will return bits of the original string as tokens. In fact strtok does not allocate memory. You may understand it better if you draw the string as a sequence of boxes.

To understand how strtok() works, one first need to know what a static variable is. This link explains it quite well....
The key to the operation of strtok() is preserving the location of the last seperator between seccessive calls (that's why strtok() continues to parse the very original string that is passed to it when it is invoked with a null pointer in successive calls)..
Have a look at my own strtok() implementation, called zStrtok(), which has a sligtly different functionality than the one provided by strtok()
char *zStrtok(char *str, const char *delim) {
static char *static_str=0; /* var to store last address */
int index=0, strlength=0; /* integers for indexes */
int found = 0; /* check if delim is found */
/* delimiter cannot be NULL
* if no more char left, return NULL as well
*/
if (delim==0 || (str == 0 && static_str == 0))
return 0;
if (str == 0)
str = static_str;
/* get length of string */
while(str[strlength])
strlength++;
/* find the first occurance of delim */
for (index=0;index<strlength;index++)
if (str[index]==delim[0]) {
found=1;
break;
}
/* if delim is not contained in str, return str */
if (!found) {
static_str = 0;
return str;
}
/* check for consecutive delimiters
*if first char is delim, return delim
*/
if (str[0]==delim[0]) {
static_str = (str + 1);
return (char *)delim;
}
/* terminate the string
* this assignmetn requires char[], so str has to
* be char[] rather than *char
*/
str[index] = '\0';
/* save the rest of the string */
if ((str + index + 1)!=0)
static_str = (str + index + 1);
else
static_str = 0;
return str;
}
And here is an example usage
Example Usage
char str[] = "A,B,,,C";
printf("1 %s\n",zStrtok(s,","));
printf("2 %s\n",zStrtok(NULL,","));
printf("3 %s\n",zStrtok(NULL,","));
printf("4 %s\n",zStrtok(NULL,","));
printf("5 %s\n",zStrtok(NULL,","));
printf("6 %s\n",zStrtok(NULL,","));
Example Output
1 A
2 B
3 ,
4 ,
5 C
6 (null)
The code is from a string processing library I maintain on Github, called zString. Have a look at the code, or even contribute :)
https://github.com/fnoyanisi/zString

This is how i implemented strtok, Not that great but after working 2 hr on it finally got it worked. It does support multiple delimiters.
#include "stdafx.h"
#include <iostream>
using namespace std;
char* mystrtok(char str[],char filter[])
{
if(filter == NULL) {
return str;
}
static char *ptr = str;
static int flag = 0;
if(flag == 1) {
return NULL;
}
char* ptrReturn = ptr;
for(int j = 0; ptr != '\0'; j++) {
for(int i=0 ; filter[i] != '\0' ; i++) {
if(ptr[j] == '\0') {
flag = 1;
return ptrReturn;
}
if( ptr[j] == filter[i]) {
ptr[j] = '\0';
ptr+=j+1;
return ptrReturn;
}
}
}
return NULL;
}
int _tmain(int argc, _TCHAR* argv[])
{
char str[200] = "This,is my,string.test";
char *ppt = mystrtok(str,", .");
while(ppt != NULL ) {
cout<< ppt << endl;
ppt = mystrtok(NULL,", .");
}
return 0;
}

For those who are still having hard time understanding this strtok() function, take a look at this pythontutor example, it is a great tool to visualize your C (or C++, Python ...) code.
In case the link got broken, paste in:
#include <stdio.h>
#include <string.h>
int main()
{
char s[] = "Hello, my name is? Matthew! Hey.";
char* p;
for (char *p = strtok(s," ,?!."); p != NULL; p = strtok(NULL, " ,?!.")) {
puts(p);
}
return 0;
}
Credits go to Anders K.

Here is my implementation which uses hash table for the delimiter, which means it O(n) instead of O(n^2) (here is a link to the code):
#include<stdio.h>
#include<stdlib.h>
#include<string.h>
#define DICT_LEN 256
int *create_delim_dict(char *delim)
{
int *d = (int*)malloc(sizeof(int)*DICT_LEN);
memset((void*)d, 0, sizeof(int)*DICT_LEN);
int i;
for(i=0; i< strlen(delim); i++) {
d[delim[i]] = 1;
}
return d;
}
char *my_strtok(char *str, char *delim)
{
static char *last, *to_free;
int *deli_dict = create_delim_dict(delim);
if(!deli_dict) {
/*this check if we allocate and fail the second time with entering this function */
if(to_free) {
free(to_free);
}
return NULL;
}
if(str) {
last = (char*)malloc(strlen(str)+1);
if(!last) {
free(deli_dict);
return NULL;
}
to_free = last;
strcpy(last, str);
}
while(deli_dict[*last] && *last != '\0') {
last++;
}
str = last;
if(*last == '\0') {
free(deli_dict);
free(to_free);
deli_dict = NULL;
to_free = NULL;
return NULL;
}
while (*last != '\0' && !deli_dict[*last]) {
last++;
}
*last = '\0';
last++;
free(deli_dict);
return str;
}
int main()
{
char * str = "- This, a sample string.";
char *del = " ,.-";
char *s = my_strtok(str, del);
while(s) {
printf("%s\n", s);
s = my_strtok(NULL, del);
}
return 0;
}

strtok() stores the pointer in static variable where did you last time left off , so on its 2nd call , when we pass the null , strtok() gets the pointer from the static variable .
If you provide the same string name , it again starts from beginning.
Moreover strtok() is destructive i.e. it make changes to the orignal string. so make sure you always have a copy of orignal one.
One more problem of using strtok() is that as it stores the address in static variables , in multithreaded programming calling strtok() more than once will cause an error. For this use strtok_r().

strtok replaces the characters in the second argument with a NULL and a NULL character is also the end of a string.
http://www.cplusplus.com/reference/clibrary/cstring/strtok/

you can scan the char array looking for the token if you found it just print new line else print the char.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main()
{
char *s;
s = malloc(1024 * sizeof(char));
scanf("%[^\n]", s);
s = realloc(s, strlen(s) + 1);
int len = strlen(s);
char delim =' ';
for(int i = 0; i < len; i++) {
if(s[i] == delim) {
printf("\n");
}
else {
printf("%c", s[i]);
}
}
free(s);
return 0;
}

So, this is a code snippet to help better understand this topic.
Printing Tokens
Task: Given a sentence, s, print each word of the sentence in a new line.
char *s;
s = malloc(1024 * sizeof(char));
scanf("%[^\n]", s);
s = realloc(s, strlen(s) + 1);
//logic to print the tokens of the sentence.
for (char *p = strtok(s," "); p != NULL; p = strtok(NULL, " "))
{
printf("%s\n",p);
}
Input: How is that
Result:
How
is
that
Explanation: So here, "strtok()" function is used and it's iterated using for loop to print the tokens in separate lines.
The function will take parameters as 'string' and 'break-point' and break the string at those break-points and form tokens. Now, those tokens are stored in 'p' and are used further for printing.

strtok is replacing delimiter with'\0' NULL character in given string
CODE
#include<iostream>
#include<cstring>
int main()
{
char s[]="30/4/2021";
std::cout<<(void*)s<<"\n"; // 0x70fdf0
char *p1=(char*)0x70fdf0;
std::cout<<p1<<"\n";
char *p2=strtok(s,"/");
std::cout<<(void*)p2<<"\n";
std::cout<<p2<<"\n";
char *p3=(char*)0x70fdf0;
std::cout<<p3<<"\n";
for(int i=0;i<=9;i++)
{
std::cout<<*p1;
p1++;
}
}
OUTPUT
0x70fdf0 // 1. address of string s
30/4/2021 // 2. print string s through ptr p1
0x70fdf0 // 3. this address is return by strtok to ptr p2
30 // 4. print string which pointed by p2
30 // 5. again assign address of string s to ptr p3 try to print string
30 4/2021 // 6. print characters of string s one by one using loop
Before tokenizing the string
I assigned address of string s to some ptr(p1) and try to print string through that ptr and whole string is printed.
after tokenized
strtok return the address of string s to ptr(p2) but when I try to print string through ptr it only print "30" it did not print whole string. so it's sure that strtok is not just returning adress but it is placing '\0' character where delimiter is present.
cross check
1.
again I assign the address of string s to some ptr (p3) and try to print string it prints "30" as while tokenizing the string is updated with '\0' at delimiter.
2.
see printing string s character by character via loop the 1st delimiter is replaced by '\0' so it is printing blank space rather than ''

Related

Why does strtok() not tokenize the string a certain way?

I'm trying to tokenize a string using brackets [] as delimiters. I can tokenize a string exactly how I want it with one input, but it has an error other times. For example, I have a string with characters before the delimiter and it works fine, but if nothing is before the delimiter then I run into errors.
This one gives me an error. The token2 ends up being NULL and token is "name]" with the bracket still on there.
char name[] = "[name]";
char *token = strtok(name, "[");
char *token2 = strtok(NULL, "]");
Output:
token = name]
token2 = NULL
However, if I have the following, then it works just fine.
char line[] = "Hello [name]";
char *tok = strtok(line, "[");
char *tok2 = strtok(NULL, "]");
Output:
tok = Hello
tok2 = name
I don't understand what I'm doing wrong when the input is simply something like "[name]". I want just what's inside the brackets only.
Edit:
Thanks for the input, everyone. I found a solution to what I'm trying to do. Per #Ryan and #StoryTeller's advice, I first checked if the input began with [ and delimited with []. Here's what I tried and worked for the input:
char name[] = "[name]", *token = NULL, *token2 = NULL;
if (name[0] == '[')
{
token = strtok(name, "[]");
}
else
{
token = strtok(name, "[");
token2 = strtok(NULL, "]");
}
In short: the 2nd time you called strtok() in your first example is the same as calling it on an empty string and this is why you get NULL.
Each call to strtok gives you the token based on your chosen delimiter. In your 1st try:
char name[] = "[name]";
char *token = strtok(name, "[");
char *token2 = strtok(NULL, "]");
The delimiter you chose is "[" so the 1st call to strtok will get "name]", since this is the first token in the string (remember that the string starts with a delimiter). The second will get NULL, since "name]" was the end of your original string and invoking strotk() now is like invoking it on an empty string.
strtok() uses a static buffer that holds your original string and each invocation "uses" another part of that buffer. After your 1st call, the function "used" the entire buffer.
In your 2nd try:
char line[] = "Hello [name]";
char *tok = strtok(line, "[");
char *tok2 = strtok(NULL, "]");
You call strtok on a string with the delimiter in the middle of it, so you get a token AND you still have a string left in the static buffer used by the function. That enables the 2nd call of strtok() to return a valid token instead of NULL.
If you are simply trying to extract the contents between a single-pair of brackets [...], then strchr provides a bit more straight-forward way to accomplish the task. When you are calling strtok with a single-delimiter (e.g. '[' and then ']'), you are essentially doing what you would do with two successive calls to strchr with the characters being the same '[' and then ']'.
For example the following will parse the string given on the command line for the characters between brackets ("[some name]" by default if no argument is given) up to a maximum of MAXNM character (including the nul-terminating character):
#include <stdio.h>
#include <string.h>
#define MAXNM 128
int main (int argc, char **argv) {
char *s = argc > 1 ? argv[1] : "[some name]", /* input */
*p = s, /* pointer */
*ep, /* end pointer */
buf[MAXNM] = ""; /* buffer for result */
/* if starting and ending bracket are present in input */
if ((p = strchr (s, '[')) && (ep = strchr (p, ']'))) {
if (ep - p > MAXNM) { /* length + 1 > MAXNM ? */
fprintf (stderr, "error: result too long.\n");
return 1;
}
/* copy betweeen brackets to buf (+1 for char after `[`) */
strncpy (buf, p + 1, ep - p - 1); /* ep - p - 1 for length */
buf[ep - p - 1] = 0; /* nul terminate, also done via initialization */
printf ("name : '%s'\n", buf); /* output the name */
}
else
fprintf (stderr, "error: no enclosing brackets found in input.\n");
return 0;
}
note: the benefit of using strchr and strncpy for paring between fixed delimiters is you do not modify the original string (like strtok does). So this method is safe for use with string literals or other constant strings.
Example Use/Output
$ ./bin/brackets
name : 'some name'
$ ./bin/brackets "this [is the name] and more"
name : 'is the name'

C string nested splitting

I'm a beginner at C and I'm stuck on a simple problem. Here it goes:
I have a string formatted like this: "first1:second1\nsecond2\nfirst3:second3" ... and so on.
As you can see from the the example the first field is optional ([firstx:]secondx).
I need to get a resulting string which contains only the second field. Like this: "second1\nsecond2\nsecond3".
I did some research here on stack (string splitting in C) and I found that there are two main functions in C for string splitting: strtok (obsolete) and strsep.
I tried to write the code using both functions (plus strdup) without success. Most of the time I get some unpredictable result.
Better ideas?
Thanks in advance
EDIT:
This was my first try
int main(int argc, char** argv){
char * stri = "ciao:come\nva\nquialla:grande\n";
char * strcopy = strdup(stri); // since strsep and strtok both modify the input string
char * token;
while((token = strsep(&strcopy, "\n"))){
if(token[0] != '\0'){ // I don't want the last match of '\n'
char * sub_copy = strdup(token);
char * sub_token = strtok(sub_copy, ":");
sub_token = strtok(NULL, ":");
if(sub_token[0] != '\0'){
printf("%s\n", sub_token);
}
}
free(sub_copy);
}
free(strcopy);
}
Expected output: "come", "si", "grande"
Here's a solution with strcspn:
#include <stdio.h>
#include <string.h>
int main(void) {
const char *str = "ciao:come\nva\nquialla:grande\n";
const char *p = str;
while (*p) {
size_t n = strcspn(p, ":\n");
if (p[n] == ':') {
p += n + 1;
n = strcspn(p , "\n");
}
if (p[n] == '\n') {
n++;
}
fwrite(p, 1, n, stdout);
p += n;
}
return 0;
}
We compute the size of the initial segment not containing : or \n. If it's followed by a :, we skip over it and get the next segment that doesn't contain \n.
If it's followed by \n, we include the newline character in the segment. Then we just need to output the current segment and update p to continue processing the rest of the string in the same way.
We stop when *p is '\0', i.e. when the end of the string is reached.

Parse Tokens from a String using strtok()

char line[] = "COPY\tSTART\t0\tCOPY";
char *tmp;
tmp = strtok(line, "\t");
printf("%s", tmp);
This code's output is COPY. And when
char line[] = "\tSTART\t0\tCOPY";
Output is START.
But! I want to check there is nothing in front of string START.
That is I think \t is first delimiter so output of strtok(line, "\t") is NULL.
But real output is START.
Is there any misunderstanding? What can I do?
As per the man page of strtok() (emphasis mine)
A sequence of two or more contiguous delimiter bytes in the parsed string is considered to be a single delimiter. Delimiter bytes at the start or end of the string are ignored. Put another way: the tokens returned by strtok() are always nonempty strings.
So, what you're experiencing is the right behaviour of strtok().
OTOH, strtok() will return NULL if there is no more tokens, so as per you have expected, returning NULL for the initial delimiter will convey wrong message and it will be confusing. So, the bottom line is,
if a token is present
the tokens returned by strtok() are always nonempty strings.
if a token is not present
strtok() will return NULL.
Note: it is useful to mention that before using the retured token, always check for NULL.
What can I do?
Build your own function, not exactly how strtok works but you can get some idea:
#include <stdio.h>
#include <string.h>
char *scan(char **pp, char c)
{
char *s, *p;
p = strchr(*pp, c);
if (p) *p++ = '\0';
s = *pp;
*pp = p;
return s;
}
int main(void)
{
char line1[] = "COPY\tSTART\t0\tCOPY";
char line2[] = "\tSTART\t0\tCOPY";
char *p;
puts("Line 1");
p = line1;
while (p) {
printf("%s\n", scan(&p, '\t'));
}
puts("Line 2");
p = line2;
while (p) {
printf("%s\n", scan(&p, '\t'));
}
return 0;
}
Output:
Line 1
COPY
START
0
COPY
Line 2
START
0
COPY

c, delete words which contain digits from a string

I need to delete all words that contain digits from the string.
E.g. if input is abdgh 67fgh 32ghj hj dfg43 11 fg, output should be abdgh hj fg.
I thought of using while( text[i] != ' '), but I don't know how to continue it for the rest of the string (after the first whitespace).
I don't have any other idea, and couldn't find anything by googling. Please, help me!
Here, i gave it a try. Works just fine for me. I tried to explain the logic throughout the code via comments. Hope it helps.
#include <stdio.h>
#include <string.h>
int containsNum(char * str);
int main()
{
char str[] = "abdgh 67fgh 32ghj hj dfg43 11 fg"; // input string
char newstr[100] = ""; //new string to create with filtered data
char * pch; //temp string to use in strtok
printf("given string : %s\n",str );
pch = strtok (str," ");
while (pch != NULL)
{
if(!containsNum(pch))// creation of new string with strcat
{ // if the current word not contains any number
strcat(newstr,pch);
strcat(newstr," "); //adding a space between words for readability
}
pch = strtok (NULL, " ");
}
printf("modified string : %s\n", newstr );
return 0;
}
//function containsNum
//gets a string and checks if it has any numbers in it
//returns 1 if so , 0 otherwise
int containsNum(char * str)
{
int i,
size =strlen(str),
flag=0;
for(i=0; i<size ; ++i)
{
if((int)str[i] >=48 && (int)str[i] <=57 ){
flag =1;
break;
}
}
return flag;
}
Regards
Algorithm:
1-You will have to break your input string into smaller components which are also called as tokens. For example: for the string abdgh 67fgh 32ghj hj dfg43 11 fg the tokens could be abdgh, 67fgh, 32ghj, hj, dfg43, 11 and fg.
2- These smaller strings or tokens can be formed using the strtok function which is defined as
char * strtok ( char * str, const char * delimiters );. Thestr in the first argument is the input sting which in the code presented below is string1. The second argument called the delimiters is what actually defines when to divide the input string into smaller pieces(tokens).
For instance, a whitespace as a delimiter will divide the input string whenever a whitespace is encountered, which is how the string is being divided in the code.
3-Since, your program needs to delete those words in the input string which contain digits we can use the isdigit() function to check exactly that.
WORKING CODE:
#include <cstring>
#include <ctype.h>
#include<stdio.h>
int main ()
{
char output[100]="";
int counter;
int check=0; /* An integer variable which takes the value of "1" whenever a digit
is encountered in one of the smaller strings or tokens.
So, whenever check is 1 for any of the tokens that token is to be ignored, that is,
not shown in the output string.*/
char string1[] = "abdgh 67fgh 32ghj hj dfg43 11 fg";
char delimiters[] = " ";//A whitespace character functions as a delimiter in the program
char * token;//Tokens are the sub-strings or the smaller strings which are part of the input string.
token=strtok(string1,delimiters);/*The first strktok call forms the first token/substring which for the input
given would be abdgh*/
while(token!=NULL)/*For the last substring(token) the strtok function call will return a NULL pointer, which
also indicates the last of the tokens(substrings) that can be formed for a given input string.
The while loop finishes when the NULL pointer is encountered.*/
{
for(counter=0;counter<=strlen(token)-1;counter++)/*This for loop iterates through each token element.
Example: In case of abdgh, it will first check for 'a',
then 'b', then 'd' and so on..*/
{
if(isdigit((int)token[counter])>0)/*This is to check if a digit has been encountered inside a token(substring).
If a digit is encountered we make check equal to 1 and break our loop, as
then that token is to be ignored and there is no real need to iterate
through the rest of the elements of the token*/
{
check=1;
break;
}
}
if(check==1) /* Outside the for loop, if check is equal to one that means we have to ignore that token and
it is not to be made a part of the output string. So we just concatenate(join) an
empty string ( represented by " " )with the output string*/
{
strcat(output,"");
check=0;
}
else /*If a token does not contain any digit we simply make it a part of the output string
by concatenating(joining) it with the output string. We also add a space for clarity.*/
{
strcat(output,token);
strcat(output," ");
}
token = strtok( NULL, delimiters ); /*This line of code forms a new token(substring) every time it is executed
inside the while loop*/
}
printf( "Output string is:: %s\n", output ); //Prints the final result
return 0;
}
#include <stdio.h>
#include <ctype.h>
#include <stdbool.h>
char *filter(char *str){
char *p, *r;
p = r = str;
while(*r){
char *prefetch = r;
bool contain_digit = false;
while(!isspace(*prefetch) && *prefetch){
if(contain_digit)
++prefetch;
else if(isdigit(*prefetch++))
contain_digit = true;
}
if(contain_digit){
r = prefetch;
}else {
while(r < prefetch){
*p++ = *r++;
}
}
if(!*r)
break;
if(p[-1] == *r)
++r;
else
*p++ =*r++;
}
*p = '\0';
return str;
}
int main(void) {
char text[] = "abdgh 67fgh 32ghj hj dfg43 11 fg";
printf("%s\n", filter(text));//abdgh hj fg
return 0;
}

How do you split a string in C?

If I have a string like:
const char* mystr = "Test Test Bla Bla \n Bla Bla Test \n Test Test \n";
How would I use the newline character '\n', to split the string into an array of strings?
I'm trying to accomplish in C, the thing string.Split() does in C# or boost's string algorithm split does in C++ .
Try to use the strtok function. Be aware that it modifies the source memory so you can't use it with a string literal.
char *copy = strdup(mystr);
char *tok;
tok = strtok(copy, "\n");
/* Do something with tok. */
while (tok) {
tok = strtok(NULL, "\n");
/* ... */
}
free(copy);
The simplest way to split a string in C is to use strtok() however that comes along with an arm's length list of caveats on its usage:
It's destructive (destroys the input string), and you couldn't use it on the string you have above.
It's not reentrant (it keeps its state between calls, and you can only be using it to tokenize one string at a time... let alone if you wanted to use it with threads). Some systems provide a reentrant version, e.g. strtok_r(). Your example might be split up like:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main (void) {
char mystr[] = "Test Test Bla Bla \n Bla Bla Test \n Test Test \n";
char *word = strtok(mystr, " \n");
while (word) {
printf("word: %s\n", word);
word = strtok(NULL, " \n");
}
return 0;
}
Note the important change of your string declaration -- it's now an array and can be modified. It's possible to tokenize a string without destroying it, of course, but C does not provide a simple solution for doing so as part of the standard library.
Remember that C makes you do all the memory allocation by hand. Remember also that C doesn't really have strings, only arrays of characters. Also, string literals are immutable, so you're going to need to copy it. It will be easier to copy the whole thing first.
So, something like this (wholly untested):
char *copy = xstrdup(mystr);
char *p;
char **arry;
size_t count = 0;
size_t i;
for (p = copy; *p; p++)
if (*p == '\n')
count++;
arry = xmalloc((count + 1) * sizeof(char *));
i = 0;
p = copy;
arry[i] = p;
while (*p)
{
if (*p == '\n')
{
*p = '\0';
arry[i++] = p+1;
}
p++;
}
return arry; /* deallocating arry and arry[0] is
the responsibility of the caller */
In the above reactions, I see only while(){} loops, where IMHO for(){} loops are more compact.
cnicutar:
for(tok = strtok(copy, "\n");tok; tok = strtok(NULL, "\n") {
/* ... */
}
FatalError:
char *word;
for ( word = strtok(mystr, " \n");word; word = strtok(NULL, " \n") {
printf("word: %s\n", word);
}
Zack:
for (arry[i=0]=p=copy; *p ; p++)
{
if (*p == '\n')
{
*p = '\0';
arry[i++] = p+1;
}
}
[the clarity of this last example is disputable]
You can use below mentioned library. It has many other useful functions.
http://www.boost.org/doc/libs/1_48_0/libs/tokenizer/index.html
Or you can use strtok function.

Resources