I typed up this block of code for an assignment:
char *tokens[10];
void parse(char* input);
void main(void)
{
char input[] = "Parse this please.";
parse(input);
for(int i = 2; i >= 0; i--) {
printf("%s ", tokens[i]);
}
}
void parse(char* input)
{
int i = 0;
tokens[i] = strtok(input, " ");
while(tokens[i] != NULL) {
i++;
tokens[i] = strtok(NULL, " ");
}
}
But, looking at it, I'm not sure how the memory allocation works. I didn't define the length of the individual strings as far as I know, just how many strings are in the string array tokens (10). Do I have this backwards? If not, then is the compiler allocating the length of each string dynamically? In need of some clarification.
strtok is a bad citizen.
For one thing, it retains state, as you've implicitly used when you call strtok(NULL,...) -- this state is stored in the private memory of the Standard C Library, which means only single threaded programs can use strtok. Note that there is a reentrant version called strtok_r in some libraries.
For another, and to answer your question, strtok modifies its input. It doesn't allocate space for the strings; it writes NUL characters in place of your delimiter in the input string, and returns a pointer into the input string.
You are correct that strtok can return more than 10 results. You should check for that in your code so you don't write beyond the end of tokens. A reliable program would either set an upper limit, like your 10, and check for it, reporting an error if it's exceeded, or dynamically allocate the tokens array with malloc, and realloc it if it gets too big. Then the error occurs when you fun out of memory.
Note that you can also work around the problem of strtok modifying your input string by strduping before passing it to strtok. Then you'll have to free the new string after both it and tokens, which points to it, are going out of scope.
tokens is an array of pointers.
The distinction between strings and pointers if often fuzzy. In some situations strings are better thought out as arrays, in other situations as pointers.
Anyway... in your example input is an array and tokens is an array of pointers to a place within input.
The data inside input is changed with each call to strtok()
So, step by step
// input[] = "foo bar baz";
tokens[0] = strtok(input, " ");
// input[] = "foo\0bar baz";
// ^-- tokens[0] points here
tokens[1] = strtok(NULL, " ");
// input[] = "foo\0bar\0baz";
// ^-- tokens[1] points here
tokens[2] = strtok(NULL, " ");
// input[] = "foo\0bar\0baz";
// ^-- tokens[2] points here
// next strtok returns NULL
i Have the following code
char inputs []="3,0,23.30,3,30/55,55,55,55,55,55,55,55,55,55,55,55,55,64,64,64,100,100,100,100,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,55,55,70/1.5,0.5,0.2,0.2,0.3,0.1";
char parameters[18];
strcpy(parameters,strtok(inputs,"/"));
and then some code to transmit my characters through uart and see them at a monitor. when i transmit the inputs i see them fine but when i transmit the parameters i see nothing the string is empty.
i have seen examples for strtok and it uses that code to split strings. I have also tried this kind of code at visual studio and when i print them it shows me the strings fine. Is there any chance that strtok doesn't function well with a microprocessor????
While working with microcontrollers, you have to take care from which memory area you are working on. On software running on a PC, everything is stored and run from the RAM. But, on a flash microcontroller, code is run from flash (also called program memory) while data are processed from RAM (also called data memory).
In the case you are working on, the inputs variable is storing an hardcoded character array, which can be const, and we don't know in which area the compiler chose to put it. So, we could rewrite you small program just to make sure that all the data are stored in program data and we will use the "_P" functions to manipulate this data.
#include <avr/pgmspace.h > // to play in program space
const char inputs PROGMEM []="3,0,23.30,3,30/55,55,55,55,55,55,55,55,55,55,55,55,55,64,64,64,100,100,100,100,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,55,55,70/1.5,0.5,0.2,0.2,0.3,0.1"; // Now, we are sure this is in program memory space
char buffer[200]; // should be long enough to contain a copy of inputs
char parameters[18];
int length = strlen_P(inputs); // Should contains string length of 186 (just to debug)
strcpy_P(buffer,inputs); // Copy the PROGMEM data in data memory space
strcpy(parameters,strtok_P(buffer,"/")); // Parse the data now in data memory space
For more info on program space with avr gcc : http://www.nongnu.org/avr-libc/user-manual/pgmspace.html
I don't use strtok much, if at all, but this appears to be the correct way to store the result of strtok() in an char[]:
const int NUM_PARAMS = 18;
const int MAX_CHARS = 64;
char parameters[NUM_PARAMS][MAX_CHARS];
char delims[] = "/";
char *result = NULL;
int count = 0;
result = strtok(inputs, delims);
while(result != NULL && count < NUM_PARAMS){
strncpy(parameters[count++], result, MAX_CHARS);
result = strtok(NULL, delims);
}
or this if you don't want to allocate unnecessary memory for smaller tokens:
const int NUM_PARAMS = 18;
char* parameters[NUM_PARAMS];
char delims[] = "/";
char *result = NULL;
int count = 0;
result = strtok(inputs, delims);
while(result != NULL && count < NUM_PARAMS){
parameters[count] = malloc(strlen(result) + 1);
strncpy(parameters[count++], result, MAX_CHARS);
result = strtok(NULL, delims);
}
parameters should now contain all of your tokens.
strtok() intentionally modifies the source string, replacing tokens with terminators as it goes. There is no reason to store the content in auxiliary storage whatsoever so long as the source string is mutable.
Splitting the string into conjoined constants (which will be assembled by the compiler, so they ultimately will be a single terminated string):
char inputs []="3,0,23.30,3,30/"
"55,55,55,55,55,55,55,55,55,55,55,55,55,64,64,64,100,100,100,100,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,64,55,55,70/"
"1.5,0.5,0.2,0.2,0.3,0.1";
Clearly the second and third sequences delimited by '/' are nowhere near 17 chars wide (remember, your copy needs one more place for the terminator, thus only 17 chars can legally be copied).
Unless there is a compelling reason to do otherwise, I see no reason you can't simply do this:
char *param = strtok(inputs, "/");
while (param != NULL)
{
// TODO: do something with param...
// ... then advance to next param
param = strtok(NULL, "/");
}
What you do with // TODO do something with param... is up to you. The length of the parameter can be retrieved by strlen(param), for example. You can make a copy, providing you have enough storage space as the destination (in the second and third cases, you don't provide enough storage with only an 18-char buffer in your example).
Regardless, remember, strtok() modifies the source inputs[] array. If that is not acceptable an alternative would be something like:
char *tmp_inputs = strdup(inputs);
if (tmp_inputs != NULL)
{
char *param = strtok(tmp_inputs, "/");
while (param != NULL)
{
// TODO: do something with param...
// ... then advance to next param
param = strtok(NULL, "/");
}
// done with copy of inputs. free it.
free(tmp_inputs);
}
There are considerably threading decisions to make as well, the very reason the version of strtok() that requires the caller (you) to tote around context between calls was invented. See strtok_r() for more information.
You have to initialialize parameters that way :
char parameters[18] = {'\0'};
Thus, it will be initialized with null characters instead of variable values. This is important, because strcpy will try to find a null character to identify the end of the string.
Just to make it clear...
parameters after its initialisation : [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
parameters after strtok : ['3',',','0',',','2','3','.','3','0',',','3',',','3','0',0,0,0,0]
So I'm new to C and the whole string manipulation thing, but I can't seem to get strtok() to work. It seems everywhere everyone has the same template for strtok being:
char* tok = strtok(source,delim);
do
{
{code}
tok=strtok(NULL,delim);
}while(tok!=NULL);
So I try to do this with the delimiter being the space key, and it seems that strtok() no only reads NULL after the first run (the first entry into the while/do-while) no matter how big the string, but it also seems to wreck the source, turning the source string into the same thing as tok.
Here is a snippet of my code:
char* str;
scanf("%ms",&str);
char* copy = malloc(sizeof(str));
strcpy(copy,str);
char* tok = strtok(copy," ");
if(strcasecmp(tok,"insert"))
{
printf(str);
printf(copy);
printf(tok);
}
Then, here is some output for the input "insert a b c d e f g"
aaabbbcccdddeeefffggg
"Insert" seems to disappear completely, which I think is the fault of strcasecmp(). Also, I would like to note that I realize strcasecmp() seems to all-lower-case my source string, and I do not mind. Anyhoo, input "insert insert insert" yields absolutely nothing in output. It's as if those functions just eat up the word "insert" no matter how many times it is present. I may* end up just using some of the C functions that read the string char by char but I would like to avoid this if possible. Thanks a million guys, i appreciate the help.
With the second snippet of code you have five problems: The first is that your format for the scanf function is non-standard, what's the 'm' supposed to do? (See e.g. here for a good reference of the standard function.)
The second problem is that you use the address-of operator on a pointer, which means that you pass a pointer to a pointer to a char (e.g. char**) to the scanf function. As you know, the scanf function want its arguments as pointers, but since strings (either in pointer to character form, or array form) already are pointer you don't have to use the address-of operator for string arguments.
The third problem, once you fix the previous problem, is that the pointer str is uninitialized. You have to remember that uninitialized local variables are truly uninitialized, and their values are indeterminate. In reality, it means that their values will be seemingly random. So str will point to some "random" memory.
The fourth problem is with the malloc call, where you use the sizeof operator on a pointer. This will return the size of the pointer and not what it points to.
The fifth problem, is that when you do strtok on the pointer copy the contents of the memory pointed to by copy is uninitialized. You allocate memory for it (typically 4 or 8 bytes depending on you're on a 32 or 64 bit platform, see the fourth problem) but you never initialize it.
So, five problems in only four lines of code. That's pretty good! ;)
It looks like you're trying to print space delimited tokens following the word "insert" 3 times. Does this do what you want?
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
int main(int argc, char **argv)
{
char str[BUFSIZ] = {0};
char *copy;
char *tok;
int i;
// safely read a string and chop off any trailing newline
if(fgets(str, sizeof(str), stdin)) {
int n = strlen(str);
if(n && str[n-1] == '\n')
str[n-1] = '\0';
}
// copy the string so we can trash it with strtok
copy = strdup(str);
// look for the first space-delimited token
tok = strtok(copy, " ");
// check that we found a token and that it is equal to "insert"
if(tok && strcasecmp(tok, "insert") == 0) {
// iterate over all remaining space-delimited tokens
while((tok = strtok(NULL, " "))) {
// print the token 3 times
for(i = 0; i < 3; i++) {
fputs(tok, stdout);
}
}
putchar('\n');
}
free(copy);
return 0;
}
I am using strtok to extract 2 words from a string names[result]. I want to get the first value from the strtok and stored it into a char array named lastName and the second value into a char array named firstName. However I got an invalid initializer error for 2 lines which are indicated by the arrow when I compiled my code. How do I resolve my problem?
char *p = NULL;
p = strtok(names[result]," ");
char lastName[50] = p; <---
p = strtok(NULL, " ");
char firstName[50] = p; <---
printf("%s %s\n",firstName,lastName);
strtok gives the pointer to the tokenized string.
char lastName[50] = p; Isn't really a good thing that you are doing there. Should use strncpy() to copy the string, or if only want the pointer, then should store in another pointer.
Array initialization in C can only use a literal, not a variable. So your code is a syntax error.
You need to use the typical strcpy() function to copy the string, or some of the more safe (and modern) varities, like strlcpy() or snprintf().
You could also do the parsing and copying in one call, using sscanf(), with proper size specifiers in the formatting string to avoid the risk of buffer overflow.
You can initialize a string to the character array like char lastName[50] = "Sample";
In this case you are trying to initialize a pointer to the character array 'char lastName[50] = p;' which is not valid.
Better you can use strcpy, memcpy function to copy the string to the character array or you can assign it in another pointer.
The other answers are all correct in that copying the string data out will make this program work, but the reason strtok is so dastardly (and generally using it is considered ill-advised) is that it changes your input by inserting NULLs into the original string. If you're going to be using it anyway, you might as well advantage of this and just use the pointers that strtok is returning directly.
Of note, though, is that since the input is changed and maybe whoever passed that input into you is not expecting that, it might be better to copy the input to a separate string first before ever calling strtok on it.
Observe the output of this code to see what I mean:
int main(int argc, char *argv[]) {
char name[] = "Firstname Lastname";
printf("Name before strtok: %s\n", name);
char *first = strtok(name, " ");
char *last = strtok(NULL, " ");
printf("Token strings: first=%s last=%s\n", first, last);
printf("Name after strtok: %s\n", name);
}
Produces:
Firstname Name before strtok: Firstname Lastname
Token strings: first=Firstname last=Firstname
Name after strtok: Firstname
I'm writing a function that gets the path environment variable of a system, splits up each path, then concats on some other extra characters onto the end of each path.
Everything works fine until I use the strcat() function (see code below).
char* prependPath( char* exeName )
{
char* path = getenv("PATH");
char* pathDeepCopy = (char *)malloc(strlen(path) + 1);
char* token[80];
int j, i=0; // used to iterate through array
strcpy(pathDeepCopy, path);
//parse and split
token[0] = strtok(pathDeepCopy, ":"); //get pointer to first token found and store in 0
//place in array
while(token[i]!= NULL) { //ensure a pointer was found
i++;
token[i] = strtok(NULL, ":"); //continue to tokenize the string
}
for(j = 0; j <= i-1; j++) {
strcat(token[j], "/");
//strcat(token[j], exeName);
printf("%s\n", token[j]); //print out all of the tokens
}
}
My shell output is like this (I'm concatenating "/which" onto everything):
...
/usr/local/applic/Maple/bin/which
which/which
/usr/local/applic/opnet/8.1.A.wdmguru/sys/unix/bin/which
which/which
Bus error (core dumped)
I'm wondering why strcat is displaying a new line and then repeating which/which.
I'm also wondering about the Bus error (core dumped) at the end.
Has anyone seen this before when using strcat()?
And if so, anyone know how to fix it?
Thanks
strtok() does not give you a new string.
It mutilates the input string by inserting the char '\0' where the split character was.
So your use of strcat(token[j],"/") will put the '/' character where the '\0' was.
Also the last token will start appending 'which' past the end of your allocated memory into uncharted memory.
You can use strtok() to split a string into chunks. But if you want to append anything onto a token you need to make a copy of the token otherwise what your appending will spill over onto the next token.
Also you need to take more care with your memory allocation you are leaking memory all over the place :-)
PS. If you must use C-Strings. use strdup() to copy the string.
char* prependPath( char* exeName )
{
char* path = getenv("PATH");
char* pathDeepCopy = strdup(path);
char* token[80];
int j, i; // used to iterate through array
token[0] = strtok(pathDeepCopy, ":");
for(i = 0;(token[i] != NULL) && (i < 80);++i)
{
token[i] = strtok(NULL, ":");
}
for(j = 0; j <= i; ++j)
{
char* tmp = (char*)malloc(strlen(token[j]) + 1 + strlen(exeName) + 1);
strcpy(tmp,token[j]);
strcat(tmp,"/");
strcat(tmp,exeName);
printf("%s\n",tmp); //print out all of the tokens
free(tmp);
}
free(pathDeepCopy);
}
strtok does not duplicate the token but instead just points to it within the string. So when you cat '/' onto the end of a token, you're writing a '\0' either over the start of the next token, or past the end of the buffer.
Also note that even if strtok did returning copies of the tokens instead of the originals (which it doesn't), it wouldn't allocate the additional space for you to append characters so it'd still be a buffer overrun bug.
strtok() tokenizes in place. When you start appending characters to the tokens, you're overwriting the next token's data.
Also, in general it's not safe to simply concatenate to an existing string unless you know that the size of the buffer the string is in is large enough to hold the resulting string. This is a major cause of bugs in C programs (including the dreaded buffer overflow security bugs).
So even if strtok() returned brand-new strings unrelated to your original string (which it doesn't), you'd still be overrunning the string buffers when you concatenated to them.
Some safer alternatives to strcpy()/strcat() that you might want to look into (you may need to track down implementations for some of these - they're not all standard):
strncpy() - includes the target buffer size to avoid overruns. Has the drawback of not always terminating the result string
strncat()
strlcpy() - similar to strncpy(), but intended to be simpler to use and more robust (http://en.wikipedia.org/wiki/Strlcat)
strlcat()
strcpy_s() - Microsoft variants of these functions
strncat_s()
And the API you should strive to use if you can use C++: the std::string class. If you use the C++ std::string class, you pretty much do not have to worry about the buffer containing the string - the class manages all of that for you.
OK, first of all, be careful. You are losing memory.
Strtok() returns a pointer to the next token and you are storing it in an array of chars.
Instead of char token[80] it should be char *token.
Be careful also when using strtok. strtok practically destroys the char array called pathDeepCopy because it will replace every occurrence of ":" with '\0'.As Mike F told you above.
Be sure to initialize pathDeppCopy using memset of calloc.
So when you are coding token[i] there is no way of knowing what is being point at.
And as token has no data valid in it, it is likely to throw a core dump because you are trying to concat. a string to another that has no valida data (token).
Perphaps th thing you are looking for is and array of pointers to char in which to store all the pointer to the token that strtok is returnin in which case, token will be like char *token[];
Hope this helps a bit.
If you're using C++, consider boost::tokenizer as discussed over here.
If you're stuck in C, consider using strtok_r because it's re-entrant and thread-safe. Not that you need it in this specific case, but it's a good habit to establish.
Oh, and use strdup to create your duplicate string in one step.
replace that with
strcpy(pathDeepCopy, path);
//parse and split
token[0] = strtok(pathDeepCopy, ":");//get pointer to first token found and store in 0
//place in array
while(token[i]!= NULL) { //ensure a pointer was found
i++;
token[i] = strtok(NULL, ":"); //continue to tokenize the string
}
// use new array for storing the new tokens
// pardon my C lang skills. IT's been a "while" since I wrote device drivers in C.
const int I = i;
const int MAX_SIZE = MAX_PATH;
char ** newTokens = new char [MAX_PATH][I];
for (int k = 0; k < i; ++k) {
sprintf(newTokens[k], "%s%c", token[j], '/');
printf("%s\n", newtoken[j]); //print out all of the tokens
}
this will replace overwriting the contents and prevent the core dump.
and don't forget to check if malloc returns NULL!