Do I need to free the strtok resulting string? - c

Question

Or rather, how does strtok produce the string to which it's return value points? Does it allocate memory dynamically? I am asking because I am not sure if I need to free the token in the following code:
The STANDARD_INPUT variables is for exit procedure in case I run out of memory for allocation and the string is the tested subject.
int ValidTotal(STANDARD_INPUT, char *str)
{
char *cutout = NULL, *temp, delim = '#';
int i = 0; //Checks the number of ladders in a string, 3 is the required number
temp = (char*)calloc(strlen(str),sizeof(char));
if(NULL == temp)
Pexit(STANDARD_C); //Exit function, frees the memory given in STANDARD_INPUT(STANDARD_C is defined as the names given in STANDARD_INPUT)
strcpy(temp,str);//Do not want to touch the actual string, so copying it
cutout = strtok(temp,&delim);//Here is the lynchpin -
while(NULL != cutout)
{
if(cutout[strlen(cutout) - 1] == '_')
cutout[strlen(cutout) - 1] = '\0'; \\cutout the _ at the end of a token
if(Valid(cutout,i++) == INVALID) //Checks validity for substring, INVALID is -1
return INVALID;
cutout = strtok(NULL,&delim);
strcpy(cutout,cutout + 1); //cutout the _ at the beginning of a token
}
free(temp);
return VALID; // VALID is 1
}

Answer 1

strtok manipulates the string you pass in and returns a pointer to it,
so no memory is allocated.
Please consider using strsep or at least strtok_r to save you some headaches later.

Answer 2

The first parameter to the strtok(...) function is YOUR string:
str
C string to truncate. Notice that this string is modified by
being broken into smaller strings (tokens). Alternativelly, a null
pointer may be specified, in which case the function continues
scanning where a previous successful call to the function ended.
It puts '\0' characters into YOUR string and returns them as terminated strings. Yes, it mangles your original string. If you need it later, make a copy.
Further, it should not be a constant string (e.g. char* myStr = "constant string";). See here.
It could be allocated locally or by malloc/calloc.
If you allocated it locally on the stack (e.g. char myStr[100];), you don't have to free it.
If you allocated it by malloc (e.g. char* myStr = malloc(100*sizeof(char));), you need to free it.
Some example code:
#include <string.h>
#include <stdio.h>
int main()
{
const char str[80] = "This is an example string.";
const char s[2] = " ";
char *token;
/* get the first token */
token = strtok(str, s);
/* walk through other tokens */
while( token != NULL )
{
printf( " %s\n", token );
token = strtok(NULL, s);
}
return(0);
}
NOTE: This example shows how you iterate through the string...since your original string was mangled, strtok(...) remembers where you were last time and keeps working through the string.

Answer 3

According to the docs:
Return Value
A pointer to the last token found in string.
Since the return pointer just points to one of the bytes in your input string where the token starts, whether you need to free depends on whether you allocated the input string or not.

Answer 4

As others mentioned, strtok uses its first parameter, your input string, as the memory buffer. It doesn't allocate anything. It's stateful and non-thread safe; if strtok's first argument is null, it reuses the previously-provided buffer. During a call, strtok destroys the string, adding nulls into it and returning pointers to the tokens.
Here's an example:
#include <stdio.h>
#include <string.h>
int main() {
char s[] = "foo;bar;baz";
char *foo = strtok(s, ";");
char *bar = strtok(NULL, ";");
char *baz = strtok(NULL, ";");
printf("%s %s %s\n", foo, bar, baz); // => foo bar baz
printf("original: %s\n", s); // => original: foo
printf("%ssame memory loc\n", s == foo ? "" : "not "); // => same memory loc
return 0;
}
s started out as foo;bar;baz\0. Three calls to strtok turned it into foo\0bar\0baz\0. s is basically the same as the first chunk, foo.
Valgrind:
==89== HEAP SUMMARY:
==89== in use at exit: 0 bytes in 0 blocks
==89== total heap usage: 1 allocs, 1 frees, 1,024 bytes allocated
==89==
==89== All heap blocks were freed -- no leaks are possible
While the code below doesn't fix all of the problems with strtok, it might help get you moving in a pinch, preserving the original string with strdup:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main() {
const char s[] = "foo;bar_baz";
const char delims[] = ";_";
char *cpy = strdup(s);
char *foo = strtok(cpy, delims);
char *bar = strtok(NULL, delims);
char *baz = strtok(NULL, delims);
printf("%s %s %s\n", foo, bar, baz); // => foo bar baz
printf("original: %s\n", s); // => original: foo;bar_baz
printf("%ssame memory loc\n", s == foo ? "" : "not "); // => not same memory loc
free(cpy);
return 0;
}
Or a more full-fledged example (still not thread safe):
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
void must(
bool predicate,
const char *msg,
const char *file,
const unsigned int line
) {
if (!predicate) {
fprintf(stderr, "%s:%d: %s\n", file, line, msg);
exit(1);
}
}
size_t split(
char ***tokens,
const size_t len,
const char *s,
const char *delims
) {
char temp[len+1];
temp[0] = '\0';
strcpy(temp, s);
*tokens = malloc(sizeof(**tokens) * 1);
must(*tokens, "malloc failed", __FILE__, __LINE__);
size_t chunks = 0;
for (;;) {
char *p = strtok(chunks == 0 ? temp : NULL, delims);
if (!p) {
break;
}
size_t sz = sizeof(**tokens) * (chunks + 1);
*tokens = realloc(*tokens, sz);
must(*tokens, "realloc failed", __FILE__, __LINE__);
(*tokens)[chunks++] = strdup(p);
}
return chunks;
}
int main() {
const char s[] = "foo;;bar_baz";
char **tokens;
size_t len = split(&tokens, strlen(s), s, ";_");
for (size_t i = 0; i < len; i++) {
printf("%s ", tokens[i]);
free(tokens[i]);
}
puts("");
free(tokens);
return 0;
}