invalid pointer when using strtok_r - c

Question

When running my code (shown in the first code block), I get this error:
*** Error in `./a.out': free(): invalid pointer: 0x0000000001e4c016 ***
I found a fix (which is shown in the second code block), but I don't understand why the error is happening in the first place.
I read the documentation regarding strtok_r, but I don't understand why assigning "str" to a new char* fixes the problem.
Doesn't "rest = str" mean that rest and str point to the same block of memory. How does this fix the problem???
Broken code:
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
int main()
{
char* str = (char*) malloc(sizeof(char) * 128);
char* token;
printf("Enter something: ");
fgets(str, 128, stdin);
while ((token = strtok_r(str, " ", &str))) {
printf("%s\n", token);
}
free(str);
return (0);
}
Fixed code:
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
int main()
{
char* str = (char*) malloc(sizeof(char) * 128);
char* token;
char* rest = str;
printf("Enter something: ");
fgets(str, 128, stdin);
while ((token = strtok_r(rest, " ", &rest))) {
printf("%s\n", token);
}
free(str);
return (0);
}

Answer 1

It looks evidently that a call of strtok_r changes the pointer str that is passed to the call by reference as the third parameter.
while ((token = strtok_r(str, " ", &str))) {
^^^^
printf("%s\n", token);
}
So after a call of the function the pointer str can point inside the original string. So it will not store the value that it had after a call of malloc.
Thus using the auxiliary variable rest allows to keep the initial value in the pointer str.
Pay attention to that you are calling the function incorrectly. Here is its description
On the first call to strtok_r(), str should point to the string to be
parsed, and the value of saveptr is ignored. In subsequent calls, str
should be NULL, and saveptr should be unchanged since the previous
call.
So for the second and subsequent calls of the function the first argument shall be NULL.

Answer 2

You should write:
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
int main()
{
char str[128];
char *token;
char *rest = str;
printf("Enter something: ");
fgets(str, sizeof str, stdin);
for (token = strtok_r(rest, " ", &rest);
token = strtok_r(NULL, " ", &rest);
/* just nothing here */)
{
printf("%s\n", token);
}
return (0);
}
First, you don't need to allocate the memory for str, as you can define a local array to store the data. You can use the sizeof operator so you don't run the risk of not updating it in two places if you decide to change the size of str. In the case of using malloc you had better #define a constant to hold the value while you use the constant everywhere you are using the size of the allocated buffer.
Second, never cast the returned value of malloc. Believe me, it is a very bad habit. When you do a cast, you tell the compiler you know what you are doing. Casting the value of malloc is a legacy from when there was no void type in C (this is so far as the middle eighties). Once upon a time, malloc() used to return a char * which normally was not the type of pointer you wanted, and you had to cast the pointer to match the one your were using. Casting malloc() return value in 2021 is not only not recommended, but it is strongly discouraged, as many errors come from having cast it (the compiler warns you when you are doing something bad, but it will not, if you cast the value, normally that is interpreted as you telling the compiler you are doing something weird on purpose, so the compiler shuts up, and doesn't say more)
Third, if you are going to extract all the tokens in a string, the first time you need to call strtok() (or his friend strtok_w) with a first parameter pointing to the start of the string, but the rest of the calls have to be done with NULL as it first parameter, or you'll be searching inside the string just returned, and not behind the first occurrence. Your problem was not about using strtok or strtok_r, as strtok_r is just a reentrant version of strtok that allows you to start a nested loop, inside the first, or to call it from different threads.

Answer 3

Heap memory management keeps track of base memory addresses for implementing library calls. We need to preserve those base-addresses to free/reallocate whenever necessary.
Now that you found a way to use strtok_r(), I prefer the below version:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main () {
char orgStr [] = "strtok does not allow you to have 2 pointers going at once on the same string";
for (char *token, *rmdStr = orgStr; token = strtok_r (NULL, " ", &rmdStr); /* empty */) {
printf ("%s\n", token);
}
/* Original string is chopped up with NULCHAR, now unreliable */
}