Does strdup allocate another memory zone and create another pointer every time?
For example: does the following code result in a memory leak?
void x(char** d, char* s){
*d = strdup(s);
}
int main(){
char* test = NULL;
x(&test, "abcd");
x(&test, "etc");
return 0;
}
Yes, the program leaks memory because it allocates objects and then loses references to them.
The first time this happens is in the line:
x(&test, "etc");
The variable test holds the one and only copy of a pointer that was allocated in a previous call to x. The new call to x overwrites that pointer. At that point, the pointer leaks.
This is what it means to leak memory: to lose all references to an existing dynamically allocated piece of storage.*
The second leak occurs when the main function returns. At that point, the test variable is destroyed, and that variable holds the one and only copy of a pointer to a duplicate of the "etc" string.
Sometimes in C programs, we sometimes not care about leaks of this second type: memory that is not freed when the program terminates, but that is not allocated over and over again in a loop (so it doesn't cause a runaway memory growth problem).
If the program is ever integrated into another program (for instance as a shared library) where the original main function becomes a startup function that could be invoked repeatedly in the same program environment, both the leaks will turn into problems.
The POSIX strdup function behaves similarly to this:
char *strdup(const char *orig)
{
size_t bytes = strlen(orig) + 1;
char *copy = malloc(bytes);
if (copy != 0)
memcpy(copy, orig, bytes);
return copy;
}
Yes; it allocates new storage each time.
If you have a garbage collector (such as Boehm) in your C image, then it's possible that the leaked storage is recycled, and so strdup is able to re-use the same memory for the second allocation. (However, a garbage collector is not going to kick in after just one allocation, unless it is operated in a stress-test mode for flushing out bugs.)
Now if you want to actually reuse the memory with realloc, then you can change your x function along these lines:
#include <stdlib.h>
#include <string.h>
void *strealloc(char *origptr, char *strdata)
{
size_t nbytes = strlen(strdata) + 1;
char *newptr = (char *) realloc(origptr, nbytes); /* cast needed for C++ */
if (newptr)
memcpy(newptr, strdata, nbytes);
return newptr;
}
(By the way, external names starting with str are in an ISO C reserved namespace, but strealloc is too nice a name to refuse.)
Note that the interface is different. We do not pass in a pointer-to-pointer, but instead present a realloc-like interface. The caller can check the return value for null to detect an allocation error, without having the pointer inconveniently overwritten with null in that case.
The main function now looks like:
int main(void)
{
char *test = strealloc(NULL, "abcd");
test = strealloc(test, "etc");
free(test);
return 0;
}
Like before, there is no error checking. If the first strealloc were to fail, test is then null. That doesn't since it gets overwritten anyway, and the first argument of strealloc may be null.
Only one free is needed to plug the memory leak.
* It's possible to have a semantic memory leak with objects that the program hasn't lost a reference to. For instance, suppose that a program keeps adding information to a list which is never used for any purpose and just keeps growing.
Yes, it allocates memory and leaks if you don't free it. From the man page:
The strdup() function returns a pointer to a new string which is a duplicate of the string s. Memory for the new string is obtained with malloc(3), and can be freed with free(3).
new_s = strdup(s) is essentially equivalent to:
new_s = malloc(strlen(s)+1);
strcpy(new_s, s);
Consider the following definition for strdup:
#include <string.h>
char *strdup(const char *string);
strdup reserves storage space for a copy of string by calling malloc. The string argument to this function is expected to contain a null character (\0) marking the end of the string. Remember to free the storage reserved with the call to strdup.
You must free the string yourself.
Related
I have a value being returned from a function which is a pointer to a malloc'd address which stores a string value. That string value is what I want to return. But to prevent leaks I need to free that same address, which in turn makes me lose the value I want to return.
I tried creating a tmp pointer as work around but I assume I'm doing something very wrong.
Here's a little code snippet, where line is what needs to be free'd, but also its value needs to be returned.
char *get_next_line(int fd)
{
static char *storage;
char *buffer;
char *line;
char *tmp;
tmp = NULL;
line = NULL;
if (!fd || BUFFER_SIZE <= 0)
return (NULL);
read_to_buffer(fd, &buffer, &storage);
free(buffer);
tmp = line;
line = extract_line(storage); // extract_line() returns a malloc'd pointer.
free(&tmp);
return (line);
}
It is always a good (better) idea to provide a buffer, instead of allocating a buffer and returning it. If you write a function and that function returns an allocated buffer, you can't be sure, that the buffer will be freed.
Therefore, the function should accept a buffer (and its size) and use that buffer instead. The specification could allow to pass a NULL pointer to let the function allocate a buffer (which has to be freed by the user via a specified function, e.g. free).
e.g.
//instead
char* get_next_line(int fd);
//do
char* get_next_line(int fd, char *dst, int dst_len);
The second version also has the advantage to reuse a single buffer, instead of an allocation at each and every call.
"A clever person solves a problem. A wise person avoids it."
Albert Einstein
tmp = line;
line = extract_line(storage); // extract_line() returns a malloc'd pointer.
free(&tmp);
return (line);
In the above code, both tmp and line are pointing to the same memory. So when tmp is freed, essentially the memory pointed to by line is also freed. And any attempt to use tmp after line has been freed will lead to undefined behaviour.
From C11, Memory Management Functions:
free():
....Otherwise, if the argument does not match a pointer earlier returned
by a memory management function, or if the space has been deallocated
by a call to free or realloc, the behavior is undefined.
Re:
But to prevent leaks I need to free that same address, which in turn
makes me lose the value I want to return.
C does not allow you to use memory after it has been freed. There's no need to free() the memory in this function if you have to return it. It will only ever result in a memory leak if you forgot to free() the memory (in the calling function) once you're done with it.
where line is what needs to be free'd, but also its value needs to be
returned.
If the memory pointed to by line is required for further processing, then there's no need for it to be freed in the function get_next_line.
Aside: free(&tmp); should be free (tmp);. And the parentheses after the return statements are redundant.
#include <stdio.h>
#include <stdlib.h>
#include <stdbool.h>
#include <string.h>
char* test() {
char* s = "Hello World";
size_t len = strlen(s);
char* t = malloc(sizeof(char)*(len+1));
strcpy(t, s);
free(t);
return t;
};
int main(void) {
printf("%s\n", test());
return 0;
};
I would like to allocate and de-allocate memory inside the function. I tested this code and works, but I am wondering:
Why does this work?
Is it good practice to use the value of a freed pointer in main ?
Once you call free on a pointer, the memory it pointed to is no longer valid. Attempting to use a pointer to freed memory triggers undefined behavior. In this particular case it happened to work, but there's no guarantee of that.
If the function returns allocated memory, it is the responsibility of the caller to free it:
char* test() {
char* s = "Hello World";
size_t len = strlen(s);
char* t = malloc(sizeof(char)*(len+1));
strcpy(t, s);
return t;
};
int main(void) {
char *t = test();
printf("%s\n", t);
free(t);
return 0;
};
malloc reserves memory for use.
free releases that reservation. In general, it does not make the memory go away, it does not change the contents of that memory, and it does not alter the value of the pointer that held the address.
After free(t), the bytes of t still contain the same bit settings they did before the free. Then return t; returns those bits to the caller.
When main passes those bits to printf, printf uses them as the address to get the characters for %s. Since nothing has changed them, they are printed.
That is why you got the behavior you did with this program. However, none of it is guaranteed. Once free was called with t, the memory reservation was gone. Something else in your program could have used that memory. For example, printf might have allocated a buffer for its own internal use, and that could have used the same memory.
For the most part, malloc and free are just methods of coordinating use of memory, so that different parts of your program do not try to use the same memory at the same time for different purposes. When you only have one part of your program using allocated memory, there are no other parts of your program to interfere with that. So the lack of coordination did not cause your program to fail. If you had multiple routines in your program using allocated memory, then attempting to use memory after it has been released is more likely to encounter problems.
Additionally, once the memory has been freed, the compiler may treat a pointer to it as if it has no fixed value. The return t; statement is not required to return any particular value.
It doesn't matter where do you free() a pointer. Once it is free()d, the pointer is not deferrenciable anymore (neither inside nor ouside the function where it was free()d)
The purpose of free() is to return the memory allocated with malloc() so the semantics are that, once you have freed a chunk of memory, it is not anymore usable.
In C, all parameters are passed by value, so free() cannot change the value expression you passed to it, and this is the reason the pointer is not changed into an invalid pointer value (like NULL) but you are advised that no more uses of the pointer can be done without incurring in Undefined Behaviour.
There could be a solution in the design of free() and it is to pass the pointer variable that holds the pointer by address, and so free() would be able to turn the pointer into a NULL. But this not only takes more work to do, but free() doesn't know how many copies you have made of the value malloc() gave to you... so it is impossible to know how many references you have over there to be nullified. That approach makes it impossible to give free() the responsibility of nullifying the reference to the returned memory.
So, if you think that free doesn't turn the pointer into NULL and for some strange reason you can still use the memory returned, don't do it anymore, because you'll be making mistakes.
You are adviced! :)
I am trying to copy a char from a char type function to a var in the main function using strcpy, but the program always give me a random value as return and the program is finished. Is there some syntax error I'm missing in the code or maybe another way to do this copy?
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>
char text()
{
char palavra1[]= "tata";
return palavra1;
}
char main()
{
char saida[50];
strcpy(saida,text()); // this line is supposed to copy the string from text to the var saida
printf("%s", saida);
return 0;
}
In the function text(), palavra1 is stored on the stack in test()s stack frame, so when you return the string and go back to main's stack frame, the memory occupied by palavra1 is no longer kept by the compiler, and can and will be overwritten by calling other functions in main. To fix this, you have two options, either have the caller of text provide a pointer to store the string to, or have text malloc() some memory for the string and return that, relying on the calling function to free it when it is done. Personally, I'd recommend going with the first method as it means you don't have to worry about freeing the pointer when you're done, but it's really your choice.
Additionally, your function return types are both wrong. text() should return a char*, and you should hopefully be getting a warning about that from your compiler. Finally, main() specified in the ISO standard as int main(void) or int main(int argc, char** argv). It's generally bad practice to stray from either of these two signatures.
Each function can return its own type, but it cannot return a pointer to memory allocated within its function stack because the function stack is released for reuse on the function return and is no longer reachable. Any attempt to access a value whose memory is no longer reachable is Undefined Behavior.
You have two options, either (1) pass saida as a parameter to text, (e.g. char *text (char *s) { strcpy (s, "tata"); return s } (the function could be declared void at that point because the characters copied to s will be available back in the caller anyway -- and there would be no need for strcpy in main()), or (2) dynamically allocate memory to hold "tata" in text() and return a pointer to the new block of memory. Memory allocated with malloc, calloc, or realloc survives the function return as it is allocated on the heap and has program duration (or until it is freed).
(note: a 3rd option is to declare palavra1 as a static character array of sufficient size to hold "tata" -- which would also cause it to survive function return. However that practice should generally be avoided in favor of the first two options)
You could dynamically allocate in a simple fashion like:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
char *text(void)
{
char *p = "tata", /* delcare pointer to string literal "tata" */
*palavra1 = malloc (strlen (p) + 1); /* allocate storage */
if (palavra1 != NULL) /* validate malloc succeeded */
strcpy (palavra1, p); /* copy "tata" to palavra1 */
return palavra1;
}
int main(void)
{
char saida[50],
*p = text(); /* must preserve pointer to beginning of memory */
/* validate p not NULL and length < 50 */
if (p != NULL && strlen (p) < sizeof saida) {
strcpy (saida, p); /* copy to saida */
printf ("%s\n", saida); /* output */
free (p); /* free allocated memory */
}
return 0;
}
Example Use/Output
$ ./bin/allocinfn
tata
Memory Use/Error Check
In any code you write that dynamically allocates memory, you have 2 responsibilities regarding any block of memory allocated: (1) always preserve a pointer to the starting address for the block of memory so, (2) it can be freed when it is no longer needed.
It is imperative that you use a memory error checking program to insure you do not attempt to access memory or write beyond/outside the bounds of your allocated block, attempt to read or base a conditional jump on an uninitialized value, and finally, to confirm that you free all the memory you have allocated.
For Linux valgrind is the normal choice. There are similar memory checkers for every platform. They are all simple to use, just run your program through it.
$ valgrind ./bin/allocinfn
==13720== Memcheck, a memory error detector
==13720== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==13720== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==13720== Command: ./bin/allocinfn
==13720==
tata
==13720==
==13720== HEAP SUMMARY:
==13720== in use at exit: 0 bytes in 0 blocks
==13720== total heap usage: 1 allocs, 1 frees, 5 bytes allocated
==13720==
==13720== All heap blocks were freed -- no leaks are possible
==13720==
==13720== For counts of detected and suppressed errors, rerun with: -v
==13720== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
Always confirm that you have freed all memory you have allocated and that there are no memory errors.
(also note: The proper declarations for main are int main (void) and int main (int argc, char **argv) (which you will see written with the equivalent char *argv[]). note: main is a function of type int and it returns a value. See: C11 Standard §5.1.2.2.1 Program startup p1 (draft n1570). See also: See What should main() return in C and C++?)
Look things over and let me know if you have further questions.
Even if you fix the return type of text() to
char *text()
{
char palavra1[]= "tata";
return palavra1;
}
this is still wrong. The reason for that is that palavra1 is a local variable
in text() and it is only valid and accessible until text() returns. After that the
pointer to palavra1 points to an invalid location, using it to read/write
values will lead to undefined behaviour.
There are two approaches for returning strings from a function.
The function allocates memory with malloc and returns a pointer to the
allocated memory. The function calling text() will need to free the memory:
char *text(void)
{
char *ptr, palavra1[] = "tata";
ptr = malloc(strlen(palavra1) + 1);
if(ptr == NULL)
return NULL;
strcpy(ptr, palavra1);
return ptr;
}
int main(void)
{
char saida[50];
char *res = text();
if(res == NULL)
{
fprintf(stderr, "failed to allocate memory\n");
return 1;
}
strncpy(saida, res, sizeof saida);
saida[sizeof(saida) - 1] = 0; // making sure to get a valid string
puts(saida);
free(res);
return 0;
}
The second option is to pass a pointer to the function and let the function
write the content through the pointer:
int text(char *buffer, size_t length)
{
palavra1[] = "tata";
if(buffer == NULL)
return 0;
strncpy(buffer, palavra1, length);
buffer[length - 1] = 0; // making sure to get a valid string
return 1;
}
int main(void)
{
char saida[50];
text(saida, sizeof saida);
puts(saida);
return 0;
}
A third option (I know I said 2) would be declaring a palavra1 in text()
as static char[] = "tata";. For sake of completeness I've added this third
option, but I'd suggest that you don't use that, because it has other problems
that you don't want to deal with, for example when your functions need to be
called in a nested kind of way. So better ignore that until you really know and
understand how to deal with static variables.
In both cases I've used strncpy, because with it you can limit the number of
bytes written thus prevent a buffer overflow. However, strncpy might not
write the '\0'-terminating byte if the destination buffer is not large enough,
that's why you have to make sure that the strings is '\0'-terminated by
writing the 0 at the last place of the destination buffer.
Also, the correct way of declaring the main function are:
int main(void);
int main(int argc, char **argv);
int main(int argc, char *argv[]);
I've never seen a char main() before, that's the first time for me. That's
incorrect and you should change that.
palavral has automatic storage duration - meaning when the function ends it will be destroyed (more clearly the block on which it is declared ends but here the function body is the block where it is declared - so when the function ends it's lifetime will end, as the block ends). Now you returned pointer to the local variable - precisely pointing to the memory which will not be there once the function ends and tried to access it via using it on strcpy - this is undefined behavior.
In fact standard says it :-
From §6.2.4 you can see under undefined behavior
An object is referred to outside of its lifetime
This was the problem:- but there are couple of solutions. If you want to return some pointer then either you can dynamically allocate memory and return the address of the allocated chunk or make the array pass static or global.
Discussion of dynamic memory allocation:
You can create a duplicate copy of the string using malloc/calloc/realloc or strdup(which internally uses one of them) which has storage duration beyond the scope of it's declaration and return it. (One choice is dynamic memory allocation and using it for returning from the function).(The code for this is shown the later part of the answer)
Discussion of making the memory static or global.
Making it having a static storage duration is another choice. (Either make the array global or make it static in by explicitly using the static keyword.).
From §6.2.4p3 Under the point of static storage duration
... Its lifetime is the entire execution of the program and its stored value is initialized only once, prior to program startup.
What is the meaning of the last line? It's (the local variable's) lifetime is not going to end unless the program ends and that means we can return it from the function. (Yes this is a solution but you will not be seeing it - not a good way to do things).
Code for dynamic allocation
const char*p = "data";
char *s = malloc(strlen(p)+1);
if(!s){
perror("malloc");
exit(EXIT_FAILURE);
}
memcpy(s,p,strlen(p)+1);
return s;
And in main()
char *p = text();
if(p!=NULL)
strcpy(saida,p);
free(p);
Code for static storage duration examples:
In this case there is nothing much to show other than just make the pointer variable static.
static char palavar1[]="tata";
...
return palavar;
Without using all those difficult words - it can be said that simply now this local variable will stay alive even when the function ends. The previous problem discussed won't be a problem now. How long will it stay alive? Until the program ends. So you can safely use the above technique discarding the possibility of
accessing object outside it's lifetime.
Other solutions
Other solution will simply pass the buffer which will be modified and the string literals content will be copied to it using strcpy or memcpy. There is no need to return something. This solution I didn't discuss at the first edit as it is a bit different approach to achieving what you want to do.
Standard directs that return type of main to be int not char. Ideally in your case int main(void) would do.
You can make it static (and this is pointer of course):
static char palavra1[]= "tata";
I was a bit confused with the concept of char pointers so I made a simple code just printing my name provided by user (me). I also wanted to practice malloc so I referenced the pointer to a certain memory in RAM, but I really didn't know what to put after "sizeof(char) *" because that is the user input, which is not yet decided.
Also, after doing that, I freed the memory, but I got an error message on command line saying:
*** Error in `./char': double free or corruption (fasttop): 0x00000000017fe030 ***
Aborted
It seems like I freed the same memory twice or something, but I don't know what to delete or add. Please help!
#include <stdio.h>
#include <cs50.h>
int main (void)
{
char *strings = malloc(sizeof(char) * 10);
printf("What is your name?\n");
//wait for use to type his/her name
strings = get_string();
printf("Hello %s\n", strings);
free (strings);
return 0;
}
The line strings = get_string(); actually assigns the value returned by get_string() to strings. It doesn't write it into the memory you allocated.
So the value returne by malloc() has been overwritten (and lost in this case).
The free(strings) is releasing whatever get_string() returned. The question doesn't provide the code for that but presumably it isn't valid to free() it.
Because the run-time told you it was freed twice I'm guessing you have allocated memory in get_string() then freed it and returned an invalid pointer.
If you want to use the memory you allocated you need to change get_string() to accept a pointer:
void get_string(char *str){
//Do whatever writing you value into str[] as an array of char..
}
Good practice would have:
void get_string(char *str, size_t max){
//Do whatever writing you value into str[] as an array of char..
//Use max to avoid writing beyond the end of the space allocated...
}
Then call as get_string(strings,10);.
EDIT: After a bit of research the flaw has been identified. get_string() doesn't directly free() the string it returns but adds it to a list of allocations made by the library which are freed on exit (in a function called teardown() registered with atexit() or other compiler dependent features).
That is poor design because consumer code is provided no safe way of itself freeing the memory which in a typical use case will not be required for the whole application execution. get_double() is worse because it never returns the allocated data but never reuses it and amounts to a straight memory leak.
The code should either:
Conform to the documentation and require consumer code to free() the string (maybe rename it as say get_string_alloc() for clarity).
Offer a library routine to free the string (get_new_string() and release_string())
There is no very nice way to shift ownership of allocated memory in C but holding onto it for the remainder of execution is definitely not the answer.
Many libraries go round the houses to push allocation onto consumer code but that is onerous when the full size of the space required can't be known such as here.
I'd suggest putting _alloc() at the end of any function that returns objects that consumer code must later free().
So the answer for the question posed is remove the malloc() and the free() because the library handles both. However beware if your program makes many calls to that function and others that internally rely on it (like get_double()) you may run out of memory because the library is sitting on dead space.
The problem is your get_strings overrides your initial malloc. A pointer value is a value. By equating it with something else, you replaced your malloc value.
Memory is allocated at the statement:
strings = get_string();
You dont have to malloc it ( char *strings = malloc(sizeof(char) * 10);
)
Without malloc it will work fine
First You have created a dynamic memory which will be pointed by *strings. But then you are pointing to the local string (from get_string() function) using *strings pointer. when you call free, program is trying delete local (stack) reference and throwing error.
To solve that error, the program should be
#include <stdio.h>
#include <cs50.h>
int main (void)
{
char *strings = malloc(sizeof(char) * 10);
printf("What is your name?\n");
//wait for use to type his/her name
strcpy(strings, get_string()); // Use strcpy instead of assigning
printf("Hello %s\n", strings);
free (strings);
return 0;
}
You don't include the code for get_string(), but you're overwriting strings with its return value which is wrong. The address you pass to free() must come from malloc(), and it seems you're violating that (in addition to losing the original returned address for your 10 bytes).
Assuming get_string() returns static storage (i.e. you don't need to free it) you can do this without involving malloc().
If you really want to, something like this might work:
printf("What is your name?\n");
const char *name = get_string();
const size_t nlen = strlen(name);
char * const name_copy = malloc(nlen + 1);
if(name_copy != NULL)
{
memcpy(name_copy, name, nlen + 1);
printf("Hello %s (from my own memory!)\n", name_copy);
free(name_copy);
}
This is rather convoluted but you get the idea.
char *strings;
No need for new malloc as string returned from get_string() function is already on the heap, you just need to pick up pointer to first character. (get_string() function reference)
strings = get_string();
printf("Hello %s\n", strings);
After printing string you should free memory allocated for it, as it is stated in get_string() function reference
Stores string on heap (via malloc); memory must be freed by caller to
avoid leak.
I think everything else is fine, try this code:
#include <stdio.h>
#include <cs50.h>
int main (void)
{
char *strings;
printf("What is your name?\n");
//wait for use to type his/her name
strings = get_string();
printf("Hello %s\n", strings);
free (strings);
return 0;
}
struct TokenizerT_ {
char* separators;
char* tks;
char* cur_pos;
char* next;
};
typedef struct TokenizerT_ TokenizerT;
TokenizerT *TKCreate(char *separators, char *ts)
{
TokenizerT *tokenizer;
tokenizer = (TokenizerT*)malloc(sizeof(TokenizerT));
//some manipulation here
tokenizer->tks = (char*) malloc (strlen(str)* sizeof(char));
tokenizer->tks=str;
printf("size of tokenizer->tks is %zu\n", strlen(tokenizer->tks)); //this prints out the correct number (e.g. 7)
return tokenizer;
}
int main(int argc, char **argv)
{
TokenizerT *tk = TKCreate(argv[1], argv[2]);
printf("tk->tks: %zu\n", strlen(tk->tks)); //HOWEVER, this prints out the wrong number (e.g. 1)
}
As seen from the above code, I'm working with pointers to structs. For some reason I am not receiving back the correct length for tk->tks. I cannot understand this because it should be the same size as tks in my TKCreate function. Can someone explain this please?
I suspect str, the definition of which is not shown in your code snippet, is a local variable defined in TKCreate(). If so, you're assigning tokenizer->tks to have the value of str, which points to a proper string inside the scope of TKCreate() but upon exiting TKCreate(), the stack contents (including parameters and local variables) are freed and wiped out so when you try to reference that pointer outside the scope of TKCreate() all bets are off.
One plausible fix is to allocate the storage for tokenizer->tks dynamically, so it persists after you exit TKCreate(). I see you do that with a call to malloc but then you overwrite that with an explicit assignment from str. Instead you should copy the contents of str (using strcpy) into the dynamically allocated memory via: strcpy(tokenizer->tks, str);
You should strcpy the contents of str to tokenizer->tks, because when you use the assign operator, you're losing the pointer malloc gave you, creating a memory leak and pointing tokenizer->tks to a local variable, which will be destroyed after the function's return.
So, the approach would be something like this:
tokenizer->tks = (char *)malloc ((strlen(str) + 1) * sizeof(char));
strcpy(tokenizer->tks, str);
Another thing:
Don't forget to free ->tks before you free tk itself.
So, after the printf, you should use:
free(tk->tks);
free(tk);
There's no problem in not freeing the structure and the string (which is in another memory location and not inside the structure's memory space, that's why you have to free they both), if your program is that small, because after it's executed, the program's memory will be wiped out anyways. But if you intend to implement this function on a fully-working and big program, freeing the memory is a good action.
It is not clear where str is defined, but if it is a local variable in the function, your problem is likely that it goes out of scope, so the data gets overwritten.
You're leaking memory because you've forgotten to use strcpy() or memcpy() or memmove() to copy the value in str over the allocated space, and you overwrite the only pointer to the newly allocated memory with the pointer str. If you copied, you would be writing out of bounds because you forgot to allocate enough space for the trailing null as well as the string. You should also check that the allocation succeeds.
Bogus code:
tokenizer->tks = (char*) malloc (strlen(str)* sizeof(char));
tokenizer->tks = str;
Fixed code:
size_t len = strlen(str) + 1;
tokenizer->tks = (char *)malloc(len);
if (tokenizer->tks == 0)
...error handling...
memmove(tokenizer->tks, str, len);
Using memmove() or memcpy() can outperform strcpy() handily (see Why is Python faster than C for some illustrations and timing). There are those who would excoriate you (and me) for using the cast on malloc(); I understand why they argue as they do, but I don't fully agree with them (and usually use the cast myself). Since sizeof(char) is 1 by definition, there's no particular need to multiply by it, though there's no harm done in doing so, either.