C bug with creating a substring - possibly a memory error? - c

Question

I'm pretty new to C and was wondering if I could get some help! I've been working on this bug for +15 hours.
So, this program is a tokenizer.
Basically, the program is supposed to take a string, or "token stream," and break it up into "tokens." A "token" is a string of either a word, hexadecimal int, octal int, decimal int, floating point int, or symbol.
The code I'm posting is only the code where things go wrong, the other portion of my program is what creates the token.
The gist of how the below code works is this: It takes a "token stream", and then finds the next token from that stream. Once that is completed, it will create a substring of the "token stream" minus the new token, and return that as the new "token stream."
Essentially, when the string "0x4356/*abdc 0777 */[]87656879jlhg kl(/j jlkh 'no thank you' /" is passed through, the program will do everything properly except when "jlhg kl(/j jlkh 'no thank you' /" passes. Once that passes through my program, a "jlhg" token is created BUT then it is added to the end of the token stream again. So, the new token stream to be broken down becomes " kl(/j jlkh 'no thank you' / jlhg" where jlhg is added on at the end, where it wasn't there before. It does this same weird thing once more, right afterwards, but with "kl" instead.
It only does this under extremely weird conditions, so I'm not sure the cause. I put print statements throughout my program and things flow normally except seemingly out of no where, the program will just add those at the end. This I why I feel like it might be a memory problem, but I have absolutely no clue where to go from here.
Any help would be GREATLY appreciated!!!!
EDIT: If you pass the string "array[xyz ] += pi 3.14159e-10 A12B" output should be:
word "array"
left brace "["
word "xyz"
right brace "]"
plusequals "+="
word "pi"
float "3.14159e-10"
word "A12B"
My TokenizerT is this:
struct TokenizerT_
{
char *tokenType;
char *token;
};
typedef struct TokenizerT_ TokenizerT;
Relevant code:
/*
* TKNewStream takes two TokenizerT objects.
* It will locate the index of the end of the last token,
* and create a substring with the new string to be tokenized.
* #tokenStream: old token stream
* #newToken: new token created from old token stream
*
*/
char *TKGetNextStream(char *tokenStream, char *newToken)
{
int i,
index = 0,
count = 0;
char last = newToken[strlen(newToken)-1];
for(i = 0; i < strlen(newToken); i++)
{
if(newToken[i] == last)
{
count++;
}
}
for(i = 0; i < strlen(tokenStream); i++)
{
if(tokenStream[i] == last && count == 1)
{
index = i + 1;
break;
}
else if(tokenStream[i] == last)
{
count--;
}
}
char *ret = malloc(sizeof(char)*(strlen(tokenStream) - index));
for(i = 0; i < strlen(tokenStream) - index; i++)
{
ret[i] = tokenStream[i+index];
}
return ret;
}
/*
* This is my main
*/
int main(int argc, char **argv)
{
char *string = "0x4356/*abdc 0777 */[]87656879jlhg kl(/j jlkh 'no thank you' /";
TokenizerT *newToken = malloc(sizeof(struct TokenizerT_)),
*tokenStream = malloc(sizeof(struct TokenizerT_));
tokenStream->token = string;
while(newToken != NULL)
{
newToken = TKCreate(TKGetNextToken(tokenStream));
if(newToken != NULL)
{
tokenStream->token = TKGetNextStream(tokenStream->token,
newToken->token);
printf("%s \"%s\"\n",
newToken->tokenType,
newToken->token);
}
}
TKDestroy(newToken);
return 0;
}

Answer 1

The string created in ret isn't properly null terminated. So all the functions dealing with strings will assume it goes on until the next random zero byte that happens to be found after the allocated memory.
To fix this allocate one more byte of space for ret and set that to zero, or use an existing function like strdup() to copy the string:
ret = strdup(tokenStream + index);