strtok only returning one token - c

I'm writing a simple shell that accepts some standard commands like cd and ls in C. I'm trying to implement a feature where the user can enter a ";" in between commands so that a bunch of commands can be written on the same line and be executed separately. So if I input "cd Desktop; ls" the shell should cd to Desktop and print the what's in the directory. The problem is it only executes the first command. Here's my main method:
char input[1024];
while(1)
{
printf("%s ", prompt);
fgets(input, 1024, stdin);
char delims[] = ";";
char *result = NULL;
result = strtok( input, delims );
while( result != NULL )
{
printf("%s\n", result);
char * copy = malloc(strlen(result) + 1); //Create a copy of the input token
strcpy(copy, result);
format(copy);
if(programs)
{
handle();
cleanup(programs);
programs = NULL;
}
free(copy);
result = strtok( NULL, delims );
cmdno++;
}
}
First I try to break up the input into tokens based on ";" and then feed the token to the format() method which looks like this:
int format(char input[])
{
input = strtok(input, "\n");
...
}
I know that strtok makes changes to the original string, which is why I create a copy of the token first before passing it to format. Is what I'm doing correct??

You can't mix multiple strtok calls. Here's what's happening:
You start splitting input so strtok takes note and stores stuff internally
You take a break from splitting input
You start splitting copy so again strtok takes note, thereby destroying the previous info
At this point strtok only knows about the copy business and doesn't know anything about the original input.
The main problem is that strtok doesn't know that you're doing two things at the same time. From its point of view, you simply started processing a different string before finishing the first string.
Possible solutions:
Use strtok_r if you have it. It's not standard C (but it is standard POSIX). The r stands for reentrant
Use your own splitting function (strchr / looping etc)
Change your program logic such that you don't need to split copy before finishing with input
About that last point:
Keep an array of char * and fill it with strtok without pausing to split sub-tokens. So each element should be a different command
When you're done with the ";" split, start processing each of the array elements

What about this:
char line[1024];
char *token;
while (1) {
printf("$ ");
fgets(line, 1000, stdin);
token = strtok(line, ";");
while (token != NULL) {
printf("%s\n", token);
token = strtok(NULL, ";");
}
}

Related

strtok not splitting spaces correctly even with correct delimiter

I am attempting to split two words (and more) and put them into an array by splitting it into tokens with strtok. My delimitors include " \t\n"; as shown below in the code. For example if I were to type in "cat program.c", it just prints the cat token and not the program.c token and I have no idea why. Are my delimitors not correct or am I not splitting the string correctly? Here is the code
char b[256];
int k = 0;
char *args[4];
char *tokens;
char delimiters[] = " \t\n";
printf("Please enter the command you want to use:\n");
scanf("%255s", b);
tokens = strtok(b, delimiters);
while (tokens != NULL){
args[k++] = tokens;
printf("%s\n",tokens);
tokens = strtok(NULL, delimiters);
}
The problem is not strtok(), but rather scanf(). An %s field directive scans a whitespace-delimited string, so when the input is cat program.c, only the "cat" ever makes it into array b in the first place. (The program.c remains waiting to be read.) If you want to read a whole line of input at a time, then I would recommend fgets(), instead.

C - Segmentation fault using strtok

I have this code where its read multiple files and print a certain value. After reading files, at a certain moment my while loop stop and show a segmentation fault ...
Here is my code
int main () {
const char s[2] = ",";
const char s2[2] = ":";
char var1[] = "fiftyTwoWeekHigh\"";
char *fiftyhigh;
char *fiftyhigh2;
char *fiftyhigh_token;
char *fiftyhigh2_token;
char var2[] = "fiftyTwoWeekLow\"";
char *fiftylow;
char *fiftylow2;
char *fiftylow_token;
char *fiftylow2_token;
char var3[] = "regularMarketPrice\"";
char *price;
char *price2;
char *price_token;
char *price2_token;
FILE *fp;
char* data = "./data/";
char* json = ".json";
char line[MAX_LINES];
char line2[MAX_LINES];
int len;
char* fichier = "./data/indices.txt";
fp = fopen(fichier, "r");
if (fp == NULL){
printf("Impossible d'ouvrir le fichier %s", fichier);
return 1;
}
while (fgets(line, sizeof(line), fp) != NULL) {
char fname[10000];
len = strlen(line);
if (line[len-1] == '\n') {
line[len-1] = 0;
}
int ret = snprintf(fname, sizeof(fname), "%s%s%s", data, line, json);
if (ret < 0) {
abort();
}
printf("%s\n", fname);
FILE* f = fopen(fname, "r");
while ( fgets( line2, MAX_LINES, f ) != NULL ) {
fiftyhigh = strstr(line2, var1);
fiftyhigh_token = strtok(fiftyhigh, s);
fiftyhigh2 = strstr(fiftyhigh_token, s2);
fiftyhigh2_token = strtok(fiftyhigh2, s2);
printf("%s\n", fiftyhigh2_token);
fiftylow = strstr(line2, var2);
fiftylow_token = strtok(fiftylow, s);
fiftylow2 = strstr(fiftylow_token, s2);
fiftylow2_token = strtok(fiftylow2, s2);
printf("%s\n", fiftylow2_token);
price = strstr(line2, var3);
price_token = strtok(price, s);
price2 = strstr(price_token, s2);
price2_token = strtok(price2, s2);
printf("%s\n", price2_token);
//printf("\n%s\t%s\t%s\t%s\t%s", line, calculcx(fiftyhigh2_token, price2_token, fiftylow2_token), "DIV-1", price2_token, "test");
}
fclose(f);
}
fclose(fp);
return 0;
}
and the output is :
./data/k.json
13.59
5.31
8.7
./data/BCE.json
60.14
46.03
56.74
./data/BNS.json
80.16
46.38
78.73
./data/BLU.json
16.68
2.7
Segmentation fault
It is like my program stop because it can't reach a certain data at a certain file... Is there a way to allocate more memory ? Because my MAX_LINES is already set at 6000.
I'm assuming that the lines in your file look something like this:
{"fiftyTwoWeekLow":32,"fiftyTwoWeekHigh":100, ... }
In other words it's some kind of JSON format. I'm assuming that the line starts with '{' so each line is a JSON object.
You read that line into line2, which now contains:
{"fiftyTwoWeekLow":32,"fiftyTwoWeekHigh":100, ... }\0
Note the \0 at the end that terminates the string. Note also that "fiftyTwoWeekLow" comes first, which turns out to be really important.
Now let's trace through the code here:
fiftyhigh = strstr(line2, var1);
fiftyhigh_token = strtok(fiftyhigh, s);
First you call strstr to find the position of "fiftyTwoWeekHigh". This will return a pointer to the position of that field name in the line. Then you call strtok to find the comma that separates this value from the next. I think that this is where things start to go wrong. After the call to strtok, line2 looks like this:
{"fiftyTwoWeekLow":32,"fiftyTwoWeekHigh":100\0 ... }\0
Note that strtok has modified the string: the comma has been replaced with \0. That's so you can use the returned pointer fiftyhigh_token as a string without seeing all the stuff that came after the comma.
fiftyhigh2 = strstr(fiftyhigh_token, s2);
fiftyhigh2_token = strtok(fiftyhigh2, s2);
printf("%s\n", fiftyhigh2_token);
Next you look for the colon and then call strtok with a pointer to the colon. Since the delimiter you're passing to strok is the colon, strtok ignores the colon and returns the next token, which (because the string we're looking at, which ends after "100," has no more colons) is the rest of the string, in other words, the number.
So you've gotten your number, but probably not in the way you expected? There was really no point in the second call to strtok since (assuming the JSON was well-formed) the position of "100" was just fiftyhigh2+1.
Now we try to find "fiftyTwoWeekLow:"
fiftylow = strstr(line2, var2);
fiftylow_token = strtok(fiftylow, s);
fiftylow2 = strstr(fiftylow_token, s2);
fiftylow2_token = strtok(fiftylow2, s2);
printf("%s\n", fiftylow2_token);
This is basically the same process, and after you call strtok, line2 like this:
{"fiftyTwoWeekLow":32\0"fiftyTwoWeekHigh":100\0 ... }\0
Note that you're only able to find "fiftyTwoWeekLow" because it comes before "fiftyTwoWeekHigh" in the line. If it had come after, then you'd have been unable to find it due to the \0 added after "fiftyTwoWeekHigh" earlier. In that case, strstr would have returned NULL, which would cause strtok to return NULL, and then you'd definitely have gotten a seg fault after passing NULL to strstr.
So the code is really sensitive to the order in which the fields appear in the line, and it's probably failing because some of your lines have the fields in a different order. Or maybe some fields are just missing from some lines, which would have the same effect.
If you're parsing JSON, you should really use a library designed for that purpose. But if you really want to use strtok then you should:
Read line2.
Call strtok(line2, ",") once, then repeatedly call strtok(NULL, ",") in a loop until it returns null. This will break up the line into tokens that each look like "someField":100.
Isolate the field name and value from each of these tokens (just call strchr(token, ':') to find the value). Do not call strtok here, because it will change the internal state of strtok and you won't be able to use strtok(NULL, ",") to continue processing the line.
Test the field name, and depending on its value, set an appropriate variable. In other words, if it's the "fiftyTwoWeekLow" field, set a variable called fiftyTwoWeekLow. You don't have to bother to strip off the quotes, just include them in the string you're comparing with.
Once you've processed all the tokens (strtok returns NULL), do something with the variables you set.
You may be to pass ",{}" as the delimiter to strtok in order to get rid of any open and close curly braces that surround the line. Or you could look for them in each token and ignore them if they appear.
You could also pass "\"{},:" as the delimiter to strtok. This would cause strtok to emit an alternating sequence of field names and values. You could call strtok once to get the field name, again to get the value, then test the field name and do something with the value.
Using strtok is a pretty primitive way of parsing JSON, but it will will work as long as your JSON only contains simple field names and numbers and doesn't include any strings that themselves contain delimiter characters.
Did you mean '\0' ?
if (line[len-1] == '\n') {
line[len-1] = 0;
}
I advise you to use gdb to see where the segfault occurs and why.
I don't think you have to allocate much more memory. But the segfault may happens because you don't have anymore data and you still print the result.
Use if(price2_token!=NULL) printf("%s\n", price2_token); for example.

How to combine a strings and store it in a array after using strtok() from a "full string" in C

Im a new learner of C and i got trouble combining the strings which separated by strtok().
Using Clion (C99)
Lets have a look at my codes.
char recipient[30];
char final_destination[50];
char status[10];
printf("Please enter 1> Recipient-, 2> Final Destination- and
3>Delivery status :\n"); Entered(L-Rat Kitchen-House Not-Arrived)
scanf("%29s%49s%9s", recipient, final_destination, status);
///only recipient first///The purpose of storing "recipient final_destination, status" in a string is because to let user input data in a same line while using scanf .After that, i use strtok to separate those words before/after "-" .Separate different string by using the spacebar in scanf which prevents data input like (L-ratKitchenHouseNot-arrive).
const char s[] = "-";
char *token;
token = strtok(recipient, s);
while( token != NULL )
???????????????????????
token = strtok(NULL, s);
At the line of ?????????????? , im stuck , i dont know how or is there a way to put those processed data in a array and combine it.
Expecting:
Input: (L-Rat House-Kitchen Not-Arrive)
Output: (L Rat House Kitchen Not Arrive)
Would be very helpful if theres a solution !!!
try this, if you want store tokens in an array.
while (token != NULL)
{
printf("%s\n", token);
token = strtok(NULL, "-");
}

C - Nested loop using strtok

I am trying to use strtok to split up a text file into strings that I can pass to a spell check function, the text file includes characters such as '\n', ' ?!,.' etc...
I need to print any words that fail the spell check and the line number that they are on. Keeping track of the line is what I'm struggling with.
I have tried this so far but it only returns results for the first line of the text file:
char str[409377];
fread(str, noOfChars, 1, file);
fclose(file);
int lines=1;
char *token;
char *line;
char splitLine[] = "\n";
char delimiters[] = " ,.?!(){}*&^%$£_-+=";
line = strtok(str, splitLine);
while(line!=NULL){
token = strtok(line, delimiters);
while(token != NULL){
//print is just to test if I can loop through all the words
printf("%s", token);
//spellCheck function & logic here
token = strtok(NULL, delimiters);
}
line = strtok(NULL, splitLine);
lines++
}
Is using the nested while loop and strtok possible? Is there a better way to keep track of the line number?
The strtok function is not reentrant! It can not be used to tokenize multiple strings simultaneously. It's because it keeps internal state about the string currently being tokenized.
If you have a modern compiler and standard library then you could use strtok_s instead. Otherwise you have to come up with another solution.
You can use strtok, but it's not very easy to use. It's a stupid function, all it really does is replace delimiters with nuls and return a pointer to the start of the sequence it has delimited. So it's destructive. It can't handle special cases like English words being allowed one apostrophe (we're is a word, we'r'e is not), you have to make sure you list all the delimiters specifically.
It's probably best to write mystrok yourself, so you understand how it works. Then use that as the basis for your own word extractor.
The reason for your bug is that you chop off the first line, then that is all that strok sees on the subsequent calls.

Remove the first part of a C String

I'm having a lot of trouble figuring this out. I have a C string, and I want to remove the first part of it. Let's say its: "Food,Amount,Calories". I want to copy out each one of those values, but not the commas. I find the comma, and return the position of the comma to my method. Then I use
strncpy(aLine.field[i], theLine, end);
To copy "theLine" to my array at position "i", with only the first "end" characters (for the first time, "end" would be 4, because that is where the first comma is). But then, because it's in a Loop, I want to remove "Food," from the array, and do the process over again. However, I cannot see how I can remove the first part (or move the array pointer forward?) and keep the rest of it. Any help would be useful!
What you need is to chop off strings with comma as your delimiter.
You need strtok to do this. Here's an example code for you:
int main (int argc, const char * argv[]) {
char *s = "asdf,1234,qwer";
char str[15];
strcpy(str, s);
printf("\nstr: %s", str);
char *tok = strtok(str, ",");
printf("\ntok: %s", tok);
tok = strtok(NULL, ",");
printf("\ntok: %s", tok);
tok = strtok(NULL, ",");
printf("\ntok: %s", tok);
return 0;
}
This will give you the following output:
str: asdf,1234,qwer
tok: asdf
tok: 1234
tok: qwer
If you have to keep the original string, then strtok. If not, you can replace each separator with '\0', and use the obtained strings directly:
char s_RO[] = "abc,123,xxxx", *s = s_RO;
while (s){
char* old_str = s;
s = strchr(s, ',');
if (s){
*s = '\0';
s++;
};
printf("found string %s\n", old_str);
};
The function you might want to use is strtok()
Here is a nice example - http://www.cplusplus.com/reference/clibrary/cstring/strtok/
Personally, I would use strtok().
I would not recommend removing extracted tokens from the string. Removing part of a string requires copying the remaining characters, which is not very efficient.
Instead, you should keep track of your positions and just copy the sections you want to the new string.
But, again, I would use strtok().
if you know where the comma is, you can just keep reading the string from that point on.
for example
void readTheString(const char *theLine)
{
const char *wordStart = theLine;
const char *wordEnd = theLine;
int i = 0;
while (*wordStart) // while we haven't reached the null termination character
{
while (*wordEnd != ',')
wordEnd++;
// ... copy the substring ranging from wordStart to wordEnd
wordStart = ++wordEnd; // start the next word
}
}
or something like that.
the null termination check is probably wrong, unless the string also ends with a ','... but you get the idea.
anyway, using strtok would probably be a better idea.

Resources