Below is a piece of my program which loads a file(dictionary) into memory using hashtables. The dictionary contains only 1 word per line. But the process is taking too much time. How do I optimize it ??
bool load(const char* dictionary)
{
// TODO
int k;
FILE* fp = fopen(dictionary,"r");
if(fp == NULL)
return false;
for(int i=0; i<26; i++)
{
hashtable[i] = NULL;
}
while(true)
{
if(feof(fp))
return true;
node* n = malloc(sizeof(node));
n->pointer = NULL;
fscanf(fp,"%s",n->word);
if(isalpha(n->word[0]))
{
k = hashfunction(n->word);
}
else return true;
if(hashtable[k] == NULL)
{
hashtable[k] = n;
total_words++;
}
else
{
node* traverse = hashtable[k];
while(true)
{
if(traverse->pointer == NULL)
{
traverse->pointer = n;
total_words++;
break;
}
traverse = traverse->pointer;
}
}
}
return false;
}
Get rid of potential functional problems, then worry about performance.
A) for(int i=0; i<26; i++) may be wrong, hashtable[] definition not posted. It is certainly unwise for performance to use such a small fixed table.
B) "%s" is as safe as gets() - both are bad. Instead of fscanf(fp,"%s",n->word);, use fgets().
C) Instead of if(feof(fp)), check return value from fscanf()/fgets().
D) isalpha(n->word[0]) --> isalpha((unsigned char) n->word[0]) to cope with negative char values.
E) Check for memory allocation failure.
F) Other issues may also exists depending on unposted code.
Then form a simple test case and with minimal code that works, consider posting on codereview.stackexchange.com to solicit performance improvements.
You are making the assumption that all the words in the file are distinct. That is a reasonable assumption for a dictionary but it is bad defensive programming. You should always assume that input is out to get you, which means you cannot really assume anything about it.
In this case, though, you could argue that repeated words in the hashtable do not prevent it from working; they just slow it down slightly. Since the erroneous input won't cause bugs, undefined behaviour, or other catastrophes, it is marginally acceptable to document the requirement that reference words be unique.
Anyway, if you are not actually checking for duplicates, there is no need to walk the entire hash bucket for every insertion. If you insert new entries at the beginning of the bucket rather than at the end, you can avoid the scan which will probably yield a noticeable speedup if the buckets are large.
Of course, that optimization can only be used when loading the dictionary. It won't help you use the hashtable once initialization is complete, and it is rarely worthwhile hyper-optimizing startup code.
Related
Just wondering if these variations of for loops are more efficient and practical.
By messing with the c for loop syntax i can embedd statements that would go in the loop-body into the loop-head like so:
Example 1:
#include <stdio.h>
int main(int argc, char ** argv)
{
// Simple program that prints out the command line arguments passed in
if (argc > 1)
{
for(int i = 1; puts(argv[i++]), i < argc;);
// This does the same as this:
// for(int i = 1; i < argc; i++)
// {
// puts(argv[i]);
// }
}
return 0;
}
I understand how the commas work in the for loop it goes through each statement in order, evaluates them then disregards all but the last one which is why it is able to iterate using the "i < argc"condition. There is no need for the final segment to increment the i variable as i did that in the middle segment of the loop head (in the puts(argv[i++]) bit).
Is this more efficient or is just just cleaner to seperate it into the loop body rather than combine it all into one line?
Example 2:
int stringLength(const char * string)
{
// Function that counts characters up until null terminator character and returns the total
int counter = 0;
for(counter; string[counter] != '\0'; counter++);
return counter;
// Same as:
//int counter = 0;
// for(int i = 0; string[i] != '\0'; i++)
//{
// counter++;
//}
//return counter;
}
This one seems more efficient than the version with the loop body as no local variable for the for-loop is initialised. Is it conventional to do these sorts of loops with no bodies?
Step 1: Correctness
Make sure code is correct.
Consider OP's code below. Does it attempt to print argv[argc] which would be bad?
if (argc > 1) {
for(int i = 1; puts(argv[i++]), i < argc;);
I initially thought it did. So did another user. Yet it OK.
… and this is exactly why code is weak.
Code not only should be correct, better code looks correct too. Using an anti-pattern as suggested by OP is rarely1 as good thing.
Step 2: Since code variations have the same big O, focus on understandably.
Sculpt your code – remove what is not needed.
for (int i = 1; i < argc; i++) {
puts(argv[i]);
}
What OP is doing is a trivial optimization concern.
Is premature optimization really the root of all evil?
Is it conventional to do these sorts of loops with no bodies?
Not really.
The key to the style of coding is to follow your group's style guide. Great software is often a team effort. If your group's likes to minimize bodies, go ahead. I have seen the opposite more common, explicit { some_code } bodies.
Note: int stringLength(const char * string) fails for strings longer than INT_MAX. Better to use size_t as the return type – thus an example of step 1 faltering.
1 All coding style rules, except this rule, have exceptions.
I am probably not going to get help here because my question is far from being specific (I don't even know what exactly wrong with it) but, according to my professor's tests, there is something wrong with it (wrong in terms of correctness - it doesn't provide correct number of direct and indirect matches) (I have no access to his tests). As far as I have been testing, it passes all of my tests. However, there are over a couple hundred million possible outcomes (I think) and I can't test them all because I don't know how to do automated testing...
Here is my code that performs the "logic" part of the game called mastermind, which is compares a string of randomly generated letter (8 max) with user input string (a guess). I wanted to see if anyone has encountered this game in the past and knows the logic of how it supposed to compare two strings and generate the correct number of exact and inexact guesses.
// userInput->position - a length of a string(max 8)
// userInput->code - randomly generated code
// userInput->arr - user input string
void checkForExactMatch(Data* userInput) {
int i;
for (i = 0; i < userInput->position; i++) {
if (userInput->code[i] == userInput->arr[i]) {
userInput->exactMatch++;
userInput->arr[i] = 'a';
}
else
checkForInExactMatch(userInput, i);
}
}
void checkForInExactMatch(Data* userInput, int i) {
int j;
for (j = 0; j < userInput->position; j++) {
if (userInput->arr[j] == userInput->code[i]) {
userInput->arr[j] = 'a';
userInput->inExactMatch++;
break;
}
}
}
Looking over your code there were a couple of observations to be made. First in your for checkForExactMatch() the call to checkForInExactMatch is inside your for loop. So on the first mismatch you call checkForInExactMatch and when you return from checkForInExactMatch -- you call it again on the next iteration unless your first mismatch just happens to be on the final character.
To address that issue, you should fully determine whether you have an exact match or not, completing the for loop before checkForInExactMatch is called.
In your checkForInExactMatch, you have to decide whether a single common-character or some minimum length substring constitutes an inexact match.
It sounds like you have things worked out, and good job for pushing through to a solution. Depending on how you approached it, keeping a simple flag in checkForExactMatch()such as int matched = 1; and then loop turning your test around
for (i = 0; i < userInput->position; i++)
if (userInput->code[i] != userInput->arr[i]) {
matched = 0;
break;
}
Then it's just a simple test of
if (matched) {
userInput->exactMatch++;
userInput->arr[i] = 'a';
}
else
checkForInExactMatch(userInput, i);
So long as what you have done accomplishes something similar, you are fine. Let me know if you have further questions.
I have a few questions about an assignment that i need to do. It might seem that what im looking for is to get the code, however, what im trying to do is to learn because after weeks of searching for information im lost. Im really new atC`.
Here is the assignment :
Given 3 files (foo.txt , bar.txt , foo2.txt) they all have a different amount of words (I need to use dynamic memory).
Create a program that ask for a word and tells you if that word is in any of the documents (the result is the name of the document where it appears).
Example :
Please enter a word: dog
"dog" is in foo.txt and bar.txt
(I guess i need to load the 3 files, create a hash table that has the keyvalues for every word in the documents but also has something that tells you which one is the document where the word is at).
I guess i need to implement:
A Hash Function that converts a word into a HashValue
A Hash Table that stores the HashValue of every word (But i think i should also store the document index?).
Use of dynamic allocation.
Check for collisions while im inserting values into the hash table (Using Quadratic Probing and also Chaining).
Also i need to know how many times the word im looking for appears in the text.
I've been searching about hashmaps implementations, hash tables , quadratic probing, hash function for strings...but my head is a mess right now and i dont really now from where i should start.
so far i`ve read :
Algorithm to get a list of all words that are anagrams of all substrings (scrabble)?
Implementing with quadratic probing
Does C have hash/dictionary data structure?
https://gist.github.com/tonious/1377667
hash function for string
http://www.cs.yale.edu/homes/aspnes/pinewiki/C(2f)HashTables.html?highlight=(CategoryAlgorithmNotes)
https://codereview.stackexchange.com/questions/115843/dictionary-implementation-using-hash-table-in-c
Sorry for my english in advance.
Hope you can help me.
Thanks.
FIRST EDIT
Thanks for the quick responses.
I'm trying to put all together and try something, however #Shasha99 I cannot use the TRIE data structure, i'm checking the links you gave me.
#MichaelDorgan Thanks for posting a solution for beginners however i must use Hashing (It's for Algorithms and Structures Class) and the teacher told us we MUST implement a Hash Function , Hash Table and probably another structure that stores important information.
After thinking for an hour I tried the following :
A Structure that stores the word, the number of documents where it appears and the index of those documents.
typedef struct WordMetadata {
char* Word;
int Documents[5];
int DocumentsCount;
} WordMetadata;
A function that Initializes that structure
void InitTable (WordMetadata **Table) {
Table = (WordMetadata**) malloc (sizeof(WordMetadata) * TABLESIZE);
for (int i = 0; i < TABLESIZE; i++) {
Table[i] = (WordMetadata*) NULL;
}
}
A function that Loads to memory the 3 documents and index every word inside the hash table.
A function that index a word in the mentioned structure
A function that search for the specific word using Quadratic Probing (If i solve this i will try with the chaining one...).
A function that calculates the hash value of a word (I think i will use djb2 or any of the ones i found here http://www.cse.yorku.ca/~oz/hash.html) but for now :
int Hash (char *WordParam) {
for (int i = 0; *WordParam != '\0';) {
i += *WordParam++;
}
return (i % TABLESIZE);}
EDIT 2
I tried to implement something, its not working but would take a look and tell me what is wrong (i know the code is a mess)
EDIT 3
This code is properly compiling and running, however , some words are not finded (maybe not indexed i' dont know), i'm thinking about moving to another hashfunction as i mentioned in my first message.
Approximately 85% of the words from every textfile (~ 200 words each) are correctly finded by the program.
The other ones are ramdom words that i think are not indexed correctly or maybe i have an error in my search function...
Here is the current (Fully functional) code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#define TABLESIZE 4001
#define LINESIZE 2048
#define DELIMITER " \t"
typedef struct TTable {
char* Word; /* The actual word */
int Documents[5]; /* Documents Index */
int DocumentsCount; /* Number of documents where the word exist */
} TTable;
int Hash (char *Word);
void Index (TTable **HashTable, char* Word, int DocumentIndex);
int Search (TTable **HashTable, char* Word);
int mystrcmp(char *s1, char *s2);
char* Documents[] = {"foo.txt","bar.txt","foo2.txt",NULL};
int main() {
FILE* file;
TTable **HashTable
int DocumentIndex;
char Line[LINESIZE];
char* Word;
char* Tmp;
HashTable = (TTable**) malloc (sizeof(TTable)*TABLESIZE);
for (int i = 0; i < TABLESIZE; i++) {
HashTable[i] = (TTable*) NULL;
}
for (DocumentIndex = 0; Documents[DocumentIndex] != NULL; DocumentIndex++) {
file = fopen(Documents[DocumentIndex],"r");
if (file == NULL) {
fprintf(stderr, "Error%s\n", Documents[DocumentIndex]);
continue;
}
while (fgets (Line,LINESIZE,file) != NULL) {
Line[LINESIZE-1] = '\0';
Tmp = strtok (Line,DELIMITER);
do {
Word = (char*) malloc (strlen(Tmp)+1);
strcpy(Word,Tmp);
Index(HashTable,Word,DocumentIndex);
Tmp = strtok(NULL,DELIMITER);
} while (Tmp != NULL);
}
fclose(file);
}
printf("Enter the word:");
fgets(Line,100,stdin);
Line[strlen(Line)-1]='\0'; //fgets stores newline as well. so removing newline.
int i = Search(HashTable,Line);
if (i != -1) {
for (int j = 0; j < HashTable[i]->DocumentsCount; j++) {
printf("%s\n", Documents[HashTable[i]->Documents[j]]);
if ( j < HashTable[i]->DocumentsCount-1) {
printf(",");
}
}
}
else {
printf("Cant find word\n");
}
for (i = 0; i < TABLESIZE; i++) {
if (HashTable[i] != NULL) {
free(HashTable[i]->Word);
free(HashTable[i]);
}
}
return 0;
}
/* Theorem: If TableSize is prime and ? < 0.5, quadratic
probing will always find an empty slot
*/
int Search (TTable **HashTable, char* Word) {
int Aux = Hash(Word);
int OldPosition,ActualPosition;
ActualPosition = -1;
for (int i = 0; i < TABLESIZE; i++) {
OldPosition = ActualPosition;
ActualPosition = (Aux + i*i) % TABLESIZE;
if (HashTable[ActualPosition] == NULL) {
return -1;
}
if (strcmp(Word,HashTable[ActualPosition]->Word) == 0) {
return ActualPosition;
}
}
return -1; // Word not found
}
void Index (TTable **HashTable, char* Word, int DocumentIndex) {
int Aux; //Hash value
int OldPosition, ActualPosition;
if ((ActualPosition = Search(HashTable,Word)) != -1) {
for (int j = 0; j < HashTable[ActualPosition]->DocumentsCount;j++) {
if(HashTable[ActualPosition]->Documents[j] == DocumentIndex) {
return;
}
}
HashTable[ActualPosition]->Documents[HashTable[ActualPosition]->DocumentsCount] = DocumentIndex; HashTable[ActualPosition]->DocumentsCount++;
return;
}
ActualPosition = -1;
Aux = Hash(Word);
for (int i = 0; i < TABLESIZE; i++) {
OldPosition = ActualPosition;
ActualPosition = (Aux + i*i) % TABLESIZE;
if (OldPosition == ActualPosition) {
break;
}
if (HashTable[ActualPosition] == NULL) {
HashTable[ActualPosition] = (TTable*)malloc (sizeof(TTable));
HashTable[ActualPosition]->Word = Word;
HashTable[ActualPosition]->Documents[0] = DocumentIndex;
HashTable[ActualPosition]->DocumentsCount = 1;
return;
}
}
printf("No more free space\n");
}
int Hash (char *Word) {
int HashValue;
for (HashValue = 0; *Word != '\0';) {
HashValue += *Word++;
}
return (HashValue % TABLESIZE);
}
I would suggest you to use TRIE data structure for storing strings present in all three files in memory as Hash would be more space consuming.
As the first step you should read all three files one by one and for each word in file_i, you should do the following:
if the word is already present in TRIE, append the file index to that node or update the word count relative to that particular file. You may need 3 variables for file1, file and file3 at each node to store the values of word count.
if the word is not present, add the word and the file index in TRIE node.
Once you are done with building your TRIE, checking whether the word is present or not would be an O(1) operation.
If you are going with Hash Tables, then:
You should start with how to get hash values for strings.
Then read about open addressing, probing and chaining
Then understand the problems in open addressing and chaining approaches.
How will you delete and element in hash table with open addressing and probing ? here
How will the search be performed in case of chaining ? here
Making a dynamic hash table with open addressing ? Amortized analysis here and here.
Comparing between chaining and open addressing. here.
Think about how these problems can be resolved. May be TRIE ?
Problem in the code of your EDIT 2:
An outstanding progress from your side !!!
After a quick look, i found the following problems:
Don't use gets() method, use fgets() instead So replace:
gets(Line);
with the following:
fgets(Line,100,stdin);
Line[strlen(Line)-1]='\0'; //fgets stores newline as well. so removing newline.
The line:
if ( j < HashTable[j]->DocumentsCount-1){
is causing segmentation fault. I think you want to access HashTable[i]:
if ( j < HashTable[i]->DocumentsCount-1){
In the line:
HashTable[ActualPosition]->Documents[HashTable[ActualPosition]->DocumentsCount];
You were supposed to assign some value. May be this:
HashTable[ActualPosition]->Documents[HashTable[ActualPosition]->DocumentsCount] = DocumentIndex;
Malloc returns void pointer. You should cast it to the appropriate
one:
HashTable[ActualPosition] = (TTable*)malloc (sizeof(TTable));
You should also initialize the Documents array with default value while creating a new node in Hash:
for(j=0;j<5;j++)HashTable[ActualPosition]->Documents[j]=-1;
You are removing everything from your HashTable after finding the
first word given by user. May be you wanted to place that code outside
the while loop.
Your while loop while(1) does not have any terminating condition, You
should have one.
All the best !!!
For a school assignment, you probably don't need to worry about hashing. For a first pass, you can just get away with a straight linear search instead:
Create 3 pointers to char arrays (or a char ** if you prefer), one for each dictionary file.
Scan each text/dictionary file to see how many individual words reside within it. Depending on how the file is formatted, this may be spaces, strings, newlines, commas, etc. Basically, count the words in the file.
Allocate an array of char * times the word count in each file and store it in the char ** for that file. (if 100 words found in the file , num_words=100; fooPtr = malloc(sizeof(char *) * num_words);
Go back through the file a second time and allocate an array of chars to the size of each word in the file and store it in the previously created array. You now have a "jagged 2D array" for every word in each dictionary file.
Now, you have 3 arrays for your dictionaries and can use them to scan for words directly.
When given a word, setup a for loop to look through each file's char array. if the entered word matches with the currently scanned dictionary, you have found a match and should print the result. Once you have scanned all dictionaries, you are done.
Things to make it faster:
Sort each dictionary, then you can binary search them for matches (O(log n)).
Create a hash table and add each string to it for O(1) lookup time (This is what most professional solutions would do, which is why you found so many links on this.)
I've offered almost no code here, just a method. Give it a shot.
One final note - even if you decide to use a the hash method or a list or whatever, the code you write with arrays will still be useful.
So, I have two files of financial data, say 'symbols', and 'volumes'. In symbols I have strings such as:
FOO
BAR
BAZINGA
...
In volumes, I have integer values such as:
0001387
0000022
0123374
...
The idea is that the stock symbols will repeat in the file and I need to find the total volume of each stock. So, each row where I observe foo I increment total volume of foo by the value observed in volumes. The problem is that these files can be huge: easily 5 - 100 million records. A typical day may have ~1K different symbols in the file.
Doing it using strcmp on symbols each new line will be very inefficient. I was thinking of using an associative array --- hash table library which allows string keys --- such as uthash or Glib's hashtable.
I am reading some pretty good things about Judy arrays? Is the licensing a problem in this case?
Any thoughts on the choice of an efficient hash-table implementation? And also, whether I should use hash tables at all or perhaps something else entirely.
Umm.. apologize for the omission earlier: I need to have a pure C solution.
Thanks.
Definitely hashtable sounds good. You should look at the libiberty implementation.
You can find it on the GCC project Here.
I would use Map of C++ STL. Here's how the pseudo-code looks like:
map< string, long int > Mp;
while(eof is not reached)
{
String stock_name=readline_from_file1();
long int stock_value=readline_from_file2();
Mp[stock_name]+=stock_value;
}
for(each stock_name in Mp)
cout<<stock_name<<" "<<stock_value<<endl;
Based on the amount of data you gave, it may be a bit inefficient, but I'd suggest this because its much easier to implement.
If the solution is to be implemented strictly in C, then hashing will be the best solution. But, if you feel that implementing a hash-table and writing the code to avoid collisions is complex, I have another idea of using trie. It may sound weird, but this can also help a bit.
I would suggest you to read this one. It has a nice explanation about what a trie is and how to construct it. The implementation in C was also given there. So, you may have a doubt of where to store the volumes for each stock. This value can be stored at the end of the stock string and can be updated easily whenever needed.
But as you say that you are new to C, i advice you to try implementing using hash table and then try this one.
Thinking why not stick to your associative array idea. I assume, at the end of execution you need to a have list of unique names with their aggregated values. Below will work as far as you have memory to hold all unique names. ofcourse, this might not be that efficient, however, few tricks can be done depending upon the patterns of your data.
Consolidate_Index =0;
struct sutruct_Customers
{
name[];
value[];
}
sutruct_Customers Customers[This_Could_be_worse_if_all_names_are_unique]
void consolidate_names(char *name , int value)
{
for(i=0;i<Consolidate_Index;i++){
if(Customers[i].name & name)
{
Customers[i].value+= Values[index];
}
else
{
Allocate memory for Name Now!
Customers[Consolidate_Index].name = name;
Customers[Consolidate_Index].value = Value;
Consolidate_Index++;
}
}
}
main(){
sutruct_Customers buffer[Size_In_Each_Iteration]
while(unless file is done){
file-data-chunk_names to buffer.name
file-data-chunk_values to buffer.Values
for(; i<Size_In_Each_Iteration;i++)
consolidate_names(buffer.Names , buffer.Values);
}
My solution:
I did end up using the JudySL array to solve this problem. After some reading, the solution was quite simple to implement using Judy. I am replicating the solution here in full for it to be useful to anyone else.
#include <stdio.h>
#include <Judy.h>
const unsigned int BUFSIZE = 10; /* A symbol is only 8 chars wide. */
int main (int argc, char const **argv) {
FILE *fsymb = fopen(argv[1], "r");
if (fsymb == NULL) return 1;
FILE *fvol = fopen(argv[2], "r");
if (fvol == NULL) return 1;
FILE *fout = fopen(argv[3], "w");
if (fout == NULL) return 1;
unsigned int lnumber = 0;
uint8_t symbol[BUFSIZE];
unsigned long volume;
/* Initialize the associative map as a JudySL array. */
Pvoid_t assmap = (Pvoid_t) NULL;
Word_t *value;
while (1) {
fscanf(fsymb, "%s", symbol);
if (feof(fsymb)) break;
fscanf(fvol, "%lu", &volume);
if (feof(fvol)) break;
++lnumber;
/* Insert a new symbol or return value if exists. */
JSLI(value, assmap, symbol);
if (value == PJERR) {
fclose(fsymb);
fclose(fvol);
fclose(fout);
return 2;
}
*value += volume;
}
symbol[0] = '\0'; /* Start from the empty string. */
JSLF(value, assmap, symbol); /* Find the next string in the array. */
while (value != NULL) {
fprintf(fout, "%s: %lu\n", symbol, *value); /* Print to output file. */
JSLN(value, assmap, symbol); /* Get next string. */
}
Word_t tmp;
JSLFA(tmp, assmap); /* Free the entire array. */
fclose(fsymb);
fclose(fvol);
fclose(fout);
return 0;
}
I tested the solution on a 'small' sample containing 300K lines. The output is correct and the elapsed time was 0.074 seconds.
Can the performance of this sequential search algorithm (taken from
The Practice of Programming) be improved using any of C's native utilities, e.g. if I set the i variable to be a register variable ?
int lookup(char *word, char*array[])
{
int i
for (i = 0; array[i] != NULL; i++)
if (strcmp(word, array[i]) == 0)
return i;
return -1;
}
Yes, but only very slightly. A much bigger performance improvement can be achieved by using better algorithms (for example keeping the list sorted and doing a binary search).
In general optimizing a given algorithm only gets you so far. Choosing a better algorithm (even if it's not completely optimized) can give you a considerable (order of magnitude) performance improvement.
I think, it will not make much of a difference. The compiler will already optimize it in that direction.
Besides, the variable i does not have much impact, word stays constant throughout the function and the rest is too large to fit in any register. It is only a matter how large the cache is and if the whole array might fit in there.
String comparisons are rather expensive computationally.
Can you perhaps use some kind of hashing for the array before searching?
There is well-known technique as sentinal method.
To use sentinal method, you must know about the length of "array[]".
You can remove "array[i] != NULL" comparing by using sentinal.
int lookup(char *word, char*array[], int array_len)
{
int i = 0;
array[array_len] = word;
for (;; ++i)
if (strcmp(word, array[i]) == 0)
break;
array[array_len] = NULL;
return (i != array_len) ? i : -1;
}
If you're reading TPOP, you will next see how they make this search many times faster with different data structures and algorithms.
But you can make things a bit faster by replacing things like
for (i = 0; i < n; ++i)
foo(a[i]);
with
char **p = a;
for (i = 0; i < n; ++i)
foo(*p);
++p;
If there is a known value at the end of the array (e.g. NULL) you can eliminate the loop counter:
for (p = a; *p != NULL; ++p)
foo(*p)
Good luck, that's a great book!
To optimize that code the best bet would be to rewrite the strcmp routine since you are only checking for equality and don't need to evaluate the entire word.
Other than that you can't do much else. You can't sort as it appears you are looking for text within a larger text. Binary search won't work either since the text is unlikely to be sorted.
My 2p (C-psuedocode):
wrd_end = wrd_ptr + wrd_len;
arr_end = arr_ptr - wrd_len;
while (arr_ptr < arr_end)
{
wrd_beg = wrd_ptr; arr_beg = arr_ptr;
while (wrd_ptr == arr_ptr)
{
wrd_ptr++; arr_ptr++;
if (wrd_ptr == wrd_en)
return wrd_beg;
}
wrd_ptr++;
}
Mark Harrison: Your for loop will never terminate! (++p is indented, but is not actually within the for :-)
Also, switching between pointers and indexing will generally have no effect on performance, nor will adding register keywords (as mat already mentions) -- the compiler is smart enough to apply these transformations where appropriate, and if you tell it enough about your cpu arch, it will do a better job of these than manual psuedo-micro-optimizations.
A faster way to match strings would be to store them Pascal style. If you don't need more than 255 characters per string, store them roughly like this, with the count in the first byte:
char s[] = "\x05Hello";
Then you can do:
for(i=0; i<len; ++i) {
s_len = strings[i][0];
if(
s_len == match_len
&& strings[i][s_len] == match[s_len-1]
&& 0 == memcmp(strings[i]+1, match, s_len-1)
) {
return 1;
}
}
And to get really fast, add memory prefetch hints for string start + 64, + 128 and the start of the next string. But that's just crazy. :-)
Another fast way to do it is to get your compiler to use a SSE2 optimized memcmp. Use fixed-length char arrays and align so the string starts on a 64-byte alignment. Then I believe you can get the good memcmp functions if you pass const char match[64] instead of const char *match into the function, or strncpy match into a 64,128,256,whatever byte array.
Thinking a bit more about this, these SSE2 match functions might be part of packages like Intel's and AMD's accelerator libraries. Check them out.
Realistically, setting I to be a register variable won't do anything that the compiler wouldn't do already.
If you are willing to spend some time upfront preprocessing the reference array, you should google "The World's Fastest Scrabble Program" and implement that. Spoiler: it's a DAG optimized for character lookups.
/* there is no more quick */
int lookup(char *word, char*array[])
{
int i;
for(i=0; *(array++) != NULL;i++)
if (strcmp(word, *array) == 0)
return i;
return -1;
}