Polynomial Hashing vs Cyclic Polynomial shifting for strings - c

Question

I am using this function for cyclic shift:
int hashcyclic(char *p, int len)
{
unsigned int h = 0;
int i;
for (i = 0; i < len; i++)
{
h = (h << 5) | (h >> 27);
h += (unsigned int)p[i];
}
return h%TABLESIZE;
}
On a text file with around 20K lines (one word/line) total amount of collisions is 45187. On a text file with 40K+ lines (again, one word/line) there are 12922252 (!) collisions with the same algorithm.
With polynomial hashing:
int hashpoly(char *K)
{
int h = 0, a = 33;
for (; *K != '\0'; K++)
h = (a * h + *K) % TABLESIZE;
return h;
}
Now I'm getting around 25K collisions on the 20K word file and 901K collisions on the 40K word file(almost 12 times less than the cyclic shift).
My question is, does this make sense or is one of my implementations messed up? I was expecting cyclic to be the fastest for my strings (the 40K word file is a series of 8 letter words seperated by newline) but polynomial faces significantly less collisions.
int HashInsertPoly(Table T, KeyType K, InfoType I)
{
int i;
int ProbeDecrement;
i = hashpoly(K);
ProbeDecrement = p(K);
while (T[i].Key[0] != EmptyKey)
{
totalcol++;
T[i].Info.col++;
i -= ProbeDecrement;
if (i < 0)
i += TABLESIZE;
}
strcpy(T[i].Key, K);
insertions++;
/*T[i].Info = I;*/
return i;
}
The same HashInsert function applies to the hash with cyclic shift, except now I call hashcyclic instead of hashpoly

Answer 1

My hunch is that the variation in plain text words isn't high, and so the cyclic hash isn't chaotic enough.
Let's look at two strings "cat" and "dog".
cat
c 01100011
a 01100001
t 01110100
h starts at
00000000 00000000 00000000 01100011 (c)
and is then cycled to
00000000 00000000 00001100 01100000
then we add `a` to get
00000000 00000000 00001100 01100000
+ 01100001
= 00000000 00000000 00001100 11000001
which is then cycled to
00000000 00000001 10011000 00100000
then we add `t` to get
00000000 00000001 10011000 00100000
+ 01110100
= 00000000 00000001 10011000 10010100
we then return this number mod 41893 for 20810
Similarly, for dog
d 01100100
o 01101111
g 01100111
start:
00000000 00000000 00000000 01100100 (d)
cycled and added o:
00000000 00000000 00001100 11101111
cycled and added t:
00000000 00000001 10011110 01000111
ends up at 22269
Because the ASCII range is small, and the cycle algorithm uses the entire space of the unsigned int, it takes long strings to really push the hash into a completely different space. Especially the last character, which really dominates the final modulus operation.
Another way of looking at it: there's very little interaction with a 7-bit ASCII character and the previous 7-bit ASCII character after you shift 5 of those bits away and replace them with 0s, especially for shorter words.
Since the polynomial hash only uses the table size, it's chaotic "faster", even for smaller strings. It doesn't have to fill a whole int before it starts being really chaotic. A single ASCII character is much larger the table size.
That's my guess, anyway. I'd confirm this by checking to see which strings collide. My guess is strings of similar length are colliding the most with the cycle algorithm.