Optimization O(n^2) to O(n) (Unsorted String) - c

Question

I'm having an optimization problem here. I would like to have this code running in O(n), which I tried for several hours now.
Byte-arrays c contains a string, e contains the same string, but sorted. Int-arrays nc and ne contain the indexes within the string, eg
c:
s l e e p i n g
nc:
0 0 0 1 0 0 0 0
e:
e e g i l n p s
ne:
0 1 0 0 0 0 0 0
The problem now is that get_next_index is linear - is there a way to solve this?
void decode_block(int p) {
BYTE xj = c[p];
int nxj = nc[p];
for (int i = 0; i < block_size; i++) {
result[i] = xj;
int q = get_next_index(xj, nxj, c, nc);
xj = e[q];
nxj = ne[q];
}
fwrite(result, sizeof (BYTE), block_size, stdout);
fflush(stdout);
}
int get_next_index(BYTE xj, int nxj, BYTE* c, int* nc) {
int i = 0;
while ( ( xj != c[i] ) || ( nxj != nc[i] ) ) {
i++;
}
return i;
}
This is part of an Burrows-Wheeler implementation
It starts with
xj = c[p]
nxj = nc[p]
Next I have to block_size (= length c = length nc = length e = length ne) times
store the result xj in result
find the number index for which c[i] == xj
xj is now e[i]
ne and nc are only used to make sure that every character in e and c is unique (e_0 != e_1).

Answer 1

Since your universe (i.e. a char) is small, I think you can get away with linear time. You need a linked list and any sequence container a lookup table for this.
First your go through your sorted string and populate a lookup table that allows you to find the first list element for a given character. For instance, your lookup table could look like std::array<std::list<size_t>,(1<<sizeof(char))> lookup. If you don't want a list, you can also use an std::deque or even an std::pair<std::vector,size_t> while the second item represents the index of the first valid entry of the vector (that way you don't need to pop the element later on, but just increment the index).
So for each element c in your sorted string you append that to you container in lookup[c].
Now, when you iterate over your unsorted array, for each element, you can lookup the corresponding index in your lookup table. Once you're done, you pop the front element in the lookup table.
All in all this is linear time and space.
To clarify; When initialising the lookup table:
// Instead of a list, a deque will likely perform better,
// but you have to test this yourself in your particular case.
std::array<std::list<size_t>,(1<<sizeof(char))> lookup;
for (size_t i = 0; i < sortedLength; i++) {
lookup[sorted[i]].push_back(i);
}
When finding the "first index" for the index i in the unsorted array:
size_t const j = lookup[unsorted[i]].front();
lookup[unsorted[i]].pop_front();
return j;

Answer 2

Scan xj and nxj once and build a lookup table. This is a two O(n) operations.
The most sensible way would be to have a binary tree, sorted on the value of xj or nxj. The node would contain your sought index. This would reduce your lookup to O(lg n).

Answer 3

Here is my complete implementation of the Burrowes-Wheeler transform:
u8* bwtCompareBuf;
u32 bwtCompareLen;
s32 bwtCompare( const void* v1, const void* v2 )
{
u8* c1 = bwtCompareBuf + ((u32*)v1)[0];
u8* c2 = bwtCompareBuf + ((u32*)v2)[0];
for ( u32 i = 0; i < bwtCompareLen; i++ )
{
if ( c1[i] < c2[i] ) return -1;
if ( c1[i] > c2[i] ) return +1;
}
return 0;
}
void bwtEncode( u8* inputBuffer, u32 len, u32& first )
{
s8* tmpBuf = alloca( len * 2 );
u32* indices = new u32[len];
for ( u32 i = 0; i < len; i++ ) indices[i] = i;
bwtCompareBuf = tmpBuf;
bwtCompareLen = len;
qsort( indices.data(), len, sizeof( u32 ), bwtCompare );
u8* tbuf = (u8*)tmpBuf + ( len - 1 );
for ( u32 i = 0; i < len; i++ )
{
u32 idx = indices[i];
if ( idx == 0 ) idx = len;
inputBuffer[i] = tbuf[idx];
if ( indices[i] == 1 ) first = i;
}
delete[] indices;
}
void bwtDecode( u8* inputBuffer, u32 len, u32 first )
{
// To determine a character's position in the output string given
// its position in the input string, we can use the knowledge about
// the fact that the output string is sorted. Each character 'c' will
// show up in the output stream in in position i, where i is the sum
// total of all characters in the input buffer that precede c in the
// alphabet, plus the count of all occurences of 'c' previously in the
// input stream.
// compute the frequency of each character in the input buffer
u32 freq[256] = { 0 };
u32 count[256] = { 0 };
for ( u32 i = 0; i < len; i++ )
freq[inputBuffer[i]]++;
// freq now holds a running total of all the characters less than i
// in the input stream
u32 sum = 0;
for ( u32 i = 0; i < 256; i++ )
{
u32 tmp = sum;
sum += freq[i];
freq[i] = tmp;
}
// Now that the freq[] array is filled in, I have half the
// information needed to position each 'c' in the input buffer. The
// next piece of information is simply the number of characters 'c'
// that appear before this 'c' in the input stream. I keep track of
// that information in the count[] array as I go. By adding those
// two numbers together, I get the destination of each character in
// the input buffer, and I just write it directly to the destination.
u32* trans = new u32[len];
for ( u32 i = 0; i < len; i++ )
{
u32 ch = inputBuffer[i];
trans[count[ch] + freq[ch]] = i;
count[ch]++;
}
u32 idx = first;
s8* tbuf = alloca( len );
memcpy( tbuf, inputBuffer, len );
u8* srcBuf = (u8*)tbuf;
for ( u32 i = 0; i < len; i++ )
{
inputBuffer[i] = srcBuf[idx];
idx = trans[idx];
}
delete[] trans;
}
The decode in O(n).