I am trying to recover raw data from an older measurement instrument, that is interfaced through a printer port.
For example, the instruments software will produce an text output file like this:
S 11/08/08 22:27:58 100 2 U 061
D ___^PR_^_^_]PP_]_^_]_^_____^_^_____^_[_\_\_[_Z_Z_X
D _W_U_T_Q^]^]^Z^V^S^T^S]]]Y]U]R]T]Q]V]Z]\]]^R^]_ZPX
D QSQYQ^RSRYSQSWS\S]SZSWSSSPR\RZRXRTQ^QWQPP[PUPRPQ_^
D _\_]_^_____\_\_Z_X_W_Y_X_X_Z_W_U_V_W_X_[_X_W_W_W
F 2
S 11/08/08 22:35:03 100 2 E 049
D QSQQP_P^QPQPQRQUQUQUQVQZQ[Q\Q]RSR\STSXSWSQR_SQSRR[
D RTQ_QWQUQWQUQZRSSQR]RTRSRQQZQRPZPVPTPTPSPWPTPQPQ_^
D _^_^__PPPPPP__PP__PR__PPPQ_____^_]_]PP_^_]_]_]_Y_^
D ___^_^_\_______^PP__PRPQPPPRPP__PPPP___]_^_^__PP
F 2
The "S" line is all good - provides the appropriate time the measurement
was taken along with some other values.
I'm interested in recovering whatever is hidden in
the "D" lines. The software generates a plot using this data, but
does not provide the raw data.
The only code I have detailing the data encoding contains the comment:
/* Packs the 8-bit data into two 7-bit ASCII chars, encoding the channel
* number into it as well, in the format:
*
* 1CCMMMM and 1CCLLLL, where CC = chn, MMMM/LLLL = Most/Least sig nibble
*/
I can send the actually packing code too if it helps - just trying to keep the
question as small as possible.
Any help - even a point in the right direction would be appreciated...
The encoding is actually pretty clever*: every combination of two letters (2*8 bits or 2*7 bits, depending how you look at it) is a single measurement. The comment tells us how the encoding works. For example, if we take 'QS' as an example:
Pattern: 01CCMMMM 01CCLLLL
Example: 01010001 01010011 = Q S
Channel: ..CC.... ..CC....
..01.... ..01.... = Channel 1
Data: ....0001 ....0011 = 10011 = 19
You simply have to take the bits labeled M and the bits labeled L, put them after each other, treat the whole thing as a single-byte number and you've got the original data. Conversely, extract the bits labeled C to get the channel number.
Here's an example of how you could parse a single measurement, assuming two bytes of input are in a and b:
/* To get the channel, mask with 00110000 = 0x30 then shift */
char channel = (a & 0x30) >> 4;
/* To get data, mask both with 00001111 = 0xF then combine */
char orgdata = ((a & 0xF) << 4) | (b & 0xF);
Putting all that together here gives the following data for the first 'frame' in your example, all on channel 1:
I'm hoping that matches what you're seeing on your plot :)
*: I'm not being sarcastic, either - this encoding packs 10 bits of useful data into 14 bits of usable space, while being a good deal simpler than something like base64 and probably faster.
Related
I'm querying an ADXL362 Digital Output MEMS Accelerometer for its axis data which it holds as two 8 bit registers which combine to give a 12 bit value and I'm trying to figure out how to combine those values. I've never been good at bitwise manipulation so any help would be greatly appreciated. I would imagine it is something like this:
number = Z_data_H << 8 | Z_data_L;
number = (number & ~(1<<13)) | (0<<13);
number = (number & ~(1<<14)) | (0<<14);
number = (number & ~(1<<15)) | (0<<15);
number = (number & ~(1<<16)) | (0<<16);
ADXL362 data sheet (page 26)
Z axis data register
Your first line should be what you need:
int16_t number;
number = (Z_data_H << 8) | Z_data_L;
The sign-extension bits mean that you can read the value as if it was a 16-bit signed integer. The value will simply never be outside the range of a 12-bit integer. It's important that you leave those bits intact in order to handle negative values correctly.
You just have to do:
signed short number;
number = Z_data_H << 8 | Z_data_L;
The shift left by 8 bit combined with the lower bits you already
had figured out are combining the 2 bytes correctly. Just use the appropriate data size to have the C code recoginize the sign of the 12 bit number correctly.
Note that short not necessarily refers to a 16bit value, depending on your compiler and architecture - so, you might want to attempt to that.
For the project i'm working on, a "word" is defined as 10 bit length, and as according to what my program does, I need to update specific bits in this word, with binary numbers (of course up to the limit of the length of the bits). My problem is that I don't know how to create these bits, and after that how to read them.
For example, "word" is set like this:
bits 0-1 - representing something A - can get values between 0-3.
bits 2-3 - representing something B - can get values between 0-3.
bits 4-5 - C - values 0-3.
bits 6-9 - D - values 0-15.
and as my program running, I need to decide what to fill in each group of bit. After that, when my word is completely full, I need to analyze the results, meaning to go over the full word, and understand from bits 0-1 what A is representing, from bits 2-3 what B is representing, and so on..
another problem is that bit number 9 is the most significant bit, which mean the word is filling up from bits 6-9 to 4-5 to 2-3 to 0-1, and later on printed from bit 9 to 0, and not as a regular array.
I tried to do it with struct of bit-fields, but the problem is that while a "word" is always 10 bits length, the sub-division as mentioned above is only one example of a "word". it can also be that the bits 0-1 representing something, and bits 2-9 something else.
I'm a bit lost and don't know how to do it, and I'll be glad if someone can help me with that.
Thanks!
Just model a "word" as an uint16_t, and set the appropriate bits.
Something like this:
typedef uint16_t word;
word word_set_A(word w, uint8_t a)
{
w &= ~3;
return w | (a & 3);
}
uint8_t word_get_A(word w)
{
return w & 3;
}
word word_set_B(word w, uint8_t b)
{
w &= ~0xc0;
return w | ((b & 3) << 2);
}
... and so on.
I'm sampling at high frequencies and need to transmit the 10-bit ADC value via UART out of my Arduino.
By default, it uses a byte per character. So if doing an analogRead would yield the value of "612", it would send via UART "6" as one byte, "1" as one byte, "2" as one byte, and the line terminator as the last byte.
Given that my sampling rate is truncated by this communication, it's important that it's as efficient and uniform as possible, so I'm trying to force it to use two bytes to transmit that data, doesn't matter what the data actual is (by default it would use three bytes to transmit "23", four bytes to transmit "883" and five bytes to transmit "1001").
Currently, I'm doing something like this, which is the best way I've found:
int a = 600; //Just an example
char high = (char)highByte(a);
char low = (char)lowByte(a);
Serial.print(high);
Serial.println(low);
Currently this uses three bytes (including \n) regardless of the value. Is there an even more efficient method?
Just printing it with something like
Serial.print(foo, BIN);
Doesn't work at all. It actually uses one byte per every single bit of the binary representation of foo, which is quite silly.
I might be missing something, but why don't you use Serial.write(byte)?
You can use a methode like this one:
void writeIntAsBinary(int value){
Serial.write(lowByte(value));
Serial.write(highByte(value));
}
what are you planning to do with the data on your computer?
If you're sending binary data over a serial line, you really shouldn't confuse everything by using a text-style linefeed separator.
On the other hand, it's kind of hard (for the other end) to know which byte is which, without some kind of synchronization help.
But, since you only have 10 bits of payload, but send 16 bits of data, you can "do a UTF-8" and use a free bit to signal "start of value". This will require using only 7 bits of each 8-bit byte for your payload, but that's fine since 7 + 7 = 14 which is way more than 10. We can let the 8th bit mean "this is the high byte of a new pair of bytes":
const int a = 600;
const unsigned char high = ((a >> 7) & 0x7f) | 0x80;
const unsigned char low = (a & 0x7f);
Serial.print(high);
Serial.print(low);
In the above, the two bytes transmitted will be:
high == ((600 >> 7) & 0x7f) | 0x80 == 4 | 0x80 == 0x84
low == (600 & 0x7f) == 88 == 0x58
The receiver will have to do the above in reverse:
const int value = ((high & 0x7f) << 7) | low;
This should work, and uses the most-significant bit of the high byte, which is sent first, to signify that that is indeed the high byte. The low byte will never have the MSB set.
Disclaimer: I am asking these questions in relation to an assignment. The assignment itself calls for implementing a bitmap and doing some operations with that, but that is not what I am asking about. I just want to understand the concepts so I can try the implementation for myself.
I need help understanding bitmaps/bit arrays and bitwise operations. I understand the basics of binary and how left/right shift work, but I don't know exactly how that use is beneficial.
Basically, I need to implement a bitmap to store the results of a prime sieve (of Eratosthenes.) This is a small part of a larger assignment focused on different IPC methods, but to get to that part I need to get the sieve completed first. I've never had to use bitwise operations nor have I ever learned about bitmaps, so I'm kind of on my own to learn this.
From what I can tell, bitmaps are arrays of a bit of a certain size, right? By that I mean you could have an 8-bit array or a 32-bit array (in my case, I need to find the primes for a 32-bit unsigned int, so I'd need the 32-bit array.) So if this is an array of bits, 32 of them to be specific, then we're basically talking about a string of 32 1s and 0s. How does this translate into a list of primes? I figure that one method would evaluate the binary number and save it to a new array as decimal, so all the decimal primes exist in one array, but that seems like you're using too much data.
Do I have the gist of bitmaps? Or is there something I'm missing? I've tried reading about this around the internet but I can't find a source that makes it clear enough for me...
Suppose you have a list of primes: {3, 5, 7}. You can store these numbers as a character array: char c[] = {3, 5, 7} and this requires 3 bytes.
Instead lets use a single byte such that each set bit indicates that the number is in the set. For example, 01010100. If we can set the byte we want and later test it we can use this to store the same information in a single byte. To set it:
char b = 0;
// want to set `3` so shift 1 twice to the left
b = b | (1 << 2);
// also set `5`
b = b | (1 << 4);
// and 7
b = b | (1 << 6);
And to test these numbers:
// is 3 in the map:
if (b & (1 << 2)) {
// it is in...
You are going to need a lot more than 32 bits.
You want a sieve for up to 2^32 numbers, so you will need a bit for each one of those. Each bit will represent one number, and will be 0 if the number is prime and 1 if it is composite. (You can save one bit by noting that the first bit must be 2 as 1 is neither prime nor composite. It is easier to waste that one bit.)
2^32 = 4,294,967,296
Divide by 8
536,870,912 bytes, or 1/2 GB.
So you will want an array of 2^29 bytes, or 2^27 4-byte words, or whatever you decide is best, and also a method for manipulating the individual bits stored in the chars (ints) in the array.
It sounds like eventually, you are going to have several threads or processes operating on this shared memory.You may need to store it all in a file if you can't allocate all that memory to yourself.
Say you want to find the bit for x. Then let a = x / 8 and b = x - 8 * a. Then the bit is at arr[a] & (1 << b). (Avoid the modulus operator % wherever possible.)
//mark composite
a = x / 8;
b = x - 8 * a;
arr[a] |= 1 << b;
This sounds like a fun assignment!
A bitmap allows you to construct a large predicate function over the range of numbers you're interested in. If you just have a single 8-bit char, you can store Boolean values for each of the eight values. If you have 2 chars, it doubles your range.
So, say you have a bitmap that already has this information stored, your test function could look something like this:
bool num_in_bitmap (int num, char *bitmap, size_t sz) {
if (num/8 >= sz) return 0;
return (bitmap[num/8] >> (num%8)) & 1;
}
This question is NOT about "How do i bitwise permutation" We now how to do that, what we are looking for is a faster way with less cpu instructions, inspired by the bitslice implementation of sboxes in DES
To speed up some cipher code we want to reduce the amount of permutation calls. The main cipher functions do multiple bitwise permutations based on lookup arrays. As the permutation operations are only bitshifts,
Our basic idea is to take multiple input values, that need the same permutation, and shift them in parallel. For example, if input bit 1 must be moved to output bit 6.
Is there any way to do this? We have no example code right now, because there is absolutly no idea how to accomplish this in a performant way.
The maximum value size we have on our plattforms are 128bit, the longest input value is 64bit.Therefore the code must be faster, then doing the whole permutation 128 times.
EDIT
Here is a simple 8bit example of a permutation
+---+---+---+---+---+---+---+---+
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | <= Bits
+---+---+---+---+---+---+---+---+
+---+---+---+---+---+---+---+---+
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | <= Input
+---+---+---+---+---+---+---+---+
| 3 | 8 | 6 | 2 | 5 | 1 | 4 | 7 | <= Output
+---+---+---+---+---+---+---+---+
The cipher makes usage of multiple input keys. It's a block cipher, so the same pattern must be applied to all 64bit blocks of the input.
As the permutations are the same for each input block, we want to process multiple input blocks in one step / to combine the operations for multiple input sequences. Instead of moving 128times one bit per call, moving 1 time 128bit at once.
EDIT2
We could NOT use threads, as we have to run the code on embedded systems without threading support. Therefore we also have no access on external libraries and we have to keep it plain C.
SOLUTION
After testing and playing with the given answers we have done it the following way:
We are putting the single bits of 128 64bit values on a uint128_t[64]* array.
For permutation we have just to copy pointers
After all is done, we revert the first operation and get 128 permuted values back
Yeah, it is realy that simple. We was testing this way early in the project, but it was too slow. It seems we had a bug in the testcode.
Thank you all, for the hints and the patience.
You could make Stan's bit-by-bit code faster by using eight look-up tables mapping bytes to 64-bit words. To process a 64-bit word from input, split it into eight bytes and look up each from a different look-up table, then OR the results. On my computer the latter is 10 times faster than the bit-by-bit approach for 32-bit permutations. Obviously if your embedded system has little cache, then 32 kB 16 kB of look-up tables may be a problem. If you process 4 bits at a time, you only need 16 look-up tables of 16*8=128 bytes each, i.e. 2 kB of look-up tables.
EDIT: The inner loop could look something like this:
void permute(uint64_t* input, uint64_t* output, size_t n, uint64_t map[8][256])
{
for (size_t i = 0; i < n; ++i) {
uint8_t* p = (uint8_t*)(input+i);
output[i] = map[0][p[0]] | map[1][p[1]] | map[2][p[2]] | map[3][p[3]]
| map[4][p[4]] | map[5][p[5]] | map[6][p[6]] | map[7][p[7]];
}
}
I think you might be looking for a bit-slicing implementation. This is how the fastest DES-cracking impelmentations work. (Or it was before SSE instructions existed, anyway.)
The idea is to write your function in a "bit-wise" manner, representing each output bit as a Boolean expression over the input bits. Since each output bit depends only on the input bits, any function can be represented this way, even things like addition, multiplication, or S-box lookups.
The trick is to use the actual bits of a single register to represent a single bit from multiple input words.
I will illustrate with a simple four-bit function.
Suppose, for example, you want to take four-bit inputs of the form:
x3 x2 x1 x0
...and for each input, compute a four-bit output:
x2 x3 x2^x3 x1^x2
And you want to do this for, say, eight inputs. (OK for four bits a lookup table would be fastest. But this is just to illustrate the principle.)
Suppose your eight inputs are:
A = a3 a2 a1 a0
B = b3 b2 b1 b0
...
H = h3 h2 h1 h0
Here, a3 a2 a1 a0 represent the four bits of the A input, etc.
First, encode all eight inputs into four bytes, where each byte holds one bit from each of the eight inputs:
X3 = a3 b3 c3 d3 e3 f3 g3 h3
X2 = a2 b2 c2 d2 e2 f2 g2 h2
X1 = a1 b1 c1 d1 e1 f1 g1 h1
X0 = a0 b0 c0 d0 e0 f0 g0 h0
Here, a3 b3 c3 ... h3 is the eight bits of X3. It consists of the high bits of all eight inputs. X2 is the next bit from all eight inputs. And so on.
Now to compute the function eight times in parallel, you just do:
Y3 = X2;
Y2 = X3;
Y1 = X2 ^ X3;
Y0 = X1 ^ X2;
Now Y3 holds the high bits from all eight outputs, Y2 holds the next bit from all eight outputs, and so on. We just computed this function on eight different inputs using only four machine instructions!
Better yet, if our CPU is 32-bit (or 64-bit), we could compute this function on 32 (or 64) inputs, still using only four instructions.
Encoding the input and decoding the output to/from the "bit slice" representation takes some time, of course. But for the right sort of function, this approach offers massive bit-level parallelism and thus a massive speedup.
The basic assumption is that you have many inputs (like 32 or 64) on which you want to compute the same function, and that the function is neither too hard nor too easy to represent as a bunch of Boolean operations. (Too hard makes the raw computation slow; too easy makes the time dominated by the bit-slice encoding/decoding itself.) For cryptography in particular, where (a) the data has to go through many "rounds" of processing, (b) the algorithm is often in terms of bits munging already, and (c) you are, for example, trying many keys on the same data... It often works pretty well.
It seems difficult to do the permutation in only one call. A special case of your problem, reversing bits in an integer, needs more than one 'call' (what do you mean by call?). See Bit Twiddling Hacks by Sean for information of this example.
If your mapping pattern is not complicated, maybe you can find a fast way to calculate the answer:) However, I don't know whether you like this direct way:
#include <stdio.h>
unsigned char mask[8];
//map bit to position
//0 -> 2
//1 -> 7
//2 -> 5
//...
//7 -> 6
unsigned char map[8] = {
2,7,5,1,4,0,3,6
};
int main()
{
int i;
//input:
//--------------------
//bit 7 6 5 4 3 2 1 0
//--------------------
//val 0 0 1 0 0 1 1 0
//--------------------
unsigned char input = 0x26;
//so the output should be 0xA1:
// 1 0 1 0 0 0 0 1
unsigned char output;
for(i=0; i<8; i++){ //initialize mask once
mask[i] = 1<<i;
}
//do permutation
output = 0;
for(i=0; i<8; i++){
output |= (input&mask[i])?mask[map[i]]:0;
}
printf("output=%x\n", output);
return 0;
}
Your best bet would be to look into some type of threading scheme ... either you can use a message-passing system where you send each block to a fixed set of worker threads, or you can possibly setup a pipeline with non-locking single producer/consumer queues that perform multiple shifts in a "synchronous" manner. I say "synchronous" because a pipeline on a general-purpose CPU would not be a truly synchronous pipeline operation like you would have on a fixed-function device, but basically for a given "slice" of time, each thread would be working on one stage of the multi-stage problem at the same time, and you would "stream" the source data into and out of the pipeline.