I'm trying to interface with a 3rd party system and they have provided a code sample to calculate a CRC value when sending text data.
The C code the vendor provided looks like this:
#define CRCRES 0xf0b8 /* residue for good verify */
#define DEBUG
unsigned crctbl[] = {0x0000, 0x1081, 0x2102, 0x3183,
0x4204, 0x5285, 0x6306, 0x7387,
0x8408, 0x9489, 0xa50a, 0xb58b,
0xc60c, 0xd68d, 0xe70e, 0xf78f};
/*
* This uses a 32 byte table to lookup the crc 4 bits at a time.
* The CRC CCITT is used.
*/
unsigned short calc_crc(unsigned char *ptr, unsigned length)
{
unsigned short crc;
unsigned short i;
unsigned char pos,ch;
crc = 0xffff; /* precondition crc */
for (i = 0; i < length; i++,ptr++) {
ch = *ptr;
pos = (crc ^ ch) & 15;
crc = ((crc >> 4) & 0x0fff) ^ crctbl[pos];
ch >>= 4;
pos = (crc^ch) & 15;
crc = ((crc >> 4) & 0xffff) ^ crctbl[pos];
}
crc = ~crc; /* post condition */
crc = (crc << 8) | (crc >> 8); /* bytewise reverse */
return crc;
}
/*
* tests the block of code containing the crc to verify it's
* content. This compares to the reversed and inverted
* residue.
*/
int test_crc(unsigned char *ptr, unsigned length)
{
unsigned short crc;
unsigned char arr [] = {'X','Y','Z'};
crc = calc_crc(arr,3);
printf("Calced crc of test to be %04x, (should be 470f)\n", crc);
return (crc == 0x470f);
}
I've copied this code and put in a sample C program. The test_crc method is not calculating the CRC to be 470f (its calculating it as DD7A).
I'm hoping someone can either verify that this code doesn't work as the vendor is saying it should or help me get test_crc to return the correct value.
Thanks for the help.
Problem solved. The specs from the vendor were incorrect. The correct checksum for XYZ is DD7A. The documentation was incorrect.
Add that to the fact that on the page before when they explain what data to pass to the calc_crc method to get the crc for an actual message is also incorrect it made it a tricky project.
Thanks for the help.
The calculation, as it goes, don't show any error that comes to my mind. Just review again the algorithm. In particular, looking for possible problems, note that constants being anded in different parts of the algorithm are 0x0fff and 0xffff. Maybe should it be 0x0fff and 0xfff0 to be more symmetric... Sometimes we think the real problem may come from another place, and forget to double check the obvious things... :)
Are the hardware platforms the same? Could it be you have different endianness?
(You have x86 they have 68000 cpu for example.)
Related
I am working on a data generator in order to generate binary files into a closed system for which we assume there is no risk of external attacks nor any malicious intent.
The idea is to find a way for these binaries to contain a quick and relatively reliable way to ensure they have not been accidentally corrupted in the toolchain so the final receiver of this binary can check the file integrity without any additional resource (such as file.sha1 containing the checksum).
As it may affect the solution, I must tell that the binary file can be 1 kB up to maybe 300 MB of data. There is a specific position in the file where I can place a checksum of any fixed length for all files. The position is already defined I can not change it but I can change the length.
So if the solution is to include 128 bytes of data at this position in order to suit any possible case then all binaries will contain this byte length at this position.
As it would be impossible to include a cryptographic hash of the file into it without the need to cut it off before checking, I read that CRC32 is a good way to achieve this goal. I have also took knowledge of utilities such as "spoof" or "CRC manipulator" but they seem not to satisfy my case.
Here is a quick example of what I need. Let's consider a binary file:
This position will never change
v
1011101100010010000000010110011100100110
^^^^^^^^
This is the fixed-length part dedicated to file integrity check
I would like to find a way to insert the right checksum so the complete file including the checksum has the overall same checksum. Maybe a known program already does this ?
Thanks for your support
You can still use a cryptographic hash, just copy out the hash and then zero out the hash block before checking the hash. Likewise, you'll need to ensure the hash block is all zeros before hashing the file.
You can use dd in a shell script to copy data into a specific byte position in a file. Or you can use standard random-access file I/O in any good programming language.
However, I would question whether this complexity is really necessary. The more standard solution would be to put the hash separately (either in a separate file, or at the very start or end of the file) which avoids this problem entirely. Putting the hash at the end of the file might make it slightly more inconvenient to read the file, depending on the file format, but putting the hash at the start of the file shouldn't have that problem.
You need to compute the CRC forwards and backwards to the insertion point, and put the exclusive-or of those two there. Then the CRC of the whole thing will be a constant. (Assuming no corruption.)
Here is example code:
// Example of the generation of a "middle" CRC, which is inserted somewhere in
// the middle of a sequence, where the CRC is generated such that the CRC of
// the complete sequence will be zero. This particular CRC has no pre or post
// processing.
//
// Placed into the public domain by Mark Adler, 11 May 2016.
#include <stddef.h> // for size_t
#include <stdint.h> // for uint32_t and uint64_t
#define POLY 0xedb88320 // CRC polynomial
// Byte-wise CRC tables for forward and reverse calculations.
uint32_t crc_forward_table[256];
uint32_t crc_reverse_table[256];
// Fill in CRC tables using bit-wise calculations.
void crc32_make_tables(void) {
for (uint32_t n = 0; n < 256; n++) {
uint32_t crc = n;
for (int k = 0; k < 8; k++)
crc = crc & 1 ? (crc >> 1) ^ POLY : crc >> 1;
crc_forward_table[n] = crc;
crc_reverse_table[crc >> 24] = (crc << 8) ^ n;
}
}
// Return the forward CRC of buf[0..len-1], starting with crc at the front.
uint32_t crc32(uint32_t crc, unsigned char *buf, size_t len) {
for (size_t n = 0; n < len; n++)
crc = (crc >> 8) ^ crc_forward_table[(crc ^ buf[n]) & 0xff];
return crc;
}
// Return the reverse CRC of buf[0..len-1], starting with crc at the end.
uint32_t crc32_reverse(uint32_t crc, unsigned char *buf, size_t len) {
while (len)
crc = (crc << 8) ^ crc_reverse_table[crc >> 24] ^ buf[--len];
return crc;
}
// Put a 32-bit value into a byte buffer in little-endian order.
void put4(uint32_t word, unsigned char *pos) {
pos[0] = word;
pos[1] = word >> 8;
pos[2] = word >> 16;
pos[3] = word >> 24;
}
#include <stdlib.h> // for random() and srandomdev()
// Fill dat[0..len-1] with uniformly random byte values. All of the bits from
// each random() call are used, except for possibly a few leftover at the end.
void ranfill(unsigned char *dat, size_t len) {
uint64_t ran = 1;
while (len) {
if (ran < 0x100)
ran = (ran << 31) + random();
*dat++ = ran;
ran >>= 8;
len--;
}
}
#include <stdio.h> // for printf()
#define LEN 1024 // length of the message without the CRC
// Demonstrate the generation of a middle-CRC, using the forward and reverse
// CRC computations. Verify that the CRC of the resulting sequence is zero.
int main(void) {
crc32_make_tables();
srandomdev();
unsigned char dat[LEN+4];
ranfill(dat, LEN/2);
put4(0, dat + LEN/2); // put zeros where the CRC will go
ranfill(dat + LEN/2 + 4, (LEN+1)/2);
put4(crc32(0, dat, LEN/2) ^ crc32_reverse(0, dat + LEN/2, (LEN+1)/2 + 4),
dat + LEN/2); // replace the zeros with the CRC
printf("%08x\n", crc32(0, dat, LEN+4));
return 0;
}
I'm trying to create a function that generates a CRC lookup table. I'm working with an 8051 micro-controller, and I'd rather do the table lookup method but at the same time, I'd rather have my computer generate the values which then I can load directly into the micro-controller. Most of this source code has been borrowed from: http://www.rajivchakravorty.com/source-code/uncertainty/multimedia-sim/html/crc8_8c-source.html
I only added in the "main" function
#include <stdio.h>
#define GP 0x107
#define DI 0x07
static unsigned char crc8_table[256];
static int made_table=0;
static void init_crc8()
{
int i,j;
unsigned char crc;
if (!made_table) {
for (i=0; i<256; i++) {
crc = i;
for (j=0; j<8; j++)
crc = (crc << 1) ^ ((crc & 0x80) ? DI : 0);
crc8_table[i] = crc & 0xFF;
}
made_table=1;
}
}
void crc8(unsigned char *crc, unsigned char m)
{
if (!made_table)
init_crc8();
*crc = crc8_table[(*crc) ^ m];
*crc &= 0xFF;
}
int main()
{
unsigned char crc[1];
crc8(crc,'S');
printf("S=%x\n",crc[0]); //different hex code almost every time
crc8(crc,'T');
printf("T=%x\n",crc[0]); //different hex code almost every time
return 0;
}
When I execute the program, I expected the same values on the screen but the hex codes after the printed equals signs changed on nearly every program execution.
What can I do to correct that issue? I don't want to be collecting incorrect CRC values.
crc[0] is not initialized. You need crc[0] = 0; or *crc = 0; before calling crc8() with crc. Then you won't get random answers coming from the random initial contents of crc[0].
You don't need the *crc &= 0xff; in crc8(). If char is eight bits, then it does nothing. If you have an odd architecture where char is more than eight bits, then you need to do *crc = crc8_table[((*crc) ^ m) & 0xff]; to assure that you don't go outside the bounds of the table. (Only the low eight bits of m will be used in the CRC calculation.) The contents of the table have already been limited to eight bits, so in any case you don't need a final & 0xff.
You may need a different initial value than zero, and you may need to exclusive-or the final CRC value with something, depending on the definition of the CRC-8 that you want. In the RevEng catalog of CRC's, there are two 8-bit CRCs with that polynomial that are not reflected. Both happen to start with an initial value of zero, but one is exclusive-ored with 0x55 at the end. Also the CRC definition you need may be reflected, in which case the shift direction changes and the polynomial is flipped. If your CRC-8 needs to be interoperable with some other software, then you need to find out the full definition of the CRC being used.
Passing a pointer seems like an odd choice here. It would be more efficient to just pass and return the CRC value directly. E.g. unsigned crc8(unsigned crc, unsigned ch) {, which would apply the eight bits in ch to the CRC crc, and return the new value. Note that you do not need to make the CRC value a char. unsigned is generally what C routines most efficiently take as an argument and return. In fact usually the first argument is passed in a register and returned in the same register.
Usually one computes a CRC on a message consisting of a series of bytes. It would be more efficient to have a routine that does the whole message with a loop, so that you don't need to check to see if the table has been built yet for every single byte of the message.
In the main, crc[0] has not been initialized. As a result, in crc8, *crc in the expression (*crc) ^ m is uninitialized, hence your random values.
Fix: initialize crc[0]. Something like
unsigned char crc[1] = { 0 };
I am trying to create a CRC-15 check in c and the output is never correct for each line of the file. I am trying to output the CRC for each line cumulatively next to each line. I use: #define POLYNOMIAL 0xA053 for the divisor and text for the dividend. I need to represent numbers as 32-bit unsigned integers. I have tried printing out the hex values to keep track and flipping different shifts around. However, I just can't seem to figure it out! I have a feeling it has something to do with the way I am padding things. Is there a flaw to my logic?
The CRC is to be represented in four hexadecimal numbers, that sequence will have four leading 0's. For example, it will look like 0000xxxx where the x's are the hexadecimal digits. The polynomial I use is 0xA053.
I thought about using a temp variable and do 4 16 bit chunks of code per line every XOR, however, I'm not quite sure how I could use shifts to accomplish this so I settled for a checksum of the letters on the line and then XORing that to try to calculate the CRC code.
I am testing my code using the following input and padding with . until the string is of length 504 because that is what the pad character needs to be via the requirements given:
"This is the lesson: never give in, never give in, never, never, never, never - in nothing, great or small, large or petty - never give in except to convictions of honor and good sense. Never yield to force; never yield to the apparently overwhelming might of the enemy."
The CRC of the first 64 char line ("This is the lesson: never give in, never give in, never, never,) is supposed to be 000015fa and I am getting bfe6ec00.
My logic:
In CRCCalculation I add each character to a 32-bit unsigned integer and after 64 (the length of one line) I send it into the XOR function.
If it the top bit is not 1, I shift the number to the left one
causing 0s to pad the right and loop around again.
If the top bit is 1, I XOR the dividend with the divisor and then shift the dividend to the left one.
After all calculations are done, I return the dividend shifted to the left four ( to add four zeros to the front) to the calculation function
Add result to the running total of the result
Code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#include <ctype.h>
#define POLYNOMIAL 0xA053
void crcCalculation(char *text, int length)
{
int i;
uint32_t dividend = atoi(text);
uint32_t result;
uint32_t sumText = 0;
// Calculate CRC
printf("\nCRC 15 calculation progress:\n");
i = length;
// padding
if(i < 504)
{
for(; i!=504; i++)
{
// printf("i is %d\n", i);
text[i] = '.';
}
}
// Try calculating by first line of crc by summing the values then calcuating, then add in the next line
for (i = 0; i < 504; i++)
{
if(i%64 == 0 && i != 0)
{
result = XOR(POLYNOMIAL, sumText);
printf(" - %x\n",result);
}
sumText +=(uint32_t)text[i];
printf("%c", text[i]);
}
printf("\n\nCRC15 result : %x\n", result);
}
uint32_t XOR(uint32_t divisor, uint32_t dividend)
{
uint32_t divRemainder = dividend;
uint32_t currentBit;
// Note: 4 16 bit chunks
for(currentBit = 32; currentBit > 0; --currentBit)
{
// if topbit is 1
if(divRemainder & 0x80)
{
//divRemainder = (divRemainder << 1) ^ divisor;
divRemainder ^= divisor;
printf("%x %x\n", divRemainder, divisor);
}
// else
// divisor = divisor >> 1;
divRemainder = (divRemainder << 1);
}
//return divRemainder; , have tried shifting to right and left, want to add 4 zeros to front so >>
//return divRemainder >> 4;
return divRemainder >> 4;
}
The first issue I see is the top bit check, it should be:
if(divRemainder & 0x8000)
The question doesn't state if the CRC is bit reflected (xor data into low order bits of CRC, right shift for cycle) or not (xor data into high order bits of CRC, left shift for cycle), so I can't offer help for the rest of the code.
The question doesn't state the initial value of CRC (0x0000 or 0x7fff), or if the CRC is post complemented.
The logic for a conventional CRC is:
xor a byte of data into the CRC (upper or lower bits)
cycle the CRC 8 times (or do a table lookup)
After generating the CRC for an entire message, the CRC can be appended to the message. If a CRC is generated for a message with the appended CRC and there are no errors, the CRC will be zero (or a constant value if the CRC is post complemented).
here is a typical CRC16, extracted from: <www8.cs.umu.se/~isak/snippets/crc-16.c>
#define POLY 0x8408
/*
// 16 12 5
// this is the CCITT CRC 16 polynomial X + X + X + 1.
// This works out to be 0x1021, but the way the algorithm works
// lets us use 0x8408 (the reverse of the bit pattern). The high
// bit is always assumed to be set, thus we only use 16 bits to
// represent the 17 bit value.
*/
unsigned short crc16(char *data_p, unsigned short length)
{
unsigned char i;
unsigned int data;
unsigned int crc = 0xffff;
if (length == 0)
return (~crc);
do
{
for (i=0, data=(unsigned int)0xff & *data_p++;
i < 8;
i++, data >>= 1)
{
if ((crc & 0x0001) ^ (data & 0x0001))
crc = (crc >> 1) ^ POLY;
else crc >>= 1;
}
} while (--length);
crc = ~crc;
data = crc;
crc = (crc << 8) | (data >> 8 & 0xff);
return (crc);
}
Since you want to calculate a CRC15 rather than a CRC16, the logic will be more complex as cannot work with whole bytes, so there will be a lot of bit shifting and ANDing to extract the desire 15 bits.
Note: the OP did not mention if the initial value of the CRC is 0x0000 or 0x7FFF, nor if the result is to be complemented, nor certain other criteria, so this posted code can only be a guide.
I found an easy to use implementation of CRC algorithms here. It includes table based and bit-wise algorithms. The code seems to work fine but there is an important limitation of table based algorithms. This is the relevant code:
unsigned long reflect (unsigned long crc, int bitnum) {
unsigned long i, j=1, crcout=0;
for (i=(unsigned long)1<<(bitnum-1); i; i>>=1) {
if (crc & i) crcout|=j;
j<<= 1;
}
return (crcout);
}
void generate_crc_table() {
// make CRC lookup table used by table algorithms
int i, j;
unsigned long bit, crc;
for (i=0; i<256; i++) {
crc=(unsigned long)i;
if (refin) crc=reflect(crc, 8);
crc<<= order-8;
for (j=0; j<8; j++) {
bit = crc & crchighbit;
crc<<= 1;
if (bit) crc^= polynom;
}
if (refin) crc = reflect(crc, order);
crc&= crcmask;
crctab[i]= crc;
}
}
unsigned long crctablefast (unsigned char* p, unsigned long len) {
// fast lookup table algorithm without augmented zero bytes, e.g. used in pkzip.
// only usable with polynom orders of 8, 16, 24 or 32.
unsigned long crc = crcinit_direct;
if (refin) crc = reflect(crc, order);
if (!refin) while (len--) crc = (crc << 8) ^ crctab[ ((crc >> (order-8)) & 0xff) ^ *p++];
else while (len--) crc = (crc >> 8) ^ crctab[ (crc & 0xff) ^ *p++];
if (refout^refin) crc = reflect(crc, order);
crc^= crcxor;
crc&= crcmask;
return(crc);
}
Please note code comments for table functions say:
only usable with polynom orders of 8, 16, 24 or 32.
Are table based algorithms generally limited to widths that are multiples of eight (especially table algorithms that use 16 and 32 bit tables)?
Is it possible to implement a table based CRC algorithm that accepts any CRC widths (not only multiples of 8)? How?
Yes, you can implement table-based CRCs for any width polynomial. See the output of crcany for example of table-based implementations for, for example, 5-bit, 13-bit, and 31-bit CRCs.
There is nothing tricky about this.
I have an Arduino serial connection with 4 bits per transmission left for a check sum. I have never done a check sum algorithm before and am sadly lost right now.
CRC-4 implementation in C# has already started a question but the answer isn't really helping me go forward.
Input should be an integer of which only 10 bits are important.
Output should be 4 bits of a byte.
Can someone explain to me how I can adapt the answer from the other question or point me another direction?
byte checksum(int message)
{
//Check sum algorithm
}
well the other algorithm is easily translatable to pure C:
uint8_t calculate(byte[] bytes) {
uint16_t crc = 0xFFFF; // initial value
// loop, calculating CRC for each byte of the string
for (uint8_t byteIndex = 0; byteIndex < bytes.Length; ++byteIndex) {
uint8_t bit = 0x80; // initialize bit currently being tested
for (uint8_t bitIndex = 0; bitIndex < 8; ++bitIndex) {
bool xorFlag = ((crc & 0x8000) == 0x8000);
crc <<= 1;
if (((bytes[byteIndex] & bit) ^ (uint8_t)0xff) != (uint8_t)0xff)
crc = crc + 1;
if (xorFlag)
crc = crc ^ 0x1021;
bit >>= 1;
}
}
return (uint8_t)crc;
}
the only difference being the use of the stdint.h types.
I also changed the type of crc to be exactly a 16 bits unsigned, and for the
indexes, only for sparing some arduino memory which is precious (every byte counts
when you got only 2.5k of RAM! :-) )
Though, I did neither test or proof read that code, so it should be as good as the
original C# one. If it's buggy, that one will be as well.
EDIT: As the OP added in comment, this resource is a good explanation of how the above CRC algorithm work: http://www.barrgroup.com/Embedded-Systems/How-To/CRC-Calculation-C-Code.
HTH