Related
I have a struct of different data types, which should be written into the non-volatile memory. It is possible to write it word after word, but I'm unsure, how I can read a word from my struct.
The struct looks like this:
typedef struct
{
uint8_t value1;
int16_t value2;
uint32_t value3;
float value4;
} __attribute__(( aligned(4))) test_struct_t;
How is it possible, to write word by word from this struct? With the aligned(4) attribute this should be possible, right?
How is it possible, to write word by word from this struct? With the aligned(4) attribute this should be possible, right?
Hmm, no, the answer to that is that writing the struct to a non-volatile storage system using 32bit "words" would make the resulting data unportable across systems due to possible Byte Endianness concerns.
EDIT: If the data isn't going anywhere
If the stored data isn't leaving the system, Endianness isn't an issue and that part of my original answer (further on) can be safely ignored.
I would still consider manually marking the padding in the struct and updating the naming as detailed below (see: Ease of use / Portability).
Also, if the NVM has a memory address and the standard library is available (or a compiler builtin function) - memcpy will be a simple solution and there's no need to worry about alignments and other implementation details.
Can we use a uint32_t *?
The issue is that if we use a pointer that violates the standard (invokes undefined behavior), than the compiler is free to assume that the data in the struct was never (properly) accessed and optimize away our read/write operations.
For example, this might leave us with old data in the struct even though we thought we performed a "read" operation (that was optimized away).
There is a good solution offered in #Lundin's answer.
Lundin's solution is one way to deal with the restrictions stated in section 6.5 (paragraph 7) of the C11 standard - restrictions that C compilers rely upon when optimizing your code:
An object shall have its stored value accessed only by an lvalue expression that has one of the following types:
— a type compatible with the effective type of the object,
— a qualified version of a type compatible with the effective type of the object,
...
— an aggregate or union type that includes one of the aforementioned types among its members (including, recursively, a member of a subaggregate or contained union), or
— a character type.
A similar issue occurs in section 6.2.7 (paragraph 2) of the standard:
declarations that refer to the same object or function shall have compatible type; otherwise, the behavior is undefined.
#Lundin's approach uses a union type, which is the second to last solution. My approach used the last approach (a char type) when storing the data.
Another approach would use void * to move the calculation to an independent function (same approved as memcpy). This will prevent the compiler from assuming anything as void * conversions are always allowed.
Original Answer:
I think the problem of writing to non-volatile storage is divided into 2 main issues:
Byte Endianness - different systems have different memory models and we want the non-volatile storage to be as system agnostic as possible.
Ease of use / Portability - we don't want to rewrite a lot of pieces of code every time we update the structure and we want to minimize compiler specific instructions.
Byte Endiannes
If we store the data on one system and than load it in another system, the byte ordering may be different between systems, breaking our data scheme.
To solve Byte Endiannes it's important to read and store 32bit numbers and 16 bit numbers in a way that is byte order agnostic. This means storing the specific bits in a pre-determined bytes.
I wrote a few macros for that (based on the facil.io open source framework):
/** write uint16_t to a buffer independently of system's of endieness. */
#define u2buf16(dest, i) \
do { \
((uint8_t *)(dest))[0] = ((uint16_t)(i)) & 0xFF; \
((uint8_t *)(dest))[1] = (((uint16_t)(i)) >> 8) & 0xFF; \
} while (0)
/** write uint32_t to a buffer independently of system's of endieness. */
#define u2buf32(dest, i) \
do { \
((uint8_t *)(dest))[0] = ((uint32_t)(i)) & 0xFF; \
((uint8_t *)(dest))[1] = (((uint32_t)(i)) >> 8) & 0xFF; \
((uint8_t *)(dest))[2] = (((uint32_t)(i)) >> 16) & 0xFF; \
((uint8_t *)(dest))[3] = (((uint32_t)(i)) >> 24) & 0xFF; \
} while (0)
/** read uint16_t from a buffer independently of system's of endieness. */
#define buf2u16(src) \
((uint16_t)(((uint8_t *)(src))[0]) | ((uint16_t)(((char *)(src))[1]) << 8))
/** read uint32_t from a buffer independently of system's of endieness. */
#define buf2u32(src) \
(((uint32_t)(((uint8_t *)(src))[0])) | \
(((uint32_t)(((uint8_t *)(src))[1])) << 8) | \
(((uint32_t)(((uint8_t *)(src))[2])) << 16) | \
(((uint32_t)(((uint8_t *)(src))[3])) << 24))
Ease of use / Portability
The best way to deal with this is to encapsulate the code in proper functions.
Before I did that, I updated the example in a number of small ways:
I removed the __attribute__(( aligned(4))) instruction, as it's both compiler specific and unnecessary.
I renamed the struct's postfix from _t to _s.
The _t type postfix is reserved by the POSIX standard and mustn't be used in user-code.
The _s postfix is a common way to indicate struct (or _u for unions, and _p for pointers, etc'). This is a personal preference and depends on where you work.
I added a uint8_t reserved; field where padding would have naturally occurred, both making room for future updates and making sure no unknown padding was present in the struct (here I use this quality only during testing).
This came out as:
typedef struct {
uint8_t value1;
uint8_t reserved; /* reserved for future versions */
int16_t value2;
uint32_t value3;
float value4;
} test_struct_s;
For the function decelerations (API) I used the root of the type name (test_struct) as the name spaced appended the function names at the end.
This approach too is a common way to manage namespaces and is a personal preference that depends on the guidelines in your workspace.
They came up as:
static void test_struct_write(char *dest, test_struct_s *src);
static void test_struct_read(test_struct_s *dest, char *src);
Now, the first thing to do is to write tests for the code.
By writing a bit in every byte before a read/write roundtrip, it is possible to effectively test the read/write roundtrip for correctness.
In addition, we want to make sure that each field in the tested struct had a different bit set, so we make sure to test that we're not mixing up values.
/** test read/write behavior. */
static void test_struct_rw_test(void) {
/*
* write defferent values to each struct field, making sure all bytes have a
* set bit at least once during the test.
*
* perform a read/write roundtri[ and test that the set bit has the same
* value.
*/
for (size_t i = 0; i < 32; ++i) {
union {
float f;
uint32_t i;
} u;
/* a float with a single bit set somewhere */
u.i = ((uint32_t)1U << ((i + 1) & 31));
/* a different bit is set in every struct field */
test_struct_s r, s = {
.value1 = (1U << ((i + 0) & 31)),
.reserved = (1U << ((i + 1) & 31)),
.value2 = (1U << ((i + 2) & 31)),
.value3 = (1U << ((i + 3) & 31)),
.value4 = u.f,
};
char buf[sizeof(s)];
test_struct_write(buf, &s);
test_struct_read(&r, buf);
/* we can use memcmp only because we control the padded bytes with
* `reserved` */
if (memcmp(&s, &r, sizeof(s))) {
fprintf(stderr, "FATAL: Read/Write rountrip test failed (at %zu)\n", i);
fprintf(stderr, "%u!=%u\n%u!=%u\n%d!=%d\n%u!=%u\n%f!=%f\n", s.value1,
r.value1, s.reserved, r.reserved, s.value2, r.value2, s.value3,
r.value3, s.value4, r.value4);
exit(-1);
}
}
}
Next we need to actually code the read / write functions themselves.
As you will notice, I am assigning each 8 bit sequence to a specific byte, allowing the code to be endianness and system agnostic.
The buffer being written to or read from MUST be (at least) 96 bits long (12 bytes), or the functions will overflow.
/** "safely" write test_struct_s to a buffer. buffer MUST be 96 bits long. */
static void test_struct_write(char *dest, test_struct_s *src) {
union {
float f;
uint32_t i;
} u;
if (sizeof(float) != sizeof(uint32_t))
goto system_error; /* will be tested by the compiler and optimized away... */
u.f = src->value4;
dest[0] = src->value1;
dest[1] = src->reserved;
u2buf16(dest + 2, src->value2);
u2buf32(dest + 4, src->value3);
u2buf32(dest + 8, u.i);
return;
system_error:
fprintf(stderr,
"FATAL: Program requires a modern system where floats are 32 bits "
"(sizeof(float) == %zu)\n",
sizeof(float));
exit(-1);
}
/** "safely" read test_struct_s from a buffer. buffer MUST be 96 bytes long. */
static void test_struct_read(test_struct_s *dest, char *src) {
if (sizeof(float) != sizeof(uint32_t))
goto system_error;
union {
float f;
uint32_t i;
} u;
dest->value1 = src[0];
dest->reserved = src[1];
dest->value2 = buf2u16(src + 2);
dest->value3 = buf2u32(src + 4);
u.i = buf2u32(src + 8);
dest->value4 = u.f;
return;
system_error:
fprintf(stderr,
"FATAL: Program requires a modern system where floats are 32 bits "
"(sizeof(float) == %zu)\n",
sizeof(float));
exit(-1);
}
And we're done.
The whole of the code looks something like this:
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/*
* added a `reserved` field to mark natural struct padding AND control padded
* bytes.
*
* now, sizeof(test_struct_s) == 96
*/
typedef struct {
uint8_t value1;
uint8_t reserved; /* reserved for future versions */
int16_t value2;
uint32_t value3;
float value4;
} test_struct_s;
static void test_struct_write(char *dest, test_struct_s *src);
static void test_struct_read(test_struct_s *dest, char *src);
/** write uint16_t to a buffer independently of system's of endieness. */
#define u2buf16(dest, i) \
do { \
((uint8_t *)(dest))[0] = ((uint16_t)(i)) & 0xFF; \
((uint8_t *)(dest))[1] = (((uint16_t)(i)) >> 8) & 0xFF; \
} while (0)
/** write uint32_t to a buffer independently of system's of endieness. */
#define u2buf32(dest, i) \
do { \
((uint8_t *)(dest))[0] = ((uint32_t)(i)) & 0xFF; \
((uint8_t *)(dest))[1] = (((uint32_t)(i)) >> 8) & 0xFF; \
((uint8_t *)(dest))[2] = (((uint32_t)(i)) >> 16) & 0xFF; \
((uint8_t *)(dest))[3] = (((uint32_t)(i)) >> 24) & 0xFF; \
} while (0)
/** read uint16_t from a buffer independently of system's of endieness. */
#define buf2u16(src) \
((uint16_t)(((uint8_t *)(src))[0]) | ((uint16_t)(((char *)(src))[1]) << 8))
/** read uint32_t from a buffer independently of system's of endieness. */
#define buf2u32(src) \
(((uint32_t)(((uint8_t *)(src))[0])) | \
(((uint32_t)(((uint8_t *)(src))[1])) << 8) | \
(((uint32_t)(((uint8_t *)(src))[2])) << 16) | \
(((uint32_t)(((uint8_t *)(src))[3])) << 24))
/** "safely" write test_struct_s to a buffer. buffer MUST be 96 bytes long. */
static void test_struct_write(char *dest, test_struct_s *src) {
union {
float f;
uint32_t i;
} u;
if (sizeof(float) != sizeof(uint32_t))
goto system_error;
u.f = src->value4;
dest[0] = src->value1;
dest[1] = src->reserved;
u2buf16(dest + 2, src->value2);
u2buf32(dest + 4, src->value3);
u2buf32(dest + 8, u.i);
return;
system_error:
fprintf(stderr,
"FATAL: Program requires a modern system where floats are 32 bits "
"(sizeof(float) == %zu)\n",
sizeof(float));
exit(-1);
}
/** "safely" read test_struct_s from a buffer. buffer MUST be 96 bytes long. */
static void test_struct_read(test_struct_s *dest, char *src) {
if (sizeof(float) != sizeof(uint32_t))
goto system_error;
union {
float f;
uint32_t i;
} u;
dest->value1 = src[0];
dest->reserved = src[1];
dest->value2 = buf2u16(src + 2);
dest->value3 = buf2u32(src + 4);
u.i = buf2u32(src + 8);
dest->value4 = u.f;
return;
system_error:
fprintf(stderr,
"FATAL: Program requires a modern system where floats are 32 bits "
"(sizeof(float) == %zu)\n",
sizeof(float));
exit(-1);
}
/** test read/write behavior. */
static void test_struct_rw_test(void) {
/*
* write defferent values to each struct field, making sure all bytes have a
* set bit at least once during the test.
*
* perform a read/write roundtri[ and test that the set bit has the same
* value.
*/
for (size_t i = 0; i < 32; ++i) {
union {
float f;
uint32_t i;
} u;
/* a float with a single bit set somewhere */
u.i = ((uint32_t)1U << ((i + 1) & 31));
test_struct_s r, s = {
.value1 = (1U << ((i + 0) & 31)),
.reserved = (1U << ((i + 1) & 31)),
.value2 = (1U << ((i + 2) & 31)),
.value3 = (1U << ((i + 3) & 31)),
.value4 = u.f,
};
char buf[sizeof(s)];
test_struct_write(buf, &s);
test_struct_read(&r, buf);
/* we can use memcmp only because we control the padded bytes with
* `reserved` */
if (memcmp(&s, &r, sizeof(s))) {
fprintf(stderr, "FATAL: Read/Write rountrip test failed (at %zu)\n", i);
fprintf(stderr, "%u!=%u\n%u!=%u\n%d!=%d\n%u!=%u\n%f!=%f\n", s.value1,
r.value1, s.reserved, r.reserved, s.value2, r.value2, s.value3,
r.value3, s.value4, r.value4);
exit(-1);
}
}
}
int main(void) {
test_struct_rw_test();
fprintf(stderr, "PASSED\n");
return 0;
}
The following solution seems to work fine:
test_struct_t test_t;
uint16_t size = sizeof(test_struct_t)/sizeof(uint32_t);
uint32_t *ptr = (uint32_t *)&test_t;
for(uint16_t i = 0; i < size; i++)
{
writeWordNVM(i, *ptr);
ptr++;
}
Apart from alignment, the issue here is the C type system, where each position in memory that contains a declared variable or has been de-referenced by the program through a value access, has an internal "type tag" formally known as effective type. This is used for optimization purposes, so that the compiler can know if pointer types of different kinds may alias. That is, if they could refer to the same memory location. Informally this whole set of rules is called the "strict aliasing rules". What is the strict aliasing rule?
Simple example of pointer aliasing: if you have void func (float* f){ *f = 1.0f; } and a global int foo;, the compiler is free to assume that the function writing to *f won't modify the contents of foo when doing so.
Traditionally, embedded system compilers never used optimization based on aliasing assumptions, but with gcc entering the embedded world, this is no longer necessarily the case. If you compile with gcc -fno-strict-aliasing, you can dodge the problem, but the code won't be portable.
This is kind of a language defect. To portably avoid the strict aliasing rules, you must use cumbersome tricks such as union type punning. That is, create a union containing the struct and an array of uint32_t both, then do all de-referencing on the array member. Example:
#include <stdio.h>
#include <stdint.h>
typedef struct
{
uint8_t value1;
int16_t value2;
uint32_t value3;
float value4;
} test_struct_t;
_Static_assert(_Alignof(test_struct_t) == _Alignof(uint32_t),
"Inconsistent alignment in test_struct_t");
typedef union
{
struct
{
uint8_t value1;
int16_t value2;
uint32_t value3;
float value4;
};
uint32_t u32 [sizeof(test_struct_t)/sizeof(uint32_t)];
} test_union_t;
int main(int argc, char **argv)
{
test_union_t test = {.value1 = 1, .value2 = 2};
printf("%.8X\n", test.u32[0]);
return 0;
}
Here I first made a typedef of the struct just to check with static assert if the alignment is ok, removing the non-standard __attribute__(( aligned(4))).
Then when creating the type punning union, I made the same struct anonymous just so that we don't have to type out test.test_struct.value1 rather than test.value1, kind of hiding the fact that it's a union.
I then initialized some members of the union by name and de-referenced them through a type punned uint32_t. This prints 00020001 on my little endian machine, apparently there's a padding byte after value1, which is as expected.
I have a problem to understand how I can use this macro to get the lenght of the IP header. I have a file called framehdr.h and I have included that file in my main.c. The struct in the framehdr.h file look like this:
struct ip_hdr {
unsigned char vhl; /* Version and header length */
unsigned char tos; /* Type of service */
unsigned short len; /* Total length */
unsigned short id; /* Identification */
unsigned short off; /* Fragment offset field */
unsigned char ttl; /* Time to live */
unsigned char p; /* Protocol */
unsigned short ip_sum; /* Checksum */
unsigned int src, dst; /* Source and dest address */
};
I can manually get the lentgh by adding all the sizes like this:
unsigned int sizeIP = sizeof(testOne.vhl) + sizeof(testOne.tos) +
sizeof(testOne.len) + sizeof(testOne.id) +
sizeof(testOne.off) + sizeof(testOne.ttl) +
sizeof(testOne.p) + sizeof(testOne.ip_sum) +
sizeof(testOne.src) + sizeof(testOne.dst);
But I'm suppose to use the macro that look like this to get the size:
#define IP_HL(ip) ((((ip)->vhl) & 0x0f) * 4)
What does this macro return? Is it a digit so I can store it in a int and then print it or how does it work?
It looks like vhl (which is an 8 bit member, unless CHAR_BIT is a weird value on your platform) is used to store the version in the significant 4 bits and the header length in the other 4 bits.
((ip)->vhl) & 0x0f) extracts the least significant 4 bits of the vhl member of the struct to which ip is pointing.
The multiplication by 4 to get the actual length is merely obeying the conventions of that member.
The way you are using sizeof to "manually get the length" is making assumptions about the header length corresponding to the sum of the sizes of the individual members. Member padding and undocumented conventions centred around the serialisation technique used for your struct might mean this is not the case.
I'm planning to use AVR-Crypto's SHA-1 implementation for HMAC. However, I can't seem to generate the correct SHA-1 sum.
For instance, if I call the function with the following
unsigned char sha1sum[20];
char *msg = "FFFFFFFFFF";
sha1( sha1sum, msg, strlen(msg));
I get 000000000000000000002C002312290000000029 rather than the expected c1bb92851109fe950a2655fa1d4ba1d04719f6fb. Does anyone know what might be wrong? Here's the AVR-Crypto's implementation
#include <string.h> /* memcpy & co */
#include <stdint.h>
#include "config.h"
#include "debug.h"
#include "sha1.h"
#ifdef DEBUG
# undef DEBUG
#endif
#include "cli.h"
#define LITTLE_ENDIAN
/********************************************************************************************************/
/**
* \brief initialises given SHA-1 context
*
*/
void sha1_init(sha1_ctx_t *state){
DEBUG_S("\r\nSHA1_INIT");
state->h[0] = 0x67452301;
state->h[1] = 0xefcdab89;
state->h[2] = 0x98badcfe;
state->h[3] = 0x10325476;
state->h[4] = 0xc3d2e1f0;
state->length = 0;
}
/********************************************************************************************************/
/* some helping functions */
uint32_t rotl32(uint32_t n, uint8_t bits){
return ((n<<bits) | (n>>(32-bits)));
}
uint32_t change_endian32(uint32_t x){
return (((x)<<24) | ((x)>>24) | (((x)& 0x0000ff00)<<8) | (((x)& 0x00ff0000)>>8));
}
/* three SHA-1 inner functions */
uint32_t ch(uint32_t x, uint32_t y, uint32_t z){
DEBUG_S("\r\nCH");
return ((x&y)^((~x)&z));
}
uint32_t maj(uint32_t x, uint32_t y, uint32_t z){
DEBUG_S("\r\nMAJ");
return ((x&y)^(x&z)^(y&z));
}
uint32_t parity(uint32_t x, uint32_t y, uint32_t z){
DEBUG_S("\r\nPARITY");
return ((x^y)^z);
}
/********************************************************************************************************/
/**
* \brief "add" a block to the hash
* This is the core function of the hash algorithm. To understand how it's working
* and what thoese variables do, take a look at FIPS-182. This is an "alternativ" implementation
*/
#define MASK 0x0000000f
typedef uint32_t (*pf_t)(uint32_t x, uint32_t y, uint32_t z);
void sha1_nextBlock (sha1_ctx_t *state, const void *block){
uint32_t a[5];
uint32_t w[16];
uint32_t temp;
uint8_t t,s,fi, fib;
pf_t f[] = {ch,parity,maj,parity};
uint32_t k[4]={ 0x5a827999,
0x6ed9eba1,
0x8f1bbcdc,
0xca62c1d6};
/* load the w array (changing the endian and so) */
for(t=0; t<16; ++t){
w[t] = change_endian32(((uint32_t*)block)[t]);
}
#if DEBUG
uint8_t dbgi;
for(dbgi=0; dbgi<16; ++dbgi){
/*
DEBUG_S("\n\rBlock:");
DEBUG_B(dbgi);
DEBUG_C(':');
*/
cli_putstr_P(PSTR("\r\nBlock:"));
cli_hexdump(&dbgi, 1);
cli_putc(':');
cli_hexdump(&(w[dbgi]) ,4);
}
#endif
/* load the state */
memcpy(a, state->h, 5*sizeof(uint32_t));
/* the fun stuff */
for(fi=0,fib=0,t=0; t<=79; ++t){
s = t & MASK;
if(t>=16){
#if DEBUG
DEBUG_S("\r\n ws = "); cli_hexdump(&(w[s]), 4);
#endif
w[s] = rotl32( w[(s+13)&MASK] ^ w[(s+8)&MASK] ^
w[(s+ 2)&MASK] ^ w[s] ,1);
#ifdef DEBUG
DEBUG_S(" --> ws = "); cli_hexdump(&(w[s]), 4);
#endif
}
uint32_t dtemp;
temp = rotl32(a[0],5) + (dtemp=f[fi](a[1],a[2],a[3])) + a[4] + k[fi] + w[s];
memmove(&(a[1]), &(a[0]), 4*sizeof(uint32_t)); /* e=d; d=c; c=b; b=a; */
a[0] = temp;
a[2] = rotl32(a[2],30); /* we might also do rotr32(c,2) */
fib++;
if(fib==20){
fib=0;
fi = (fi+1)%4;
}
#if DEBUG
/* debug dump */
DEBUG_S("\r\nt = "); DEBUG_B(t);
DEBUG_S("; a[]: ");
cli_hexdump(a, 5*4);
DEBUG_S("; k = ");
cli_hexdump(&(k[t/20]), 4);
DEBUG_S("; f(b,c,d) = ");
cli_hexdump(&dtemp, 4);
#endif
}
/* update the state */
for(t=0; t<5; ++t){
state->h[t] += a[t];
}
state->length += 512;
}
/********************************************************************************************************/
void sha1_lastBlock(sha1_ctx_t *state, const void *block, uint16_t length){
uint8_t lb[SHA1_BLOCK_BYTES]; /* local block */
while(length>=SHA1_BLOCK_BITS){
sha1_nextBlock(state, block);
length -= SHA1_BLOCK_BITS;
block = (uint8_t*)block + SHA1_BLOCK_BYTES;
}
state->length += length;
memset(lb, 0, SHA1_BLOCK_BYTES);
memcpy (lb, block, (length+7)>>3);
/* set the final one bit */
lb[length>>3] |= 0x80>>(length & 0x07);
if (length>512-64-1){ /* not enouth space for 64bit length value */
sha1_nextBlock(state, lb);
state->length -= 512;
memset(lb, 0, SHA1_BLOCK_BYTES);
}
/* store the 64bit length value */
#if defined LITTLE_ENDIAN
/* this is now rolled up */
uint8_t i;
for (i=0; i<8; ++i){
lb[56+i] = ((uint8_t*)&(state->length))[7-i];
}
#elif defined BIG_ENDIAN
*((uint64_t)&(lb[56])) = state->length;
#endif
sha1_nextBlock(state, lb);
}
/********************************************************************************************************/
void sha1_ctx2hash (void *dest, sha1_ctx_t *state){
#if defined LITTLE_ENDIAN
uint8_t i;
for(i=0; i<5; ++i){
((uint32_t*)dest)[i] = change_endian32(state->h[i]);
}
#elif BIG_ENDIAN
if (dest != state->h)
memcpy(dest, state->h, SHA1_HASH_BITS/8);
#else
# error unsupported endian type!
#endif
}
/********************************************************************************************************/
/**
*
*
*/
void sha1 (void *dest, const void *msg, uint32_t length){
sha1_ctx_t s;
DEBUG_S("\r\nBLA BLUB");
sha1_init(&s);
while(length & (~0x0001ff)){ /* length>=512 */
DEBUG_S("\r\none block");
sha1_nextBlock(&s, msg);
msg = (uint8_t*)msg + SHA1_BLOCK_BITS/8; /* increment pointer to next block */
length -= SHA1_BLOCK_BITS;
}
sha1_lastBlock(&s, msg, length);
sha1_ctx2hash(dest, &s);
}
Here's the header:
#ifndef SHA1_H_
#define SHA1_H_
#include "stdint.h"
/** \def SHA1_HASH_BITS
* definees the size of a SHA-1 hash in bits
*/
/** \def SHA1_HASH_BYTES
* definees the size of a SHA-1 hash in bytes
*/
/** \def SHA1_BLOCK_BITS
* definees the size of a SHA-1 input block in bits
*/
/** \def SHA1_BLOCK_BYTES
* definees the size of a SHA-1 input block in bytes
*/
#define SHA1_HASH_BITS 160
#define SHA1_HASH_BYTES (SHA1_HASH_BITS/8)
#define SHA1_BLOCK_BITS 512
#define SHA1_BLOCK_BYTES (SHA1_BLOCK_BITS/8)
/** \typedef sha1_ctx_t
* \brief SHA-1 context type
*
* A vatiable of this type may hold the state of a SHA-1 hashing process
*/
typedef struct {
uint32_t h[5];
// uint64_t length;
uint8_t length;
} sha1_ctx_t;
/** \typedef sha1_hash_t
* \brief hash value type
* A variable of this type may hold a SHA-1 hash value
*/
/*
typedef uint8_t sha1_hash_t[SHA1_HASH_BITS/8];
*/
/** \fn sha1_init(sha1_ctx_t *state)
* \brief initializes a SHA-1 context
* This function sets a ::sha1_ctx_t variable to the initialization vector
* for SHA-1 hashing.
* \param state pointer to the SHA-1 context variable
*/
void sha1_init(sha1_ctx_t *state);
/** \fn sha1_nextBlock(sha1_ctx_t *state, const void *block)
* \brief process one input block
* This function processes one input block and updates the hash context
* accordingly
* \param state pointer to the state variable to update
* \param block pointer to the message block to process
*/
void sha1_nextBlock (sha1_ctx_t *state, const void *block);
/** \fn sha1_lastBlock(sha1_ctx_t *state, const void *block, uint16_t length_b)
* \brief processes the given block and finalizes the context
* This function processes the last block in a SHA-1 hashing process.
* The block should have a maximum length of a single input block.
* \param state pointer to the state variable to update and finalize
* \param block pointer to themessage block to process
* \param length_b length of the message block in bits
*/
void sha1_lastBlock (sha1_ctx_t *state, const void *block, uint16_t length_b);
/** \fn sha1_ctx2hash(sha1_hash_t *dest, sha1_ctx_t *state)
* \brief convert a state variable into an actual hash value
* Writes the hash value corresponding to the state to the memory pointed by dest.
* \param dest pointer to the hash value destination
* \param state pointer to the hash context
*/
void sha1_ctx2hash (void *dest, sha1_ctx_t *state);
/** \fn sha1(sha1_hash_t *dest, const void *msg, uint32_t length_b)
* \brief hashing a message which in located entirely in RAM
* This function automatically hashes a message which is entirely in RAM with
* the SHA-1 hashing algorithm.
* \param dest pointer to the hash value destination
* \param msg pointer to the message which should be hashed
* \param length_b length of the message in bits
*/
void sha1(void *dest, const void *msg, uint32_t length_b);
#endif /*SHA1_H_*/
UPDATE If I initialise sha1sum with unsigned char sha1sum[20] = 0;, the resulting sum is all 0x00.
There are at least two bugs in the question's code (detailed below), but neither can explain the result shown, and the added fact that unsigned char sha1sum[20] = {0} in the calling code changes the result. Something is wrong with the translation from the C source code that we read into machine code! Most likely, sha1_ctx2hash did not write where it should.
Problem could be in a header not in the question, a compiler bug... Since we are on a 8051, that could be/have been a problem with pointer types, especially in the pointer casts that MUST be to a pointer of the same size.
Also, is it dead sure the 8051 compiler is little-endian? It seems the common Keil C51 uses big-endian convention. That's an arbitrary choice of the compiler+support library, since on the original 8051 there is no multi-byte data-related instruction, the closest thing is LCALL which stack pushes are little-endian, but LJMP and MOV DPTR,# code is big-endian. Update: We are told the compiler is by IAR. According to IAR's documentation, version 5 was big-endian, and that changed to little-endian in version 6.
Update: we found another critical issue (beyond likely unsafe pointer casting, and the two bugs discussed below). At some point in the hunt, replacing the code with a single procedure having no endianness dependency or pointer cast, the output became 0000eb1700007f3d000004f0000059290000fc21 and that suggested would-be-32-bit values are truncated to 16-bit. Indeed, the OP revealed:
I have this in my stdint.h: typedef unsigned uint32_t;
This is correct only on compilers where unsigned int is exactly 32-bit, when the only insurance given by the C standard is that it is at least 16-bit, and that minimum is used by most C compilers for less-than-32-bit CPUs (for efficiency reasons; some even have an option to disable promotion of byte operands to integer, and are even happy with 80+80+96 being 0).
Bug in the test code: sha1( sha1sum, msg, strlen(msg)) should be sha1( sha1sum, msg, strlen(msg)*8) or the like, because the length parameters is in bits.
Bug in the sha1_lastBlock w.r.t. header file: the code reading
for (i=0; i<8; ++i){
lb[56+i] = ((uint8_t*)&(state->length))[7-i];
}
assumes that state->length is 8 bytes, when it is not, because uint64_t length was changed to uint8_t length in the header (it is common that uint64_t is not available on 8051 compilers). The code for the big-endian case (not presently compiled) is affected too.
If indeed uint8_t length and thus a restriction to length of 31 bytes at most is acceptable, both the little-endian and big-endian cases reduce to lb[SHA1_BLOCK_BYTES-1] = state->length; (without a loop).
Or, for whatever unsigned type and endianness length might use:
for (i = SHA1_BLOCK_BYTES; state->length != 0; state->length >>= 8)
lb[--i] = (uint8_t)(state->length);
Note: The code *((uint64_t*)&(lb[56])) = state->length was writing the 8 bytes of length to the end of array lb[], but is correct only on big-endian machines with proper uint64_t.
The code has as potential extra problem when (length+7)%8 < 6: at least one bit in the last byte to hash is not masked, and if set it enters the hash and makes it wrong. That won't harm in the use case where full bytes are hashed.
The original code might be correct (except for the above potential extra problem), but is needlessly complex given that the objective is hashing in-memory data with a single call (what sha1 does), and neither compact nor readable. Among other issues:
there is (rightly) a block loop in sha1_lastBlock, thus the restriction worded in the header The block should have a maximum length of a single input block does not exist;
that makes the other block loop in sha1 redundant;
both of these loops can be removed if using uint8_t length, or otherwise hashing less than 56 bytes;
the round loop is likely slowed by a memmove of 16 bytes, and a function call in a vector from an indexed table;
endianness conversion in the little-endian case is rather inefficient;
in sha1_ctx2hash, the #elif BIG_ENDIAN triggers an error in my mental compiler, since BIG_ENDIAN seems undefined and #elif is supposed to have an argument; that should be #elif defined BIG_ENDIAN (as used a few lines above);
pf_t f[] = {ch,parity,maj,parity}; is a good candidate for const, perhaps static: every C compiler for 8051 that I ever used won't recognize that the array is not changed after setup, and thus can be carved in code;
with such compilers, needlessly using function pointers (as above) is a tried and test method to harm performance, or worse; at the very least it blocks the analysis of the call tree, needed to allocate automatic variables at static addresses with overlay, which in turn improves performance and code size significantly.
If you are after speed, the code you start from is inadequate, and nothing will quite match assembly language. Like two decades ago I wrote SHA-1 for some 8051 toolchain, and assembly tuneups gave huge savings compared to C only (IIRC: mostly because 32-bit rotations where abyssal from a performance standpoint).
Updated: Here is illustrative code to hash a short message, in an endian-neutral way, without any pointer cast, and without depending on <stdint.h> (which turns out to be inadequate for the compiler used). Beware that the length parameter is in bytes (not bits), with a limit of 55 bytes that does NOT allow implementing HMAC-SHA-1 on top. That's to keep the code simple: over that limit we need several iterations of the compression function, thus either heavy code duplication, at least two functions, or some sort of state machine.
#include <limits.h> // for UCHAR_MAX, UINT_MAX, ULONG_MAX
// Compute the SHA-1 hash of a short msg, of length at most 55 bytes
// Result hash must be 20 bytes; it can overlap msg.
// CAUTION: if length>55 the result is wrong, and if length>59
// we loose second-preimage resistance, thus collision-resistance.
void sha1upto55bytes(
unsigned char *hash, // result, 20 bytes
const unsigned char *msg, // bytes to hash
unsigned char length // length of msg in bytes, maximum 55
)
{
// We locally (re)define uint8_t and uint32_t so as not to depend of <stdint.h>
// which is not available on some old C compilers for embedded systems.
#if 255==UCHAR_MAX
typedef unsigned char uint8_t;
#endif
#if 16383==UINT_MAX>>9>>9
typedef unsigned int uint32_t;
#elif 16383==ULONG_MAX>>9>>9
typedef unsigned long uint32_t;
#endif
// Internal buffer (64 bytes)
// We require 8-bit uint8_t, 32-bit uint32_t, and integer promotion; otherwise,
// we try to abort compilation on the following declaration.
uint32_t w[
99==(uint8_t)355 && // check uint8_t
4303==(uint32_t)(-1)/999u/999 && // check uint32_t
440==(uint8_t)55<<3 // check integer promotion
? 16 : -1]; // negative index if error
// Type for state, so that we can use struct copy for that
typedef struct state_t { uint32_t q[5]; } state_t;
// Initial state; use single quotes if the compiler barks
const state_t s = {{ 0x67452301,0xefcdab89,0x98badcfe,0x10325476,0xc3d2e1f0 }};
// Active state (20 bytes); on 8051 should be in internal RAM for best performance
state_t h = s; // initialize the state using a struct copy
// Workhorse temporary; on 8051 should be in internal RAM for best performance
uint32_t x;
// Workhorse index; on 8051 should be a register for performance
uint8_t j;
// Prepare the single block to hash; this code works regardless of endianness,
// and does not perform misaligned memory accesses if msg is misaligned.
x = 0; // This is only to prevent a bogus compiler warning
j = 0;
do
{ // for each block byte, up to and including high 4 bytes of length
x <<= 8;
if (j < length)
x |= *msg++; // message byte
else
if (j == length)
x |= 0x80; // padding byte
if ((j&3)==3)
w[j >> 2] = x;
}
while (++j!=60);
w[15] = length << 3; // length in bits, needs integer promotion for length>31
// Hash that block
j = 0;
do { // round loop, run 80 times
do { // dummy loop (avoid a goto)
if (j<40) {
if (j<20) { // for rounds 0..19
x = (((h.q[2] ^ h.q[3])&h.q[1]) ^ h.q[3]) + 0x5A827999;
break; // out of dummy loop
}
else
x = 0x6ED9EBA1; // for rounds 20..39
}
else {
if (j<60) { // for rounds 40..59
x = (h.q[1] | h.q[2])&h.q[3];
x |= h.q[1] & h.q[2];
x += 0x8F1BBCDC;
break;
}
else
x = 0xCA62C1D6; // for rounds 60..79
}
// for rounds 20..39 and 60..79
x += h.q[1] ^ h.q[2] ^ h.q[3];
}
while (0); // end of of dummy loop
// for all rounds
x += (h.q[0] << 5) | (h.q[0] >> 27);
x += h.q[4];
h.q[4] = h.q[3];
h.q[3] = h.q[2];
h.q[2] = (h.q[1] << 30) | (h.q[1] >> 2);
h.q[1] = h.q[0];
h.q[0] = x;
x = w[j & 15];
if (j>=16) { // rounds 16..79
x ^= w[(j + 2) & 15];
x ^= w[(j + 8) & 15];
x ^= w[(j + 13) & 15];
w[j & 15] = x = (x << 1) | (x >> 31);
}
h.q[0] += x; // for all rounds
}
while (++j != 80);
// The five final 32-bit modular additions are made in the next loop, and
// reuse the constants (rather than a RAM copy), saving code and RAM.
// Final addition and store result; this code works regardless of endianness,
// and does not perform misaligned memory accesses if hash is misaligned.
j = 0;
do
{
x = h.q[j] + s.q[j]; // final 32-bit modular additions
*hash++ = (uint8_t)(x>>24);
*hash++ = (uint8_t)(x>>16);
*hash++ = (uint8_t)(x>> 8);
*hash++ = (uint8_t)(x );
}
while (++j != 5);
}
I'm getting an error on a struct call inside a function. In the code, I am converting two bytes of data into one 16-bit data and I want to store it into one of my struct variables. However, I am getting an error that I can't pinpoint. The compiler is telling me that the error is in the line unsigned int fat.sector_size = combinedBytes;. What am I doing wrong here? Thanks in advance!
EDIT: The error that I'm getting is
main.c:62:19: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘.’ token
main.c:62:19: error: expected expression before ‘.’ token`
struct fileSystem_info{
unsigned int sector_size; //Sector size
int cluster_size_in_sectors; //Cluster size in sectors
int entries_in_root_directory; //Number of entries in root directory
int sectors_per_fat; //Sectors per file allocation table
int reserved_sectors; //Number of reserved sectors on the disk
int hidden_sectors; //Number of hidden sectors on the disk
int sector_number_of_first_copy_of_fat; //Sector number of the first copy of the file allocation table
int sector_number_of_first_sector_of_root_directory; //Sector number of the first sector of the root directory
int sector_numner_of_first_sector_of_first_usable_data_cluster; //Sector number of the first sector of the first usable data cluster
};
//Converts two 8 bit data to one 16 bit data
unsigned converter(unsigned mostSignificant_bit, unsigned leastSignificant_bit){
uint16_t value = (uint16_t)(mostSignificant_bit << 8) | leastSignificant_bit;
//return((mostSignificant_bit * 256) + leastSignificant_bit);
return (value);
}
unsigned int sectorSize (){
struct fileSystem_info fat;
unsigned char first_byte = buffer[11];
printf("%hhu \n", first_byte);
unsigned char second_byte = buffer[12];
printf("%hhu \n", second_byte);
unsigned int combinedBytes = converter ((int)second_byte, (int)first_byte);
unsigned int fat.sector_size = combinedBytes;
return (combinedBytes);
}
Here
unsigned int fat.sector_size = combinedBytes;
remove the type
fat.sector_size = combinedBytes;
(... and perhaps have a break ... ;-)
Refering the question on version-info in the comments below:
Do something like this:
#define VERSION_MAJOR (0) /* \ */
#define VERSION_MINOR (1) /* +--<-- adjust those for upcoming releases. */
#define VERSION_MICRO (42) /* / */
struct fileSystem_info
{
unsigned int version;
unsigned int sector_size; //Sector size
...
When initialising an instance of struct fileSystem_info, do:
struct fileSystem_info fsi = {0};
fsi.version = (VERSION_MAJOR << 24) | (VERSION_MINOR << 16) | VERSION_MICRO};
Doing so allows you a maximum version number of 255.255.65535.
As you always write version to disk you could later determine which version you wrote by reading the first unsigned int version from the structure an then in decide how to go on reading. This might be relevant in case the structure and how its content is to be interpreted following version changed during the different releases of your program.
An identifier cannot have ..
At
unsigned int fat.sector_size = combinedBytes;
you are defining a new variable of unsigned int type, called fat.sector_size (which is an invalid identifier.
If you want to refer to the member sector_size of the fat variable, don't use the syntax for variable definition. Just write
fat.sector_size = combinedBytes;
I have a C program that receives data from a mainframe in a UDP packet over sockets. The host of the C program is changing from Unix (big endian) to Linux (little endian) and the program no longer works. I do not currently have the option of changing the mainframe client program.
The program performs a recvfrom and receives data into a char array. Previously we were able to simply cast this buffer to a structure matching what was passed from MF and everything worked perfect. Now, due to the different byte alignments the mapping to the structure fails. Here is the structure and some code.
struct CCClntPkt
{
unsigned short packet_type;
unsigned short reply_socket;
unsigned long msg_ID;
unsigned short msg_length;
unsigned char client_string[250];
};
The code previously used to cast the received data buffer to this structure looks something like:
char BCpacket_in[MAX_PACKET];
struct CCClntPkt *pClntPkt;
<snip>
rcv_cnt = recvfrom(BCServerSocket, BCpacket_in,
sizeof(BCpacket_in),0x0,(struct sockaddr *)&from,
&fromlen);
if (rcv_cnt > 0)
{
pClntPkt = (struct CCClntPkt *) &BCpacket_in;
}
I was able to get the correct values for packet_type and reply_socket by using ntohs but the character field client_string is mangled. I also tried putting a pragma pack(1) before and pragma pack(0) after the structure but there still seems to be an alignment problem.
I also tried bit shifting values from BCpacket_in and was able to get the correct values for packet_type and reply_socket but cannot figure out how to pull out the ulong msg_ID. Code for that was:
packet_type = BCpacket_in[0] << 8;
packet_type |= BCpacket_in[1];
reply_to_socket = BCpacket_in[2] << 8;
reply_to_socket |= BCpacket_in[3];
/*
msg_ID = BCpacket_in[4] << 24;
msg_ID |= BCpacket_in[5] << 16;
msg_ID |= BCpacket_in[6] << 8;
msg_ID |= BCpacket_in[7];
*/
I am pretty stumped at this point so any help is appreciated. I am not the original author of this program and my C knowledge is pretty limited. I don't mind doing the work though so I would appreciate being provided any relevant references as well. Thanks!
You'll have to parse the received packet (BCpacket_in) into a struct CCClntPkt packet manually, that's the only portable way of doing this. Endianness translations are properly handled with the ntohl (network-to-host long) family of functions; see the manpages byteorder(3) and endian(3).
These functions assume that all packets are sent over the wire as big-endian, because that's the internet standard.
It's possible that the sizes of the various types differ from your big endian host to your new little endian host.
If you compile this program on your both of your hosts, it will show you the sizes and layout of the struct:
#include <stddef.h>
#include <stdio.h>
struct CCClntPkt
{
unsigned short packet_type;
unsigned short reply_socket;
unsigned long msg_ID;
unsigned short msg_length;
unsigned char client_string[250];
};
int main()
{
printf("sizeof(unsigned short) = %u\n", (unsigned)sizeof(unsigned short));
printf("sizeof(unsigned long) = %u\n", (unsigned)sizeof(unsigned long));
printf("offsetof(struct CCClntPkt, reply_socket) = %u\n", (unsigned)offsetof(struct CCClntPkt, reply_socket));
printf("offsetof(struct CCClntPkt, msg_ID) = %u\n", (unsigned)offsetof(struct CCClntPkt, msg_ID));
printf("offsetof(struct CCClntPkt, msg_length) = %u\n", (unsigned)offsetof(struct CCClntPkt, msg_length));
printf("offsetof(struct CCClntPkt, client_string) = %u\n", (unsigned)offsetof(struct CCClntPkt, client_string));
return 0;
}
In particular, it's quite possible that long is longer on your new host than your old one. This is likely to be a good place to use the C99 exact-width types from <stdint.h> - if, on your original host, short is a 16 bit type and long is a 32 bit type, replace those with uint16_t and uint32_t respectively.
You can then use ntohs() and ntohl() to perform the endianness corrections.
msg_ID = BCpacket_in[4] << 24;
msg_ID |= BCpacket_in[5] << 16;
msg_ID |= BCpacket_in[6] << 8;
msg_ID |= BCpacket_in[7];
This seems correct to me.
Try to use unsigned char for your buffer to protect yourself from signedness problem.
BTW, is msg_id in big endian, and are you sure of the offset: as you said "packing" didn't work on client side, so one can conclude that the structure is sent to the wire using the packing rules of the mainframe.
This is what I usually do for packed structures sent/received to/from the network:
#define PACKED __attribute__((__packed__))
struct PACKED message { ... };
This is GCC-specific, see here. Then you have to figure out what size the long is here. It's different on 32- and 64-bit platforms. You might want to look into using stdint.h types instead. Also look info __builtin_bswap32() and __builtin_bswap64() GCC intrinsics.