I am working on an embedded application where RAM is extremely tight.
For this purpose I need to create a 24 bit unsigned integer data type. I am doing this using a struct:
typedef struct
{
uint32_t v : 24;
} uint24_t;
However when I interrogate the size of a variable of this type, it returns "4", i.e.:
uint24_t x;
x.v = 0;
printf("Size = %u", sizeof(x));
Is there a way I can force this variable to have 3 bytes?
Initially I thought it was because it is forcing datatypes to be word aligned, but I can for example do this:
typedef struct
{
uint8_t blah[3];
} mytype;
And in that case the size comes out at 3.
Well, you could try to ensure that the structure only takes up the space you need, with something like:
#pragma pack(push, 1)
typedef struct { uint8_t byt[3]; } UInt24;
#pragma pack(pop)
You may have to provide those compiler directives (like the #pragma lines above) to ensure there's no padding but this will probably be the default for a structure with only eight-bit fields(a).
You would probably then have to pack/unpack real values to and from the structure, something like:
// Inline suggestion used to (hopefully) reduce overhead.
inline uint32_t unpack(UInt24 x) {
uint32_t retVal = x.byt[0];
retVal = retVal << 8 | x.byt[1];
retVal = retVal << 8 | x.byt[2];
return retVal;
}
inline UInt24 pack(uint32_t x) {
UInt24 retVal;
retVal.byt[0] = (x >> 16) & 0xff;
retVal.byt[1] = (x >> 8) & 0xff;
retVal.byt[2] = x & 0xff;
return retVal;
}
Note that this gives you big-endian values regardless of your actual architecture. This won't matter if you're exclusively packing and unpacking yourself, but it may be an issue if you want to use the memory blocks elsewhere in a specific layout (in which case you can just change the pack/unpack code to use the desired format).
This method adds a little code to your system (and a probably minimal performance penalty) so you'll have to decide if that's worth the saving in data space used.
(a) For example, both gcc 7.3 and clang 6.0 show 3 6 for the following program, showing that there is no padding either within or following the structure:
#include <stdio.h>
#include <stdint.h>
typedef struct { uint8_t byt[3]; } UInt24;
int main() {
UInt24 x, y[2];
printf("%zd %zd\n", sizeof(x), sizeof(y));
return 0;
}
However, that is just a sample so you may want to consider, in the interest of portable code, using something like #pragma pack(1), or putting in code to catch environments where this may not be the case.
A comment by João Baptista on this site says that you can use #pragma pack. Another option is to use __attribute__((packed)):
#ifndef __GNUC__
# define __attribute__(x)
#endif
struct uint24_t { unsigned long v:24; };
typedef struct uint24_t __attribute__((packed)) uint24_t;
This should work on GCC and Clang.
Note, however, that this will probably screw up alignment unless your processor supports unaligned access.
Initially I thought it was because it is forcing datatypes to be word aligned
Different datatypes can have different alignment. See for example the Objects and alignment doc.
You can use alignof to check, but it's totally normal for char or uint8_t to have 1-byte (ie, effectively no) alignment, but for uint32_t to have 4-bye alignment. I don't know if the alignment of bitfields is explicitly described, but inheriting it from the storage type seems reasonable enough.
NB. The reason for having alignment requirements is generally that it works better with the underlying hardware. If you do use #pragma pack or __attribute__((packed)) or whatever, you may take a performance hit as the compiler - or the memory hardware - silently handle misaligned accesses.
Just explicitly storing a 3-byte array is probably better, IMO.
To begin with, don't use bit-fields or structs. They may include padding as they please and bit-fields are non-portable in general.
Unless your CPU explicitly got 24 bit arithmetic instructions - which doesn't seem very likely unless it's some oddball DSP - then a custom data type will achieve nothing but extra stack bloat.
Most likely you will have to use uint32_t for all arithmetic. Meaning that your 24 bit type might not achieve much when it comes to saving RAM. If you invent some custom ADT with setter/getter (serialization/de-serialization) access, you are probably just wasting RAM since you get higher stack peak usage if the functions can't be inlined.
To actually save RAM you should rather revise your program design.
That being said, you can create a custom type based on an array:
typedef unsigned char u24_t[3];
Whenever you need to access the data, you memcpy it to/from a 32 bit type and then do all arithmetic on 32 bits:
u24_t u24;
uint32_t u32;
...
memcpy(&u32, u24, sizeof(u24));
...
memcpy(&u24, &u32, sizeof(u24));
But note that this assumes little endian, since we are only working with bits 0 to 2. In case of a big endian system, you will have to do memcpy((uint8_t*)&u32+1, ... to discard the MS byte.
Related
Conversion of several values to a byte-string for radio transmission has to avoid unneeded bytes. Using GCC on an ARM target (32 bit) I use "attribute ((packed))". This directive is GCC based (as I read somewhere here) and so not generally portable - which I would prefer. Example:
typedef struct __attribute__ ((packed)) {
uint8_t u8; // (*)
int16_t i16; // (*)
float v;
...
uint16_t cs; // (*)
}valset_t; // valset_t is more often used
valset_t vs;
(*) values would use 4 bytes without the ((packed)) attribute, instead of one or two as desired. Byte-wise access for transmission with:
union{
valset_t vs; // value-set
uint8_t b[sizeof(valset_t)]; // byte array
}vs_u;
using vs_u.b[i] .
Processing time is not critical here, as the transmission is much slower.
Endian is also not to be considered here, but a different C-compiler might be applied in some cases.
Elder posts to this task gave some insight, but maybe packing and alignment features were improved meanwhile in C ?c) .
Is there a more portable solution in C to perform this task?
Packing/padding isn't standardized, and therefore structs/unions are strictly speaking not portable. #pragma pack(1) is somewhat more common and supported by many different compilers (including gcc), but it is still not fully portable.
Also please note that padding exists for a reason, these structs with non-standard packing could be dangerous or needlessly inefficient on some systems.
The only 100% portable data type for storing protocols etc is an array of unsigned char. You can get only get fully portable structs if you write serializer/deserializer routines for them. This naturally comes at the expense of extra code. Example of deserialization:
valset_t data_to_valset (const unsigned char* data)
{
valset_t result;
result.something = ... ;
...
return result;
}
In case of a certain network endianess, you can convert from network endianess to CPU endianess inside the same routine.
Please note that you have to type it out like in the function example above. You cannot write code such as:
unsigned char data[n] = ... ;
valset_t* vs = (valset_t*)data; // BAD
This is bad in multiple different ways: alignment, padding, strict aliasing, endianess and so on.
It is possible to go the other way around though, using a unsigned char* to inspect or serialize a struct byte by byte. However, doing so doesn't solve the issues of padding bytes or endianess still.
I need to store the state of 8 relays in EEPROM. I didn't want to bother with shifting and I like using bitfields. So I wanted to define them like this:
typedef struct{
uint8_t RELAY0_STATE:1;
uint8_t RELAY1_STATE:1;
uint8_t RELAY2_STATE:1;
uint8_t RELAY3_STATE:1;
uint8_t RELAY4_STATE:1;
uint8_t RELAY5_STATE:1;
uint8_t RELAY6_STATE:1;
uint8_t RELAY7_STATE:1;
}relay_nvm_t;
relay_nvm_t relay_nvm;
In my main code flow, I set the state of each relay using relay_nvm variable. Example
...
if(something)
{
relay_nvm.RELAY0_STATE = 1;
relay_nvm.RELAY1_STATE = 0;
relay_nvm.RELAY2_STATE = 1;
relay_nvm.RELAY3_STATE = 0;
relay_nvm.RELAY4_STATE = 1;
relay_nvm.RELAY5_STATE = 1;
relay_nvm.RELAY6_STATE = 0;
relay_nvm.RELAY7_STATE = 1;
}
Then finally when I need to read/write to EEPROM, I just cast relay_nvm to uint8_t to read/write one byte to EEPROM. But I get error: conversion to non-scalar type requested error. Below are my functions.
static void NVM_Relay_Read(void)
{
relay_nvm = (relay_nvm_t)NVM_ReadEepromByte(NVM_RELAY_INDEX);
}
static void NVM_Relay_Write(relay_nvm_t rs)
{
NVM_WriteEepromByte(NVM_RELAY_INDEX, (uint8_t)rs);
}
Is there any way we can get over this error? I thought I could just do it by typecasting. The use of bitfields makes my job very easy and makes the code easy to understand.
I know that bitfields may not be safe in this case due to the padding but I think I can get over it using POP-PUSH (is it worth?)
I see more ways to handle this:
Use union.
Use pointer type-cast: *((uint8_t*)&relay_nvm)
Use uint8_t:
uint8_t relay_nvm;
#define RELAY0_MASK 1
#define RELAY1_MASK 2
#define RELAY2_MASK 4
...
#define RELAY7_MASK 128
// set exact relays state:
relay_nvm = RELAY0_MASK | RELAY2_MASK | RELAY4_MASK | ... ;
// set single relay (others left unchanged):
relay_nvm |= RELAY2_MASK;
// clear single relay (others left unchanged):
relay_nvm &= ~RELAY2_MASK;
// check current state of a relay:
if (relay_nvm & RELAY2_MASK) { ... }
I didn't want to bother with shifting and I like using bitfields.
If using the bitwise operators is a "bother", then you probably shouldn't be writing embedded systems code until you've gotten a hang of them... this is a very poor rationale for writing non-standard, non-portable code.
Unlike the bitwise version, bit-fields come with tons of problems: undefined bit order, endianess-dependence, poorly specified signedness, alignment & padding hiccups and so on. You are already writing platform-specific code with a uint8_t bit-fields, because those are not covered by the C standard.
If you insist on using bit-fields, you have to read the specific compiler docs about how they are implemented. Don't assume there are any guarantees regarding how things are allocated, because there is no standardization in this case.
Your specific problem is that you can't convert directly from a struct type (an aggregate - in plain english a "container type") to a uint8_t and back, for the same reason as why you can't do that with an array. You will have to use a pointer to the first element instead, then cast that one to uint8_t* and de-reference. But that comes with a bunch of other problems related to alignment, compatible types and "strict aliasing".
Structs and unions in general aren't very suitable for memory-mapping purposes, especially not when it comes to portability. At a minimum, you have to enable packing with #pragma pack(1) or similar compiler-specific commands.
So you should really consider dropping the bit-field entirely and use bitwise operators on a raw uint8_t, since you get rid of so many problems that way.
As a side-note, all variables stored in EEPROM must be volatile qualified, as must all pointers to them. Otherwise there is a big chance that the program goes haywire when you enable optimizations.
Our team is currently using some ported code from an old architecture to a new product based on the ARM Cortex M3 platform using a customized version of GCC 4.5.1. We are reading data from a communications link, and attempting to cast the raw byte array to a struct to cleanly parse the data. After casting the pointer to a struct and dereferencing, we are getting a warning: "dereferencing type-punned pointer will break strict-aliasing rules".
After some research, I've realized that since the char array has no alignment rules and the struct have to be word aligned, casting the pointers causes undefined behavior (a Bad Thing). I'm wondering if there is a better way to do what we're trying.
I know we can explicitly word-align the char array using GCC's "attribute ((aligned (4)))". I believe this will make our code "safer", but the warnings will still clutter up our builds, and I don't want to disable the warnings in case this situation arises again. What we want is a way to safely do what we are trying, that will still inform us if we attempt to do something unsafe in another place later. Since this is an embedded system, RAM usage and flash usage are important to some degree.
Portability (compiler and architecture) is not a huge concern, this is just for one product. However, if a portable solution exists, it would be preferred.
Here is the a (very simplified) example of what we are currently doing:
#define MESSAGE_TYPE_A 0
#define MESSAGE_TYPE_B 1
typedef struct MessageA __attribute__((__packed__))
{
unsigned char messageType;
unsigned short data1;
unsigned int data2;
}
typedef struct MessageB __attribute__((__packed__))
{
unsigned char messageType;
unsigned char data3;
unsigned char data4;
}
// This gets filled by the comm system, assume from a UART interrupt or similar
unsigned char data[100];
// Assume this gets called once we receive a full message
void ProcessMessage()
{
MessageA* messageA;
unsigned char messageType = data[0];
if (messageType == MESSAGE_TYPE_A)
{
// Cast data to struct and attempt to read
messageA = (MessageA*)data; // Not safe since data may not be word aligned
// This may cause undefined behavior
if (messageA->data1 == 4) // warning would be here, when we use the data at the pointer
{
// Perform some action...
}
}
// ...
// process different types of messages
}
As has already been pointed out, casting pointers about is a dodgy practice.
Solution: use a union
struct message {
unsigned char messageType;
union {
struct {
int data1;
short data2;
} A;
struct {
char data1[5];
int data2;
} B;
} data;
};
void func (...) {
struct message msg;
getMessage (&msg);
switch (msg.messageType) {
case TYPEA:
doStuff (msg.data.A.data1);
break;
case TYPEB:
doOtherStuff (msg.data.B.data1);
break;
}
}
By this means the compiler knows you're accessing the same data via different means, and the warnings and Bad Things will go away.
Of coure, you'll need to make sure the structure alignment and packing matches your message format. Beware endian issues and such if the machine on the other end of the link doesn't match.
Type punning through cast of types different than char * or a pointer to a signed/unsigned variant of char is not strictly conforming as it violates C aliasing rules (and sometimes alignment rules if no care is given).
However, gcc permits type punning through union types. Manpage of gcc explicitly documents it:
The practice of reading from a different union member than the one most recently written to (called "type-punning") is common. Even with
-fstrict-aliasing, type-punning is allowed, provided the memory is accessed through the union type.
To disable optimizations related to aliasing rules with gcc (and thus allow the program to break C aliasing rules), the program can be compiled with: -fno-strict-aliasing. Note that with this option enabled, the program is no longer strictly conforming, but you said portability is not a concern. For information, the Linux kernel is compiled with this option.
GCC has a -fno-strict-aliasing flag that will disable strict-aliasing-based optimizations and make your code safe.
If you're really looking for a way to "fix" it, you have to rethink the way your code works. You can't just overlay the structure the way you're trying, so you need to do something like this:
MessageA messageA;
messageA.messageType = data[0];
// Watch out - endianness and `sizeof(short)` dependent!
messageA.data1 = (data[1] << 8) + data[2];
// Watch out - endianness and `sizeof(int)` dependent!
messageA.data2 = (data[3] << 24) + (data[4] << 16)
+ (data[5] << 8) + data[6];
This method will let you avoid packing your structure, which might also improve its performance characteristics elsewhere in your code. Alternately:
MessageA messageA;
memcpy(&messageA, data, sizeof messageA);
Will do it with your packed structures. You would do the reverse operations to translate the structures back into a flat buffer if necessary.
Stop using packed structures and memcpy the individual fields into variables of the correct size and type. This is the safe, portable, clean way to do what you're trying to achieve. If you're lucky, gcc will optimize the tiny fixed-size memcpy into a few simple load and store instructions.
The Cortex M3 can handle unaligned accesses just fine. I have done this in similar packet processing systems with the M3. You don't need to do anything, you can just use the flag -fno-strict-aliasing to get rid of the warning.
For unaligned accesses, look at the linux macros get_unaligned/put_unaligned.
How can I disable structure padding in C without using pragma?
There is no standard way of doing this. The standard states that padding may be done at the discretion of the implementation. From C99 6.7.2.1 Structure and union specifiers, paragraph 12:
Each non-bit-field member of a structure or union object is aligned in an implementation-defined manner appropriate to its type.
Having said that, there's a couple of things you can try.
The first you've already discounted, using #pragma to try and convince the compiler not to pack. In any case, this is not portable. Nor are any other implementation-specific ways but you should check into them as it may be necessary to do it if you really need this capability.
The second is to order your fields in largest to smallest order such as all the long long types followed by the long ones, then all the int, short and finally char types. This will usually work since it's most often the larger types that have the more stringent alignment requirements. Again, not portable.
Thirdly, you can define your types as char arrays and cast the addresses to ensure there's no padding. But keep in mind that some architectures will slow down if the variables aren't aligned properly and still others will fail miserably (such as raising a BUS error and terminating your process, for example).
That last one bears some further explanation. Say you have a structure with the fields in the following order:
char C; // one byte
int I; // two bytes
long L; // four bytes
With padding, you may end up with the following bytes:
CxxxIIxxLLLL
where x is the padding.
However, if you define your structure as:
typedef struct { char c[7]; } myType;
myType n;
you get:
CCCCCCC
You can then do something like:
int *pInt = &(n.c[1]);
int *pLng = &(n.c[3]);
int myInt = *pInt;
int myLong = *pLng;
to give you:
CIILLLL
Again, unfortunately, not portable.
All these "solutions" rely on you having intimate knowledge of your compiler and the underlying data types.
Other than compiler options like pragma pack, you cannot, padding is in the C Standard.
You can always attempt to reduce padding by declaring the smallest types last in the structure as in:
struct _foo {
int a; /* No padding between a & b */
short b;
} foo;
struct _bar {
short b; /* 2 bytes of padding between a & b */
int a;
} bar;
Note for implementations which have 4 byte boundaries
On some architectures, the CPU itself will object if asked to work on misaligned data. To work around this, the compiler could generate multiple aligned read or write instructions, shift and split or merge the various bits. You could reasonably expect it to be 5 or 10 times slower than aligned data handling. But, the Standard doesn't require compilers to be prepared to do that... given the performance cost, it's just not in enough demand. The compilers that support explicit control over padding provide their own pragmas precisely because pragmas are reserved for non-Standard functionality.
If you must work with unpadded data, consider writing your own access routines. You might want to experimenting with types that require less alignment (e.g. use char/int8_t), but it's still possible that e.g. the size of structs will be rounded up to multiples of 4, which would frustrate packing structures tightly, in which case you'll need to implement your own access for the entire memory region.
Either you let compiler do padding, or tell it not to do using #pragma, either you just use some bunch of bytes like a char array, and you build all your data by yourself (shifting and adding bytes). This is really inefficient but you'll exactly control the layout of the bytes. I did that sometimes preparing network packets by hand, but in most case it's a bad idea, even if it's standard.
If you really want structs without padding: Define replacement datatypes for short, int, long, etc., using structs or classes that are composed only of 8 bit bytes. Then compose your higher level structs using the replacement datatypes.
C++'s operator overloading is very convenient, but you could achieve the same effect in C using structs instead of classes. The below cast and assignment implementations assume the CPU can handle misaligned 32bit integers, but other implementations could accommodate stricter CPUs.
Here is sample code:
#include <stdint.h>
#include <stdio.h>
class packable_int { public:
int8_t b[4];
operator int32_t () const { return *(int32_t*) b; }
void operator = ( int32_t n ) { *(int32_t*) b = n; }
};
struct SA {
int8_t c;
int32_t n;
} sa;
struct SB {
int8_t c;
packable_int n;
} sb;
int main () {
printf ( "sizeof sa %d\n", sizeof sa ); // sizeof sa 8
printf ( "sizeof sb %d\n", sizeof sb ); // sizeof sb 5
return 0;
}
We can disable structure padding in c program using any one of the following methods.
-> use __attribute__((packed)) behind definition of structure. for eg.
struct node {
char x;
short y;
int z;
} __attribute__((packed));
-> use -fpack-struct flag while compiling c code. for eg.
$ gcc -fpack-struct -o tmp tmp.c
Hope this helps.
Thanks.
I'm writing C cross-platform library but eventually I've got error in my unittests, but only on Windows machines. I've tracked the problem and found it's related to alignment of structures (I'm using arrays of structures to hold data for multiple similar objects). The problem is: memset(sizeof(struct)) and setting structures members one by one produce different byte-to-byte result and therefore memcmp() returns "not equal" result.
Here the code for illustration:
#include <stdio.h>
#include <string.h>
typedef struct {
long long a;
int b;
} S1;
typedef struct {
long a;
int b;
} S2;
S1 s1, s2;
int main()
{
printf("%d %d\n", sizeof(S1), sizeof(S2));
memset(&s1, 0xFF, sizeof(S1));
memset(&s2, 0x00, sizeof(S1));
s1.a = 0LL; s1.b = 0;
if (0 == memcmp(&s1, &s2, sizeof(S1)))
printf("Equal\n");
else
printf("Not equal\n");
return 0;
}
This code with MSVC 2003 # Windows produce following output:
16 8
Not equal
But the same code with GCC 3.3.6 # Linux works as expected:
12 8
Equal
This makes my unit-testing very hard.
Am I understand correctly that MSVC uses size of biggest native type (long long) to determine alignment to structure?
Can somebody give me advice how can I change my code to make it more robust against this strange alignment problem? In my real code I'm working with arrays of structures via generic pointers to execute memset/memcmp and I'm usually don't know exact type, I have only sizeof(struct) value.
Your unit test's expectation is wrong. It (or the code it tests) should not scan the structure's buffer byte-by-byte. For byte-precise data the code should create a byte buffer explicitly on stack or on heap and fill it with the extracts from each member. The extracts can be obtained in CPU-endianness-independent way by using the right shift operation against the integer values and casting the result by the byte type such as (unsigned char).
BTW, your snippet writes past s2. You could fix that by changing this
memset(&s2, 0x00, sizeof(S1));
s1.a = 0LL; s1.b = 0;
if (0 == memcmp(&s1, &s2, sizeof(S1)))
to this,
memset(&s2, 0x00, sizeof(S2));
s1.a = 0LL; s1.b = 0;
if (0 == memcmp(&s1, &s2, sizeof(S2)))
but the result is technically "undefined" because the alignment of members in the structures is compiler-specific.
GCC Manual:
Note that the alignment of any given struct or union type is required by the ISO C standard to be at least a perfect multiple of the lowest common multiple of the alignments of all of the members of the struct or union in question.
Also, this typically introduces an element of padding (i.e. filler bytes to have the structure aligned). You can use the #pragma with an argument of packed. Note, #pragmas are NOT a portable way of working. Unfortunately, this is also about the only way of working in your case.
References:
Here GCC on structure alignment.
MSDN structure alignment.
What we have done is used the #pragma pack to specify how big the objects should be:
#pragma pack(push, 2)
typedef struct {
long long a;
int b;
} S1;
typedef struct {
long a;
int b;
} S2;
#pragma pack(pop)
If you do this, the structures will be the same size on both platforms.
Note that this is not a 'strange' alignment problem. MSVC has chosen to ensure that the struct is aligned on a 64-bit boundary since it has a 64-bit member so it adds some padding at the end of the struct to ensure that arrays of those objects will have each element properly aligned. I'm actually surprised that GCC doesn't do the same.
I'm curious what you're unit testing does that hits a snag with this - most of the time alignment of structure members isn't necessary unless you need to match a binary file format or a wire protocol or you really need to reduce the memory used by a structure (especially used in embedded systems). Without knowing what you're trying to do in your tests I don't think a good suggestion can be given. Packing the structure might be a solution, but it comes at some cost - performance (especially on non-Intel platforms) and portability (how struct packing is set up is can be different from compiler to compiler). These may not matter to you, but there might be a better way to solve the problem in your case.
You can either do something like
#ifdef _MSC_VER
#pragma pack(push, 16)
#endif
/* your struct defs */
#ifdef _MSC_VER
#pragma pack(pop)
#endif
to give a compiler directive forcing alignment
Or go into the project options and change the default struct alignment [under Code Generation]
Structure padding for 64-bit values is different on different compilers. I've seen differences between even between gcc targets.
Note that explicitly padding to 64-bit alignment will only hide the problem. It will come back if you begin naively nesting structures, because the compilers will still disagree on the natural alignment of the inner structures.