For various reasons, I have some structs I want to force to be specific sizes (in this case 64 bytes and 512 bytes). Both however, are below the somewhat below the sizes I want them to be.
Is there anyway for me to tell the compiler to set them to these specific sizes and pad with zeros, or would I be best off just declaring an array inside the struct that makes up the excess space so that it aligns on the size I want?
You can use a union.
struct mystruct_s {
... /* who knows how long */
};
typedef union {
struct mystruct_s s;
unsigned char padding[512];
} mystruct;
This will ensure the union is 512 bytes or more. Then, you can ensure that it is no more than 512 bytes using a static assertion somewhere in your code:
/* Causes a compiler error if sizeof(mystruct) != 512 */
char array[sizeof(mystruct) != 512 ? -1 : 1];
If you are using C11, there is a better way to do this. I don't know anybody who uses C11 yet. The standard was published a matter of weeks ago.
_Static_assert(sizeof(mystruct) == 512, "mystruct must be 512 bytes");
Note that the only way to pad with zeroes is to put the zeroes there manually (calloc or memset). The compiler ignores padding bytes.
I don't think that there's any way to automatize this, at least in gcc which is the compiler I use. You have to pad your structs.
Be careful about automatic alignment of variables in your struct. For example
struct example{
char a;
int b;
}
does not take 5 bytes, but 8.
Related
I am porting an application to an ARM platform in C, the application also runs on an x86 processor, and must be backward compatible.
I am now having some issues with variable alignment. I have read the gcc manual for
__attribute__((aligned(4),packed)) I interpret what is being said as the start of the struct is aligned to the 4 byte boundry and the inside remains untouched because of the packed statement.
originally I had this but occasionally it gets placed unaligned with the 4 byte boundary.
typedef struct
{
unsigned int code;
unsigned int length;
unsigned int seq;
unsigned int request;
unsigned char nonce[16];
unsigned short crc;
} __attribute__((packed)) CHALLENGE;
so I change it to this.
typedef struct
{
unsigned int code;
unsigned int length;
unsigned int seq;
unsigned int request;
unsigned char nonce[16];
unsigned short crc;
} __attribute__((aligned(4),packed)) CHALLENGE;
The understand I stated earlier seems to be incorrect as both the struct is now aligned to a 4 byte boundary, and and the inside data is now aligned to a four byte boundary, but because of the endianess, the size of the struct has increased in size from 42 to 44 bytes. This size is critical as we have other applications that depend on the struct being 42 bytes.
Could some describe to me how to perform the operation that I require. Any help is much appreciated.
If you're depending on sizeof(yourstruct) being 42 bytes, you're about to be bitten by a world of non-portable assumptions. You haven't said what this is for, but it seems likely that the endianness of the struct contents matters as well, so you may also have a mismatch with the x86 there too.
In this situation I think the only sure-fire way to cope is to use unsigned char[42] in the parts where it matters. Start by writing a precise specification of exactly what fields are where in this 42-byte block, and what endian, then use that definition to write some code to translate between that and a struct you can interact with. The code will likely be either all-at-once serialisation code (aka marshalling), or a bunch of getters and setters.
This is one reason why reading whole structs instead of memberwise fails, and should be avoided.
In this case, packing plus aligning at 4 means there will be two bytes of padding. This happens because the size must be compatible for storing the type in an array with all items still aligned at 4.
I imagine you have something like:
read(fd, &obj, sizeof obj)
Because you don't want to read those 2 padding bytes which belong to different data, you have to specify the size explicitly:
read(fd, &obj, 42)
Which you can keep maintainable:
typedef struct {
//...
enum { read_size = 42 };
} __attribute__((aligned(4),packed)) CHALLENGE;
// ...
read(fd, &obj, obj.read_size)
Or, if you can't use some features of C++ in your C:
typedef struct {
//...
} __attribute__((aligned(4),packed)) CHALLENGE;
enum { CHALLENGE_read_size = 42 };
// ...
read(fd, &obj, CHALLENGE_read_size)
At the next refactoring opportunity, I would strongly suggest you start reading each member individually, which can easily be encapsulated within a function.
I've been moving structures back and forth from Linux, Windows, Mac, C, Swift, Assembly, etc.
The problem is NOT that it can't be done, the problem is that you can't be lazy and must understand your tools.
I don't see why you can't use:
typedef struct
{
unsigned int code;
unsigned int length;
unsigned int seq;
unsigned int request;
unsigned char nonce[16];
unsigned short crc;
} __attribute__((packed)) CHALLENGE;
You can use it and it doesn't require any special or clever code. I write a LOT of code that communicates to ARM. Structures are what make things work. __attribute__ ((packed)) is my friend.
The odds of being in a "world of hurt" are nil if you understand what is going on with both.
Finally, I can't for the life make out how you get 42 or 44. Int is either 4 or 8 bytes (depending on the compiler). That puts the number at either 16+16+2=34 or 32+16+2=50 -- assuming it is truly packed.
As I say, knowing your tools is part of your problem.
What is your true goal?
If it's to deal with data that's in a file or on the wire in a particular format what you should do is write up some marshaling/serialization routines that move the data between the compiler struct that represents how you want to deal with the data inside the program and a char array that deals with how the data looks on the wire/file.
Then all that needs to be dealt with carefully and possibly have platform specific code is the marshaling routines. And you can write some nice-n-nasty unit tests to ensure that the marshaled data gets to and from the struct properly no matter what platform you might have to port to today and in the future.
I would guess that the problem is that 42 isn't divisible by 4, and so they get out of alignment if you put several of these structs back to back (e.g. allocate memory for several of them, determining the size with sizeof). Having the size as 44 forces the alignment in these cases as you requested. However, if the internal offset of each struct member remains the same, you can treat the 44 byte struct as though it was 42 bytes (as long as you take care to align any following data at the correct boundary).
One trick to try might be putting both of these structs inside a single union type and only use 42-byte version from within each such union.
As I am using linux, I have found that by echo 3 > /proc/cpu/alignment it will issue me with a warning, and fix the alignment issue. This is a work around but it is very helpful with locating where the structures are failing to be misaligned.
This question already has answers here:
Why isn't sizeof for a struct equal to the sum of sizeof of each member?
(13 answers)
Closed 6 years ago.
I was looking into structs in C when I noticed this oddity.
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
struct vulnStruct {
char start[20];
char overflow[10];
char *controlledPointer;
};
int main(int argc, char *argv[]){
struct vulnStruct *test = malloc(sizeof(struct vulnStruct));
printf("Sizeof vulnStruct: %d\r\n", sizeof(struct vulnStruct));
printf("Sizeof start: %d\r\n", sizeof(test->start));
printf("Sizeof overflow: %d\r\n", sizeof(test->overflow));
printf("Sizeof controlledPointer: %d\n\r", sizeof(test->controlledPointer));
free(test);
return 0;
}
When compiled this program should output the sizeof the vulnStruct and its elements. Manually adding up the struct shows that it should be 38 bytes long however when this program is run it shows that the struct's size is instead 40 bytes long. At first I thought it was just padding for the struct as answered here (Size of a structure in C) however, when I add a 2 byte field (uint16_t) to the end of the struct in an attempt to fill the padding it just increases the struct by 2 bytes. Is this because the controlledPointer is being padded to 10 bytes by the compiler and won't let anything else use that space or is there something like a canary at the end that is taking up the space. If it is just padding is there anyway to arrange the elements so that the padding can be used by another field?
EDIT:
It would probably help if I get the output of the program as well
Sizeof vulnStruct: 40
Sizeof start: 20
Sizeof overflow: 10
Sizeof controlledPointer: 8
Thanks in advance!
The padding is between overflow[10]; and *controlledPointer.
The problem is that it's doing 4-byte alignment of the fields--something that almost always is worthwhile to do. Two bytes of memory is almost always cheaper than the time penalty for a non-aligned memory access.
The only cases where this would be an issue is if you're trying to match some other structure or when you have a huge number of items. Declare it accordingly in these cases. My C is too rusty to tell you how to declare it, though.
The problem with structures is that they have "padding" meaning that if you have 1 char in a structure the structure will MOST PROBABLY have a size of 4bytes/8bytes depending on the data bus of your processor. This is due to the fact that to optimize the access speed, when compiling the structure is padded with 0xff bytes.
There is some compilation directives which don't use padding but this is not recommended UNLESS you are programming an actual device. This can be done in gcc(the default unix compile) with the following directive but it depends on each compiler:
#pragma (push,pack 1)
//define structure(...)
#pragma (pop)
The pragma in this case has it's own stack, look at this post if you want a clearer idea on padding and the pragma directive
Note that your results will very much vary per target/compiler.
Here's what I get when I try to find out more about padding specifically:
$ clang -Wpadded -Wno-format padd.c
padd.c:10:11: warning: padding struct 'struct vulnStruct' with 2 bytes to align 'controlledPointer' [-Wpadded]
char *controlledPointer;
^
1 warning generated.
This explains which field was aligned and for what reasons.
Your controlledPointer needs to start at an address that is a multiple of 8.
struct vulnStruct {
char start[20];
char overflow[10]; // 30 bytes
// 30 is not a multiple of 8.
// 2 bytes padding here
// now controlledPointer starts at an address that is a multiple of 8
char *controlledPointer;
};
I want to use a structure as given below on my ARM Cortex M0. I am using C programming.
struct path_table_t {
uint8_t lut_index;
uint8_t flag : 1;
};
flag field is made to be single bit by bit fields. How will be the memory allocation for the array of above mentioned structure will happen?
Will I get the benefit of bit fields as saving in total memory?
Most likely the sizeof(struct path_table_t) == 2. That is, structure with 2 elements, with each being 8 bits. 7 bits of flag should remain unused, but are still reserved.
Note that C standard gives a lot of room for compiler implementation to vary the size required by the bit fields. For example (N1570, 6.7.2.1 p11):
An implementation may allocate any addressable storage unit large enough to hold a bit-
field.
and
The alignment of the addressable storage unit is unspecified.
So the space being reserved by the flag and its padding might be even larger.
If you wish to be sure, check the size with sizeof. Just remember that changing compiler or compiler settings might affect the result.
To answer your question Will I get the benefit of bit fields as saving in total memory? NO
consider two struct's
struct path_table1_t {
uint8_t lut_index;
uint8_t flag : 1;
}a;
and
struct path_table2_t {
uint8_t lut_index;
uint8_t flag;
}b;
sizeof(a) and sizeof(b) is equal.
But incase you are planning to use all the bits in the given bit field then the memory allocated is reduced. Like
struct path_table3_t {
uint8_t lut_index : 1;
uint8_t flag : 1;
}c;
In this case the sizeof(c) is the sizeof(uint8_t) which will be 1 byte.
To summarize:
sizeof(c) < sizeof(a) and sizeof(a) = sizeof(b)
If you are very much interested to save memory then use the keyword __attribute__((__packed__)) , where this directive would make compiler to squeeze in the memory allocated for this struct upto which the user is using.
Memory allocation for bit fields is implementation defined.
Let's consider your example:
struct path_table_t {
uint8_t lut_index;
uint8_t flag : 1;
};
int main()
{
struct path_table_t p;
printf("%d\n",sizeof(p));
}
Here you want totally 9 bits of usage so it will be padded to 2 bytes.
If your structure was like
struct path_table_t {
uint8_t lut_index : 1;
uint8_t flag : 1;
};
int main()
{
struct path_table_t p;
printf("%d\n",sizeof(p));
}
You see here only 2 bits is required so your sizeof(struct) would be 1 byte which answers your question.
Yes using bit fields can save memory as shown above.
Say I have a structure:
struct myStruct
{
int a;
short b;
char c;
};
In Windows, MSDN states that int takes 4 bytes, short takes 2 bytes and char takes 1 byte. This totals up to 7 bytes.
I understand that most compilers will pad structures with an unspecified number of bytes to improve alignment. So, when I execute this program,
#include <stdio.h>
int main(void) {
printf("%d\n", sizeof(struct myStruct));
return 0;
}
I get an output of 8. This means 1 byte was padded.
Is there any way I can maintainably determine struct sizes in code (short of adding up individual sizeofs)?
I ask this because later, if I need to change my structure to include about fifteen elements more and if I add five more structures, all my struct sizeofs will change causing things to get messy.
You can enforce it:
#define C_ASSERT(expr) extern char CAssertExtern[(expr)?1:-1]
C_ASSERT(sizeof(struct myStruct) == 7); // or 8, whichever you want
Whenever the size diverges from 7, the code will simply cease to compile.
You can do similar things to enforce offsets of structure members. You'll need the offsetof() macro for that.
You can try pragma pack. It helps maintaining an alignment and has many flexibility. check this
http://publib.boulder.ibm.com/infocenter/comphelp/v101v121/index.jsp?topic=/com.ibm.xlcpp101.aix.doc/compiler_ref/pragma_pack.html
Update: just now saw your comment, perhaps pack is not your solution then
I am porting an application to an ARM platform in C, the application also runs on an x86 processor, and must be backward compatible.
I am now having some issues with variable alignment. I have read the gcc manual for
__attribute__((aligned(4),packed)) I interpret what is being said as the start of the struct is aligned to the 4 byte boundry and the inside remains untouched because of the packed statement.
originally I had this but occasionally it gets placed unaligned with the 4 byte boundary.
typedef struct
{
unsigned int code;
unsigned int length;
unsigned int seq;
unsigned int request;
unsigned char nonce[16];
unsigned short crc;
} __attribute__((packed)) CHALLENGE;
so I change it to this.
typedef struct
{
unsigned int code;
unsigned int length;
unsigned int seq;
unsigned int request;
unsigned char nonce[16];
unsigned short crc;
} __attribute__((aligned(4),packed)) CHALLENGE;
The understand I stated earlier seems to be incorrect as both the struct is now aligned to a 4 byte boundary, and and the inside data is now aligned to a four byte boundary, but because of the endianess, the size of the struct has increased in size from 42 to 44 bytes. This size is critical as we have other applications that depend on the struct being 42 bytes.
Could some describe to me how to perform the operation that I require. Any help is much appreciated.
If you're depending on sizeof(yourstruct) being 42 bytes, you're about to be bitten by a world of non-portable assumptions. You haven't said what this is for, but it seems likely that the endianness of the struct contents matters as well, so you may also have a mismatch with the x86 there too.
In this situation I think the only sure-fire way to cope is to use unsigned char[42] in the parts where it matters. Start by writing a precise specification of exactly what fields are where in this 42-byte block, and what endian, then use that definition to write some code to translate between that and a struct you can interact with. The code will likely be either all-at-once serialisation code (aka marshalling), or a bunch of getters and setters.
This is one reason why reading whole structs instead of memberwise fails, and should be avoided.
In this case, packing plus aligning at 4 means there will be two bytes of padding. This happens because the size must be compatible for storing the type in an array with all items still aligned at 4.
I imagine you have something like:
read(fd, &obj, sizeof obj)
Because you don't want to read those 2 padding bytes which belong to different data, you have to specify the size explicitly:
read(fd, &obj, 42)
Which you can keep maintainable:
typedef struct {
//...
enum { read_size = 42 };
} __attribute__((aligned(4),packed)) CHALLENGE;
// ...
read(fd, &obj, obj.read_size)
Or, if you can't use some features of C++ in your C:
typedef struct {
//...
} __attribute__((aligned(4),packed)) CHALLENGE;
enum { CHALLENGE_read_size = 42 };
// ...
read(fd, &obj, CHALLENGE_read_size)
At the next refactoring opportunity, I would strongly suggest you start reading each member individually, which can easily be encapsulated within a function.
I've been moving structures back and forth from Linux, Windows, Mac, C, Swift, Assembly, etc.
The problem is NOT that it can't be done, the problem is that you can't be lazy and must understand your tools.
I don't see why you can't use:
typedef struct
{
unsigned int code;
unsigned int length;
unsigned int seq;
unsigned int request;
unsigned char nonce[16];
unsigned short crc;
} __attribute__((packed)) CHALLENGE;
You can use it and it doesn't require any special or clever code. I write a LOT of code that communicates to ARM. Structures are what make things work. __attribute__ ((packed)) is my friend.
The odds of being in a "world of hurt" are nil if you understand what is going on with both.
Finally, I can't for the life make out how you get 42 or 44. Int is either 4 or 8 bytes (depending on the compiler). That puts the number at either 16+16+2=34 or 32+16+2=50 -- assuming it is truly packed.
As I say, knowing your tools is part of your problem.
What is your true goal?
If it's to deal with data that's in a file or on the wire in a particular format what you should do is write up some marshaling/serialization routines that move the data between the compiler struct that represents how you want to deal with the data inside the program and a char array that deals with how the data looks on the wire/file.
Then all that needs to be dealt with carefully and possibly have platform specific code is the marshaling routines. And you can write some nice-n-nasty unit tests to ensure that the marshaled data gets to and from the struct properly no matter what platform you might have to port to today and in the future.
I would guess that the problem is that 42 isn't divisible by 4, and so they get out of alignment if you put several of these structs back to back (e.g. allocate memory for several of them, determining the size with sizeof). Having the size as 44 forces the alignment in these cases as you requested. However, if the internal offset of each struct member remains the same, you can treat the 44 byte struct as though it was 42 bytes (as long as you take care to align any following data at the correct boundary).
One trick to try might be putting both of these structs inside a single union type and only use 42-byte version from within each such union.
As I am using linux, I have found that by echo 3 > /proc/cpu/alignment it will issue me with a warning, and fix the alignment issue. This is a work around but it is very helpful with locating where the structures are failing to be misaligned.