This question already has answers here:
Structure padding and packing
(11 answers)
Closed 6 days ago.
I find a lot of ambiguity while working on structure padding in C.
I need the following :
Trustworthy resource or url to understand structure padding in c.
How memory is allocated to structures, by individual members of by pages (like 1 page=4 bytes for 32 bit machines and 8 bytes for 64 bit machines).
#include <stdio.h>
typedef struct
{
unsigned short Flag;
char DebounceSamples;
unsigned short Device;
char *BufferPtr;
char *OverridePtr;
} IoHwAb_DIN_DescriptorType;
int main()
{
IoHwAb_DIN_DescriptorType tt;
printf("%lu",sizeof(tt));
return 0;
}
When I execute this I see the output is 24 for the size of struct. I kind of understand why, but also I have a confusion. I need a clear explanation on this.
Each object type has some alignment requirement. Any object of that type is required to start at an address that is a multiple of the alignment requirement. Each C implementation may determine its own alignment requirements for its object types, but alignment requirements are usually set for hardware efficiency. Most notably, the requirements are set so that any correctly aligned object will not cross a memory boundary that requires an extra load or store operation.
For example, consider hardware that can perform transfers of data between the processor and memory in four-byte chunks starting at addresses that are multiples of four bytes. The hardware can transfer bytes addressed 0-3 in one operation, bytes addressed 4-7 in one operation, bytes address 8-11 in one operation, and so on. However, if we wanted bytes 6-9, that would require two memory transfer operations, one to get byes 4-7 and one to get bytes 8-11. In this hardware, we would set the alignment requirement of a four-byte int to be four bytes, so that all four bytes of the int are always inside one of these aligned groups.
For a two-byte short, we could allow a short object to be in bytes 5-6 in memory, because we can fetch both bytes in one operation (loading bytes 4-7). However, if we have an array of short with the first in bytes 5-6, then the next would be in 7-8, and we cannot fetch that short in one operation; we would have to get bytes 4-7 and 8-11. So, to ensure that short objects always have good alignment, we require them to start at multiples of two bytes.
Once the alignment requirements are set, the algorithm usually used to lay out a structure is:
Each member in the structure has some size s and some alignment requirement a.
The compiler starts with a size S set to zero and an alignment requirement A set to one (byte).
The compiler processes each member in the structure in order:
Consider the member’s alignment requirement a. If S is not currently a multiple of a, then add just enough bytes to S so that it is a multiple of a. This determines where the member will go; it will go at offset S from the beginning of the structure (for the current value of S).
Set A to the least common multiple1 of A and a.
Add s to S, to set aside space for the member.
When the above process is done for each member, consider the structure’s alignment requirement A. If S is not currently a multiple of A, then add just enough to S so that it is a multiple of A.
The size of the structure is the value of S when the above is done.
Additionally:
If any member is an array, its size is the number of elements multiplied by the size of each element, and its alignment requirement is the alignment requirement of an element.
If any member is a structure, its size and alignment requirement are calculated as above.
If any member is a union, its size is the size of its largest member plus just enough to make it a multiple of the least common multiple1 of the alignments of all the members.
The rules above follow largely from logic; they put each member where it must be to satisfy alignment requirements without using more space than necessary. The C standard allows implementations to add more padding between elements or at the end of the structure, but this is generally not done. (It could be done for special purposes, such as adding padding during debugging to test whether some bug manifests without padding but not with it, to give clues about the nature of the bug.)
In your structure, char is one byte with alignment requirement one byte, unsigned short is probably two bytes with requirement two bytes, and char * is probably eight bytes with requirement eight bytes. Then the structure layout proceeds:
S is zero and A is one byte.
No padding is needed, since S is a multiple of the two-byte requirement for unsigned short Flag.
unsigned short Flag adds two to S and makes A two bytes.
No padding is needed, since S is a multiple of the one-byte requirement for char DebounceSamples.
char DebounceSamples adds one to S, making it three, and does not change A, leaving it two bytes.
One byte of padding is needed, because S is not a multiple of the two-byte requirement for unsigned short Device, so S is increased to four.
unsigned short Device adds two to S, making it six, and does not change A, leaving it two bytes.
Two bytes of padding are needed, because S is not a multiple of the eight-byte requirement for char *BufferPtr, so S is increased to eight.
char *BufferPtr adds eight to S, making it 16, and changes A to eight bytes.
No padding is needed because S is a multiple of the eight-byte requirement for char *OverridePtr.
char *OverridePtr adds eight to S, making it 24, and does not change A.
At the end, the size of the structure is 24 bytes, and its alignment requirement is eight bytes.
Footnote
1 I have worded this for a general case as using the least common multiple of alignment requirements. However, since alignment requirements are always powers of two, the least common multiple of any set of alignment requirements is the largest of them.
I was going through the text "The C Programming Language" by Kernighan and Ritchie. While discussing about bit-fields at the end of that section, the authors say:
"Fields are assigned left to right on some machines and right to left on others. This means that although fields are useful for maintaining internally-defined data structures, the question of which end comes first has to be carefully considered when picking apart externally-defined data; programs that depend on such things are not portable."
- The C Programming Language [2e] by Kernighan & Ritchie [Section 6.9, p.150]
Strictly I do not get the meaning of these lines. Can anyone please explain me with a possible diagram?
PS: Well I have taken a computer organization and architecture course. I know how computers deal with bits and bytes. In a computer system, the smallest unit of information is a single bit which can be either 0 or 1. 8 such bits form a byte. Memories are byte-addressable, which means that each byte in the memory has an address associated with it. But usually, the processors have word lengths as 2 bytes (very old systems),4 bytes, 8 bytes... This means in one memory cycle, the CPU can take up a word length number of bytes from the main memory and put it inside its registers. Now how these bytes are placed in registers depends on the endianness of the system.
But I do not get what the authors mean by "left to right" or "right to left". The words seem like they are related to the endianness but endianness depends on the CPU and C compilers have nothing to do with it... The question which comes to my mind is "left to right" of "what"? What object are the authors referring to?
When a structure contains bit-fields, the C implementation uses some storage unit to hold them (or multiple storage units if needed). The storage unit might be one eight-bit byte or it might be four bytes, or it might be other sizes—this is a determination made by each C implementation. The C standard only requires that it be addressable, which effectively means it has to be a whole number of bytes.
Once we have a storage unit, it is some number of bits. Say it is 32 bits, and number the bits from 31 to 0, where, if we consider the bits to represent a binary numeral, bit 0 represents 20, and bit 31 represents 231. Note that Kernighan and Ritchie are imprecise to use “left” and “right” here. There is no inherent left or right. We usually write numerals with the most significant digits on the left, so we might consider bit 31 to be the leftmost and bit 0 to be the rightmost.
Now we have a storage unit with some number of bits and some labeling for those bits (31 to 0 or left to right). Say you want to put two bit-fields in them, say fields of width 7 and 5.
Which 7 of the bits from bit 31 to bit 0 are used for the first field? Which 5 of the bits are used for the second field?
We could use bits 31-25 for the first field and bits 24-20 for the second field. Or we could use bits 6-0 for the first field and bits 11-7 for the second field.
In theory, we could also use bits 27-21 for the first field and bits 15-11 for the second field. However, the C standard does say that “If enough space remains, a bit-field that immediately follows another bit-field in a structure shall be packed into adjacent bits of the same unit” (C 2018 6.7.2.1 11). “Adjacent” is not formally defined, but we can assume it means consecutively numbered bits. So, if the C implementation puts the first field in bits 31-25, it is required to put the second field in bits 24-20. Conversely, it it puts the first field in bits 6-0, it must put the second field in 11-7.
Thus, the C standard requires an implementation to arrange successive bit-fields in a storage unit from left-to-right or from right-to-left, but it does not say which.
(I do not see anything in the standard that says the first field must start at one end of the storage unit or the other, rather than somewhere in the middle. That would lead to wasting some bits.)
When you write:
struct {
unsigned int version: 4;
unsigned int length: 4;
unsigned char dcsn;
you end up with a big headache you weren't expecting because your code is non-portable.
When you set version to 4 and length to 5, some systems may set the first byte of the structure to 0x45 and other systems may set the first byte of the structure to 0x54.
When I went to college this thing was #ifdef'd as follows (incorrect):
struct {
#if BIG_ENDIAN
unsigned int version: 4;
unsigned int length: 4;
#else
unsigned int length: 4;
unsigned int version: 4;
#endif
unsigned char dcsn;
but this is still rolling the dice as there's no rule that the order of the bits in the bytes in a bitfield corresponds to the order of bytes in the word in the machine. I would not be surprised that when you cross-compile the bit order in the struct comes from the host machine's rules while the bit order of integers comes from the target machine's rules (as it must). In theory the code could be corrected by having a separate #ifdef for BIG_ENDIAN_BITFIELD but I've never seen it done.
Here is some demonstration code. The only goal is to demonstrate what you are asking about. Clean coding etc. is neglected.
#include <stdio.h>
#include <stdint.h>
union
{
uint32_t Everything;
struct
{
uint32_t FirstMentionedBit : 1;
uint32_t FewOTherBits :30;
uint32_t LastMentionedBit : 1;
} bitfield;
} Demonstration;
int main()
{
Demonstration.Everything =0;
Demonstration.bitfield.LastMentionedBit=1;
printf("%x\n", Demonstration.Everything);
Demonstration.Everything =0;
Demonstration.bitfield.FirstMentionedBit=1;
printf("%x\n", Demonstration.Everything);
return 0;
}
If you use this here https://www.tutorialspoint.com/compile_c_online.php
the output is
80000000
1
But in other environments it might easily be
1
80000000
This is because compilers are free to consider the first mentioned bit the MSB or the LSB and correspondingly the last mentioned bit to be the LSB or MSB.
And that is what your quote describes.
On the question 'why do we need to use bit-fields?', searching on Google I found that bit fields are used for flags.
Now I am curious,
Is it the only way bit-fields are used practically?
Do we need to use bit fields to save space?
A way of defining bit field from the book:
struct {
unsigned int is_keyword : 1;
unsigned int is_extern : 1;
unsigned int is_static : 1;
} flags;
Why do we use int?
How much space is occupied?
I am confused why we are using int, but not short or something smaller than an int.
As I understand only 1 bit is occupied in memory, but not the whole unsigned int value. Is it correct?
A quite good resource is Bit Fields in C.
The basic reason is to reduce the used size. For example, if you write:
struct {
unsigned int is_keyword;
unsigned int is_extern;
unsigned int is_static;
} flags;
You will use at least 3 * sizeof(unsigned int) or 12 bytes to represent three small flags, that should only need three bits.
So if you write:
struct {
unsigned int is_keyword : 1;
unsigned int is_extern : 1;
unsigned int is_static : 1;
} flags;
This uses up the same space as one unsigned int, so 4 bytes. You can throw 32 one-bit fields into the struct before it needs more space.
This is sort of equivalent to the classical home brew bit field:
#define IS_KEYWORD 0x01
#define IS_EXTERN 0x02
#define IS_STATIC 0x04
unsigned int flags;
But the bit field syntax is cleaner. Compare:
if (flags.is_keyword)
against:
if (flags & IS_KEYWORD)
And it is obviously less error-prone.
Now I am curious, [are flags] the only way bitfields are used practically?
No, flags are not the only way bitfields are used. They can also be used to store values larger than one bit, although flags are more common. For instance:
typedef enum {
NORTH = 0,
EAST = 1,
SOUTH = 2,
WEST = 3
} directionValues;
struct {
unsigned int alice_dir : 2;
unsigned int bob_dir : 2;
} directions;
Do we need to use bitfields to save space?
Bitfields do save space. They also allow an easier way to set values that aren't byte-aligned. Rather than bit-shifting and using bitwise operations, we can use the same syntax as setting fields in a struct. This improves readability. With a bitfield, you could write
directions.alice_dir = WEST;
directions.bob_dir = SOUTH;
However, to store multiple independent values in the space of one int (or other type) without bitfields, you would need to write something like:
#define ALICE_OFFSET 0
#define BOB_OFFSET 2
directions &= ~(3<<ALICE_OFFSET); // clear Alice's bits
directions |= WEST<<ALICE_OFFSET; // set Alice's bits to WEST
directions &= ~(3<<BOB_OFFSET); // clear Bob's bits
directions |= SOUTH<<BOB_OFFSET; // set Bob's bits to SOUTH
The improved readability of bitfields is arguably more important than saving a few bytes here and there.
Why do we use int? How much space is occupied?
The space of an entire int is occupied. We use int because in many cases, it doesn't really matter. If, for a single value, you use 4 bytes instead of 1 or 2, your user probably won't notice. For some platforms, size does matter more, and you can use other data types which take up less space (char, short, uint8_t, etc.).
As I understand only 1 bit is occupied in memory, but not the whole unsigned int value. Is it correct?
No, that is not correct. The entire unsigned int will exist, even if you're only using 8 of its bits.
Another place where bitfields are common are hardware registers. If you have a 32 bit register where each bit has a certain meaning, you can elegantly describe it with a bitfield.
Such a bitfield is inherently platform-specific. Portability does not matter in this case.
We use bit fields mostly (though not exclusively) for flag structures - bytes or words (or possibly larger things) in which we try to pack tiny (often 2-state) pieces of (often related) information.
In these scenarios, bit fields are used because they correctly model the problem we're solving: what we're dealing with is not really an 8-bit (or 16-bit or 24-bit or 32-bit) number, but rather a collection of 8 (or 16 or 24 or 32) related, but distinct pieces of information.
The problems we solve using bit fields are problems where "packing" the information tightly has measurable benefits and/or "unpacking" the information doesn't have a penalty. For example, if you're exposing 1 byte through 8 pins and the bits from each pin go through their own bus that's already printed on the board so that it leads exactly where it's supposed to, then a bit field is ideal. The benefit in "packing" the data is that it can be sent in one go (which is useful if the frequency of the bus is limited and our operation relies on frequency of its execution), and the penalty of "unpacking" the data is non-existent (or existent but worth it).
On the other hand, we don't use bit fields for booleans in other cases like normal program flow control, because of the way computer architectures usually work. Most common CPUs don't like fetching one bit from memory - they like to fetch bytes or integers. They also don't like to process bits - their instructions often operate on larger things like integers, words, memory addresses, etc.
So, when you try to operate on bits, it's up to you or the compiler (depending on what language you're writing in) to write out additional operations that perform bit masking and strip the structure of everything but the information you actually want to operate on. If there are no benefits in "packing" the information (and in most cases, there aren't), then using bit fields for booleans would only introduce overhead and noise in your code.
To answer the original question »When to use bit-fields in C?« … according to the book "Write Portable Code" by Brian Hook (ISBN 1-59327-056-9, I read the German edition ISBN 3-937514-19-8) and to personal experience:
Never use the bitfield idiom of the C language, but do it by yourself.
A lot of implementation details are compiler-specific, especially in combination with unions and things are not guaranteed over different compilers and different endianness. If there's only a tiny chance your code has to be portable and will be compiled for different architectures and/or with different compilers, don't use it.
We had this case when porting code from a little-endian microcontroller with some proprietary compiler to another big-endian microcontroller with GCC, and it was not fun. :-/
This is how I have used flags (host byte order ;-) ) since then:
# define SOME_FLAG (1 << 0)
# define SOME_OTHER_FLAG (1 << 1)
# define AND_ANOTHER_FLAG (1 << 2)
/* test flag */
if ( someint & SOME_FLAG ) {
/* do this */
}
/* set flag */
someint |= SOME_FLAG;
/* clear flag */
someint &= ~SOME_FLAG;
No need for a union with the int type and some bitfield struct then. If you read lots of embedded code those test, set, and clear patterns will become common, and you spot them easily in your code.
Why do we need to use bit-fields?
When you want to store some data which can be stored in less than one byte, those kind of data can be coupled in a structure using bit fields.
In the embedded word, when one 32 bit world of any register has different meaning for different word then you can also use bit fields to make them more readable.
I found that bit fields are used for flags. Now I am curious, is it the only way bit-fields are used practically?
No, this not the only way. You can use it in other ways too.
Do we need to use bit fields to save space?
Yes.
As I understand only 1 bit is occupied in memory, but not the whole unsigned int value. Is it correct?
No. Memory only can be occupied in multiple of bytes.
Bit fields can be used for saving memory space (but using bit fields for this purpose is rare). It is used where there is a memory constraint, e.g., while programming in embedded systems.
But this should be used only if extremely required because we cannot have the address of a bit field, so address operator & cannot be used with them.
A good usage would be to implement a chunk to translate to—and from—Base64 or any unaligned data structure.
struct {
unsigned int e1:6;
unsigned int e2:6;
unsigned int e3:6;
unsigned int e4:6;
} base64enc; // I don't know if declaring a 4-byte array will have the same effect.
struct {
unsigned char d1;
unsigned char d2;
unsigned char d3;
} base64dec;
union base64chunk {
struct base64enc enc;
struct base64dec dec;
};
base64chunk b64c;
// You can assign three characters to b64c.enc, and get four 0-63 codes from b64dec instantly.
This example is a bit naive, since Base64 must also consider null-termination (i.e. a string which has not a length l so that l % 3 is 0). But works as a sample of accessing unaligned data structures.
Another example: Using this feature to break a TCP packet header into its components (or other network protocol packet header you want to discuss), although it is a more advanced and less end-user example. In general: this is useful regarding PC internals, SO, drivers, an encoding systems.
Another example: analyzing a float number.
struct _FP32 {
unsigned int sign:1;
unsigned int exponent:8;
unsigned int mantissa:23;
}
union FP32_t {
_FP32 parts;
float number;
}
(Disclaimer: Don't know the file name / type name where this is applied, but in C this is declared in a header; Don't know how can this be done for 64-bit floating-point numbers since the mantissa must have 52 bits and—in a 32 bit target—ints have 32 bits).
Conclusion: As the concept and these examples show, this is a rarely used feature because it's mostly for internal purposes, and not for day-by-day software.
To answer the parts of the question no one else answered:
Ints, not Shorts
The reason to use ints rather than shorts, etc. is that in most cases no space will be saved by doing so.
Modern computers have a 32 or 64 bit architecture and that 32 or 64 bits will be needed even if you use a smaller storage type such as a short.
The smaller types are only useful for saving memory if you can pack them together (for example a short array may use less memory than an int array as the shorts can be packed together tighter in the array). For most cases, when using bitfields, this is not the case.
Other uses
Bitfields are most commonly used for flags, but there are other things they are used for. For example, one way to represent a chess board used in a lot of chess algorithms is to use a 64 bit integer to represent the board (8*8 pixels) and set flags in that integer to give the position of all the white pawns. Another integer shows all the black pawns, etc.
You can use them to expand the number of unsigned types that wrap. Ordinary you would have only powers of 8,16,32,64... , but you can have every power with bit-fields.
struct a
{
unsigned int b : 3 ;
} ;
struct a w = { 0 } ;
while( 1 )
{
printf("%u\n" , w.b++ ) ;
getchar() ;
}
To utilize the memory space, we can use bit fields.
As far as I know, in real-world programming, if we require, we can use Booleans instead of declaring it as integers and then making bit field.
If they are also values we use often, not only do we save space, we can also gain performance since we do not need to pollute the caches.
However, caching is also the danger in using bit fields since concurrent reads and writes to different bits will cause a data race and updates to completely separate bits might overwrite new values with old values...
Bitfields are much more compact and that is an advantage.
But don't forget packed structures are slower than normal structures. They are also more difficult to construct since the programmer must define the number of bits to use for each field. This is a disadvantage.
Why do we use int? How much space is occupied?
One answer to this question that I haven't seen mentioned in any of the other answers, is that the C standard guarantees support for int. Specifically:
A bit-field shall have a type that is a qualified or unqualified version of _Bool, signed int, unsigned int, or some other implementation defined type.
It is common for compilers to allow additional bit-field types, but not required. If you're really concerned about portability, int is the best choice.
Nowadays, microcontrollers (MCUs) have peripherals, such as I/O ports, ADCs, DACs, onboard the chip along with the processor.
Before MCUs became available with the needed peripherals, we would access some of our hardware by connecting to the buffered address and data buses of the microprocessor. A pointer would be set to the memory address of the device and if the device saw its address along with the R/W signal and maybe a chip select, it would be accessed.
Oftentimes we would want to access individual or small groups of bits on the device.
In our project, we used this to extract a page table entry and page directory entry from a given memory address:
union VADDRESS {
struct {
ULONG64 BlockOffset : 16;
ULONG64 PteIndex : 14;
ULONG64 PdeIndex : 14;
ULONG64 ReservedMBZ : (64 - (16 + 14 + 14));
};
ULONG64 AsULONG64;
};
Now suppose, we have an address:
union VADDRESS tempAddress;
tempAddress.AsULONG64 = 0x1234567887654321;
Now we can access PTE and PDE from this address:
cout << tempAddress.PteIndex;
what i am trying to accomplish is
user enters bit field widths, say 17 5 8 19 0 (can be more or less bit fields) 0 states end of bit field inputs
then user enters in values to be stored in a allocated array set to the bit field sizes.
say 1 2 3 4
how do i scan in several bit field values to be put in a struct like this?
struct{
unsigned int bit0:1;
unsigned int bit1:1;
unsigned int bit2:1;
unsigned int bit3:1;
}pack;
Then would this be correct when setting up the array to bit field size?
I'm using "bit field inputs" in place of however i would scan them in for now.
pack = (int *)malloc(sizeof(struct)*bit field inputs);
i believe i asked my original question wrong,
what im trying to do here is take say value 1 and put it in to a bit field width say 17
and keep repeating this for up to 5 values.
if the user inputs the bit field widths how would i take value one and store it in a field width of 17?
If you need dynamic bit field widths, you need to do your own bit-fiddling. For compatibility, you should do it anyway.
If not, you must read it into a standard type like int and then asign to the bitfield, so the compiler does your bitpacking for you. But beware, the standard gives few guarantees regarding the compilers choices in this.
Also, never cast the return value of malloc: Do I cast the result of malloc?.
It is up to the compiler to determine the order and padding for bitfields. There are no guarantees.
In general, you should only use bitfields "internally" in your program. Anytime you serialize bitfields, you may run into incompatibilities.
You would generally be better off by serializing into a structure with known size and alignment, then explicitly copying the values into your bitfield to use internally.
After re-reading your question again, I think you would be better off using bit-masking operations on contiguous bytes. This allows you to control the memory layout of your internal representation.
It sounds like you want to:
read and store the bit offsets from the input string
sum up how many bits the user wants to use from their input string
malloc some bytes of contiguous memory that is large enough to
contain the user bits
provide accessor functions to the bit position and length of data,
then use masking to set the bits.
Or if you don't care about size, just make everything an array of unsigned ints and let the compiler do the alignment for you.
This question already has answers here:
Closed 12 years ago.
Possible Duplicate:
Why isn’t sizeof for a struct equal to the sum of sizeof of each member?
I guess similar (duplicate) questions must have been asked on SO before. But I'm unable to find them. Basically I don't know what to search for. So asking it here.
Why doesn't size of struct is equals to sum of sizes of its individual member types?
I'm using visual C++ compiler.
For example, assuming 32-bit machine. {=> sizeof(int) == 4; sizeof(char) == 1; sizeof(short) == 2; }
struct {
int k;
char c;
} s;
The size expected is 4+1 = 5; but sizeof(s) gives 8. Here char is occupying 4 bytes instead of 1. I don't know the exact reason for this but my guess is compiler is doing so for efficiency purposes.
struct{
long long k;
int i;
} s;
expected size is 4+4 = 8 (on 32 bit machine) and 8+4=12 (on 64 bit machine). But strangely sizeof(s) give 16. Here both int & long long are occupying 8 bytes each.
What is this thing called?
What exactly is going on?
Why is compiler doing this?
Is there a way to tell compiler to stop doing this?
It is because the compiler uses padding to bring each element into word alignment that is specific to the architecture for which you are compiling.
It can be for one of several reasons but usually:
Because some CPU's simply cannot read a long or long long multi-byte value when it isn't on an address multiple of its own size.
Because CPU's that can read off-aligned data may do it much slower than aligned.
You can often force it off with a compiler-specific directive or pragma.
When you do this, the compiler will generate relatively inefficient code to access the off-aligned data using multiple read/write operations.
This is called padding; which involves adding some more bytes in order to align the structure on addresses that are divisible by some special number, usually 2, 4, or 8. A compiler can even place padding between members to align the fields themselves on those boundaries.
This is a performance optimization: access to aligned memory addresses is faster, and some architectures don't even support accessing unaligned addresses.
For VC++, you can use the pack pragma to control padding between fields. However, note that different compilers have different ways of handling this, so if, for example, you also want to support GCC, you'll have to use a different declaration for that.
The compiler can insert padding between members or at the end of the struct. Padding between members is typically done to keep the members aligned to maximize access speed. Padding at the end is done for roughly the same reason, in case you decide to create an array of the structs.
To prevent it from happening, you use something like #pragma pack(1).