So, I can understand that a word of 0x1234, when stored as little-endian, becomes 0x3412 in memory. I am also seeing that byte 0x12 as a bitfield a:4 and b:4 would be stored as 0x21. But what if I have something more complex? Data like 0x1700581001FFFFFF with the following struct ordering? I'm seeing the data stored as 0x7180051001FFFFFF which is making very little sense to me. It seems 'a' and 'b' got swapped but they remained at the beginning of the struct and g remained at the end along with other seemingly random swaps. Why? Also, I left the "LONGWORD" denotion because that is there in the code. I'm not sure how 4 bits can be a longword, but perhaps that has something to do with this craziness?
LONGWORD a: 4
LONGWORD b: 4
LONGWORD c: 4
LONGWORD d: 12
LONGWORD e: 8
LONGWORD f: 8
LONGWORD g: 24
In an "implementation-defined manner". Per 6.7.2.1 Structure and union specifiers, paragraph 11, of the C Standard:
An implementation may allocate any addressable storage unit large
enough to hold a bit-field. If enough space remains, a bit-field
that immediately follows another bit-field in a structure shall be
packed into adjacent bits of the same unit. If insufficient space
remains, whether a bit-field that does not fit is put into the next
unit or overlaps adjacent units is implementation-defined. The order
of allocation of bit-fields within a unit (high-order to low-order or
low-order to high-order) is implementation-defined. The alignment of
the addressable storage unit is unspecified.
To answer your question But what if I have something more complex? Data like 0x1700581001FFFFFF with the following struct ordering?
The proper answer in that case, if you want portable and reliable code, is to not use bit-fields. The fact that you have failed to provide enough information in your question for anyone to provide an answer as to how that data will be placed into the bit-fields you described should inform you what the problems are when using bit-fields.
For example, given your bit-fields
LONGWORD a: 4
LONGWORD b: 4
LONGWORD c: 4
LONGWORD d: 12
LONGWORD e: 8
LONGWORD f: 8
LONGWORD g: 24
If one assumes 16-bit int-type values are used for bit-fields, it would be perfectly proper to lay out the data thusly:
16-bit `int` with `c`,`b`,`a` - in that order
16-bit `int` with `d`
16-bit `int` with `f`,`e` - in that order
16-bit `int` with first 16 bits of `g`
16-bit `int` with last 8 bits of `g` - **after** 8 bits of padding.
And that's not even getting into endianness of the storage.
Questions like (and the point made in a comment about how to "designate, in order, the meaning of bits to data") inevitably boil down to: what are you trying to do with the data?
If you're declaring a data structure so that some C code can write to it, and other C code can read from it, you rarely if ever care about the byte order, or the bitfield order (or the padding, or the alignment, or any of that).
Where it gets tricky -- very tricky -- is when you try to take that data structure, as your C compiler laid it out in memory, and write it out to or read it in from the outside world. When you try to do that, you end up having to worry forever about type sizes, and byte order, and padding, and alignment, and the order in which bitfields are assigned.
In fact there are so many things to worry about, and nailing them all down is such a nuisance, that many people (myself included) recommend simply not trying to define data structures which can be directly read and written in this way, at all.
My memory is that compilers for big-endinan machines tend to lay out the bits in bitfields one way, and for little-endian the other way. But I can never remember which way is which. (And even if I thought I could remember, you shouldn't trust me.) If for some reason you care, you're going to have to do what I always do, which is to write some little test programs to construct some binary data and print it out in hex and figure out how it's done for the machine/compiler combination you're using today. (And of course you also have to decide what you're going to do about the possibility that your machine/compiler combination might change next week.)
Upon re-reading the documentation, I do not see any allowance for packing the bitfields in simply any order. There is indeed a specified ordering but it is implementation dependent in which way it is done. But it is still quite determinable. In short, from what I am seeing, it is packing up the bits by groups of 8 IN ORDER. The difference for our Little Endian compiler (or maybe some option somewhere) is that the concatenation of the bits puts the first-defined bits AFTER the next-defined bits (i.e. made the first defined less significant than the next-defined). For example:
a:3 = 111 (binary)
b:4 = 0000 (binary)
c:9 = 011111111 (binary)
Our Little Endian compiler (or, again, perhaps some other option) will take the 3 bits from 'a' and concatenate with b by adding 'a' to the end of 'b'. This, I believe, is opposite what Big Endian compilers would do which would put 'a' before the 'b'. So I'm speculating it's the endianness that does this, but ours would get 7 bits of 0000111 by making ba rather than ab. It then needs one more bit from c to create a full 8. It takes the least significant bit of 'c' which is a 1 and, again, the previous bits get tacked on to the end of that new bit. So we have 10000111. This byte, 0x87, is then stored to memory and it grabs another 8 bits. In this example the next 8 bits is 01111111 and so it stores that byte, 0x7F, after the 0x87. So in memory we now have 0x877F. Another method (likely Big Endian) would have ended up with 0xE0FF. The 0x877F which is now in memory, if interpreted as a word in Little Endian would be a value of 0x7F87 or, in binary, 0111111110000111. This happens to be the exact reverse of the data structure above concatenating 'cba'.
So let's do that same reverse ordering of the data I provided earlier:
(0x1700581001FFFFFF was meant to be parsed as below but I guess that might not have been obvious since it is a Big Endian construct I assumed)
LONGWORD a: 4 = 0x1
LONGWORD b: 4 = 0x7
LONGWORD c: 4 = 0x0
LONGWORD d: 12 = 0x058
LONGWORD e: 8 = 0x10
LONGWORD f: 8 = 0x01
LONGWORD g: 24 = 0xFFFFFF
With the Little Endian configuration we have, this could be interpreted as one giant structure with a value of 0xFFFFFF0110058071 by concatenating in the order gfedcba. If we store this back to memory in Little Endian format, we would get 0x7180051001FFFFFF which is the exact data I said we were seeing. Big Endian, in theory, would have done it in the order I assumed as obvious (0x1700581001FFFFFF) both as interpreted and stored.
Well, it makes sense to me. Hopefully it makes sense to someone else trying to figure out the same thing! I still don't get why they all say LONGWORD before them though...
Related
I was going through the text "The C Programming Language" by Kernighan and Ritchie. While discussing about bit-fields at the end of that section, the authors say:
"Fields are assigned left to right on some machines and right to left on others. This means that although fields are useful for maintaining internally-defined data structures, the question of which end comes first has to be carefully considered when picking apart externally-defined data; programs that depend on such things are not portable."
- The C Programming Language [2e] by Kernighan & Ritchie [Section 6.9, p.150]
Strictly I do not get the meaning of these lines. Can anyone please explain me with a possible diagram?
PS: Well I have taken a computer organization and architecture course. I know how computers deal with bits and bytes. In a computer system, the smallest unit of information is a single bit which can be either 0 or 1. 8 such bits form a byte. Memories are byte-addressable, which means that each byte in the memory has an address associated with it. But usually, the processors have word lengths as 2 bytes (very old systems),4 bytes, 8 bytes... This means in one memory cycle, the CPU can take up a word length number of bytes from the main memory and put it inside its registers. Now how these bytes are placed in registers depends on the endianness of the system.
But I do not get what the authors mean by "left to right" or "right to left". The words seem like they are related to the endianness but endianness depends on the CPU and C compilers have nothing to do with it... The question which comes to my mind is "left to right" of "what"? What object are the authors referring to?
When a structure contains bit-fields, the C implementation uses some storage unit to hold them (or multiple storage units if needed). The storage unit might be one eight-bit byte or it might be four bytes, or it might be other sizes—this is a determination made by each C implementation. The C standard only requires that it be addressable, which effectively means it has to be a whole number of bytes.
Once we have a storage unit, it is some number of bits. Say it is 32 bits, and number the bits from 31 to 0, where, if we consider the bits to represent a binary numeral, bit 0 represents 20, and bit 31 represents 231. Note that Kernighan and Ritchie are imprecise to use “left” and “right” here. There is no inherent left or right. We usually write numerals with the most significant digits on the left, so we might consider bit 31 to be the leftmost and bit 0 to be the rightmost.
Now we have a storage unit with some number of bits and some labeling for those bits (31 to 0 or left to right). Say you want to put two bit-fields in them, say fields of width 7 and 5.
Which 7 of the bits from bit 31 to bit 0 are used for the first field? Which 5 of the bits are used for the second field?
We could use bits 31-25 for the first field and bits 24-20 for the second field. Or we could use bits 6-0 for the first field and bits 11-7 for the second field.
In theory, we could also use bits 27-21 for the first field and bits 15-11 for the second field. However, the C standard does say that “If enough space remains, a bit-field that immediately follows another bit-field in a structure shall be packed into adjacent bits of the same unit” (C 2018 6.7.2.1 11). “Adjacent” is not formally defined, but we can assume it means consecutively numbered bits. So, if the C implementation puts the first field in bits 31-25, it is required to put the second field in bits 24-20. Conversely, it it puts the first field in bits 6-0, it must put the second field in 11-7.
Thus, the C standard requires an implementation to arrange successive bit-fields in a storage unit from left-to-right or from right-to-left, but it does not say which.
(I do not see anything in the standard that says the first field must start at one end of the storage unit or the other, rather than somewhere in the middle. That would lead to wasting some bits.)
When you write:
struct {
unsigned int version: 4;
unsigned int length: 4;
unsigned char dcsn;
you end up with a big headache you weren't expecting because your code is non-portable.
When you set version to 4 and length to 5, some systems may set the first byte of the structure to 0x45 and other systems may set the first byte of the structure to 0x54.
When I went to college this thing was #ifdef'd as follows (incorrect):
struct {
#if BIG_ENDIAN
unsigned int version: 4;
unsigned int length: 4;
#else
unsigned int length: 4;
unsigned int version: 4;
#endif
unsigned char dcsn;
but this is still rolling the dice as there's no rule that the order of the bits in the bytes in a bitfield corresponds to the order of bytes in the word in the machine. I would not be surprised that when you cross-compile the bit order in the struct comes from the host machine's rules while the bit order of integers comes from the target machine's rules (as it must). In theory the code could be corrected by having a separate #ifdef for BIG_ENDIAN_BITFIELD but I've never seen it done.
Here is some demonstration code. The only goal is to demonstrate what you are asking about. Clean coding etc. is neglected.
#include <stdio.h>
#include <stdint.h>
union
{
uint32_t Everything;
struct
{
uint32_t FirstMentionedBit : 1;
uint32_t FewOTherBits :30;
uint32_t LastMentionedBit : 1;
} bitfield;
} Demonstration;
int main()
{
Demonstration.Everything =0;
Demonstration.bitfield.LastMentionedBit=1;
printf("%x\n", Demonstration.Everything);
Demonstration.Everything =0;
Demonstration.bitfield.FirstMentionedBit=1;
printf("%x\n", Demonstration.Everything);
return 0;
}
If you use this here https://www.tutorialspoint.com/compile_c_online.php
the output is
80000000
1
But in other environments it might easily be
1
80000000
This is because compilers are free to consider the first mentioned bit the MSB or the LSB and correspondingly the last mentioned bit to be the LSB or MSB.
And that is what your quote describes.
Below is an excerpt from the red dragon book.
Example 7.3. Figure 7.9 is a simplification of the data layout used by C compilers for two machines that we call Machine 1 and Machine 2.
Machine 1 : The memory of Machine 1 is organized into bytes consisting of 8 bits each. Even though every byte has an address, the instruction set favors short integers being positioned at bytes whose addresses are even, and integers being positioned at addresses that are divisible by 4. The compiler places short integers at even addresses, even if it has to skip a byte as padding in the process. Thus, four bytes, consisting of 32 bits, may be allocated for a character followed by a short integer.
Machine 2: each word consists of 64 bits, and 24 bits are allowed for the address of a word. There are 64 possibilities for the individual bits inside a word, so 6 additional bits are needed to distinguish between them. By design, a pointer to a character on Machine 2 takes 30 bits — 24 to find the word and 6 for the position of the character inside the word. The strong word orientation of the instruction set of Machine 2 has led the compiler to allocate a complete word at a time, even when fewer bits would suffice to represent all possible values of that type; e.g., only 8 bits are needed to represent a character. Hence, under alignment, Fig. 7.9 shows 64 bits for each type. Within each word, the bits for each basic type are in specified positions. Two words consisting of 128 bits would be allocated for a character followed by a short integer, with the character using only 8 of the bits in the first word and the short integer using only 24 of the bits in the second word. □
I found about the concept of alignment here ,here and here. What I could understand from them is as follows: In word addressable CPUs (where size is more than a byte), there certain paddings are introduced in the data objects, such that CPU can efficiently retrieve data from the memory with minimum no. of memory cycles.
Now the Machine 1 here is actually a byte address one. And the conditions in the Machine 1 specification are probably more difficult than a simple word addressable machine having word size of say 4 bytes. In such a 64 bit machine, we need to make sure that our data items are just word aligned ,no more difficulty. But how to find the alignment in systems like Machine 1 (as given in the table above) where the simple concept of word alignment does not work, because it is byte addressable and has much more difficult specifications.
Moreover I find it quite weird that in the row for double the size of the type is more than what is given in the alignment field. Shouldn't alignment(in bits) ≥ size (in bits) ? Because alignment refers to the memory actually allocated for the data object (?).
"each word consists of 64 bits, and 24 bits are allowed for the address of a word. There are 64 possibilities for the individual bits inside a word, so 6 additional bits are needed to distinguish between them. By design, a pointer to a character on Machine 2 takes 30 bits — 24 to find the word and 6 for the position of the character inside the word." - Moreover how should this statement about the concept of the pointers, based on alignment is to be visualized (2^6 = 64, it is fine but how is this 6 bits correlating with the alignment concept)
First of all, the machine 1 is not special at all - it is exactly like a x86-32 or 32-bit ARM.
Moreover I find it quite weird that in the row for double the size of the type is more than what is given in the alignment field. Shouldn't alignment(in bits) ≥ size (in bits) ? Because alignment refers to the memory actually allocated for the data object (?).
No, this isn't true. Alignment means that the address of the lowest addressable byte in the object must be divisible by the given number of bytes.
Additionally, with C, it is also true that within arrays sizeof (ElementType) will need to be greater than or equal to the alignment of each member and sizeof (ElementType) be divisible by alignment, thus the footnote a. Therefore on the latter computer:
struct { char a, b; }
might have sizeof 16 because the characters are in distinct addressable words, whereas
struct { char a[2]; }
could be squeezed into 8 bytes.
how should this statement about the concept of the pointers, based on alignment is to be visualized (2^6 = 64, it is fine but how is this 6 bits correlating with the alignment concept)
As for the character pointers, the 6 bits is bogus. 3 bits are needed to choose one of the 8 bytes within the 8-byte words, so this is an error in the book. An ordinary byte would select just a word with 24 bits, and a character (a byte) pointer would select the word with 24 bits, and one of the 8-bit bytes inside the word with 3 bits.
My question has two parts.
First, as a newbie to this address space, I would like to know what is the meaning of memory alignment of an address. I Googled about it but wanted to ask this question here as well since I found answers here very useful.
The second part of my question is related to alignment and programming: how do I find if an address is 4 byte aligned or not ?
Somewhere I read:
if(address & 0x3) // for 32 bit register
But I don't really know how this checks for a 4 byte alignment.
Could anyone explain it in detail?
Edit: It would be great If someone can draw pictorial view on this subject.
Thanks
Sequential addresses refer to sequential bytes in memory.
An address that is "4-byte aligned" is a multiple of 4 bytes. In other words, the binary representation of the address ends in two zeros (00), since in binary, it's a multiple of the binary value of 4 (100b). The test for 4-byte aligned address is, therefore:
if ( (address & 0x3) == 0 )
{
// The address is 4-byte aligned here
}
or simply
if ( !(address & 0x3) )
{
// The address is 4-byte aligned here
}
The 0x3 is binary 11, or a mask of the lowest two bits of the address.
Alignment is important since some CPU operations are faster if the address of a data item is aligned. This is because CPUs are 32-bit or 64-bit word based. Small amounts of data (say 4 bytes, for example) fit nicely in a 32-bit word if it is 4-byte aligned. If it is not aligned, it can cross a 32-bit boundary and require additional memory fetches. Modern CPUs have other optimizations as well that improve performance for address aligned data.
Here's a sample article regarding the topic of alignment and speed.
Here are some some nice diagrams of alignment.
I am trying to write to a file binary data that does not fit in 8 bits. From what I understand you can write binary data of any length if you can group it in a predefined length of 8, 16, 32,64.
Is there a way to write just 9 bits to a file? Or two values of 9 bits?
I have one value in the range -+32768 and 3 values in the range +-256. What would be the way to save most space?
Thank you
No, I don't think there's any way using C's file I/O API:s to express storing less than 1 char of data, which will typically be 8 bits.
If you're on a 9-bit system, where CHAR_BIT really is 9, then it will be trivial.
If what you're really asking is "how can I store a number that has a limited range using the precise number of bits needed", inside a possibly larger file, then that's of course very possible.
This is often called bitstreaming and is a good way to optimize the space used for some information. Encoding/decoding bitstream formats requires you to keep track of how many bits you have "consumed" of the current input/output byte in the actual file. It's a bit complicated but not very hard.
Basically, you'll need:
A byte stream s, i.e. something you can put bytes into, such as a FILE *.
A bit index i, i.e. an unsigned value that keeps track of how many bits you've emitted.
A current byte x, into which bits can be put, each time incrementing i. When i reaches CHAR_BIT, write it to s and reset i to zero.
You cannot store values in the range –256 to +256 in nine bits either. That is 513 values, and nine bits can only distinguish 512 values.
If your actual ranges are –32768 to +32767 and –256 to +255, then you can use bit-fields to pack them into a single structure:
struct MyStruct
{
int a : 16;
int b : 9;
int c : 9;
int d : 9;
};
Objects such as this will still be rounded up to a whole number of bytes, so the above will have six bytes on typical systems, since it uses 43 bits total, and the next whole number of eight-bit bytes has 48 bits.
You can either accept this padding of 43 bits to 48 or use more complicated code to concatenate bits further before writing to a file. This requires additional code to assemble bits into sequences of bytes. It is rarely worth the effort, since storage space is currently cheap.
You can apply the principle of base64 (just enlarging your base, not making it smaller).
Every value will be written to two bytes and and combined with the last/next byte by shift and or operations.
I hope this very abstract description helps you.
Lets say I have 4Byte integer and I want to cast it to 2Byte short integer. Am I right that in both (little and big endian) short integer will consist of 2 least significant bytes of this 4Byte integer?
Second question:
What will be the result of such code in little endian and big endian processor?
int i = some_number;
short s = *(short*)&i;
IMHO in big endian processor 2 most significant bytes would be copied, and in little endian 2 least significant bytes would be copied.
Am I right that in both short integer will consist of 2 least significant bytes of this 4Byte integer?
Yes, by definition.
The difference between bigE and littleE is whether the least significant byte is at the lowest address or not. On a little endian processor, the lowest addresses are the least significant bits, x86 does it this way.
These give the same result on little E.
short s = (short)i;
short s = *(short*)&i;
On a big endian processor, the highest addresses are the least significant bits, 68000 and Power PC do it this way (actually Power PC can be both, but PPC machines from Apple use bigE)
These give the same result on big E.
short s = (short)i;
short s = ((short*)&i)[1]; // (assuming i is 4 byte int)
So, as you can see, little endian allows you to get at the least significant bits of an operand without knowning how big it is. little E has advantages for preserving backward compatibility.
So what's the advantage of big endian? It creates hex dumps that are easier to read.
Really, the engineers at Motorola thought that easing the burden of reading hex dumps was more important than backward compatibility. The engineers at Intel believed the opposite.
Yes. When you convert values, you don't have to worry about endianness.
Yes. When you convert pointers, you do.
First of all, you may already know it but let me mention that the size of int is not guaranteed to be 4 bytes and that of short, 2 bytes across all platforms.
If in your first question you mean something like this:
int i = ...;
short s = (short)i;
then yes, s will contain the lower byte(s) of i.
I think the answer to your second question is also yes; at the byte level the endianness of the system does come into play.
You should be aware that your second example
int i = some_number;
short s = *(short*)&i;
is not valid C code as it violates strict aliasing rules. It is likely to fail under some optimization levels and/or compilers.
Use unions for that:
union {
int i;
short s;
} my_union;
my_union.i = some_number;
printf("%d\n",my_union.s);
Also, as others noted, you can't assume that your ints will be 4 bytes. Better use int32_t and int16_t when you need specific sizes.
If you really want to convert an int to a short, then just do that:
short int_to_short(int n) {
if (n < SHRT_MIN) return SHRT_MIN;
if (n > SHRT_MAX) return SHRT_MAX;
return (short)n;
}
You don't have to even worry about endian, the language handles that for you. If you are sure n is within the range of a short, then you can skip the check, too.