For a specific need I am building a four byte integer out of four one byte chars, using nothing too special (on my little endian platform):
return (( v1 << 24) | (v2 << 16) | (v3 << 8) | v4);
I am aware that an integer stored in a big endian machine would look like AB BC CD DE instead of DE CD BC AB of little endianness, although would it affect the my operation completely in that I will be shifting incorrectly, or will it just cause a correct result that is stored in reverse and needs to be reversed?
I was wondering whether to create a second version of this function to do (yet unknown) bit manipulation for a big-endian machine, or possibly to use ntonl related function which I am unclear of how that would know if my number is in correct order or not.
What would be your suggestion to ensure compatibility, keeping in mind I do need to form integers in this manner?
As long as you are working at the value level, there will be absolutely no difference in the results you obtain regardless of whether your machine is little-endian or big-endian. I.e. as long as you are using language-level operators (like | and << in your example), you will get exactly the same arithmetical result from the above expression on any platform. The endianness of the machine is not detectable and not visible at this level.
The only situations when you need to care about endianness is when the data you are working with is examined at the object representation level, i.e. in situations when its raw memory representation is important. What you said above about "AB BC CD DE instead of DE CD BC AB" is specifically about the raw memory layout of the data. That's what functions like ntonl do: they convert one memory layout to another memory layout. So far you gave no indication that the actual raw memory layout is in any way important to you. Is it?
Again, if you only care about the value of the above expression, it is fully and totally endianness-independent. Basically, you are not supposed to care about endianness at all when you write C programs that don't attempt to access and examine the raw memory contents.
although would it affect the my operation completely in that I will be shifting incorrectly (?)
No.
The result will be the same regardless of the endian architecture. Bit shifting and twiddling are just like regular arithmetic operations. Is 2 + 2 the same on little endian and big endian architectures? Of course. 2 << 2 would be the same as well.
Little and big endian problems arise when you are dealing directly with the memory. You will run into problems when you do the following:
char bytes[] = {1, 0, 0, 0};
int n = *(int*)bytes;
On little endian machines, n will equal 0x00000001. On big endian machines, n will equal 0x01000000. This is when you will have to swap the bytes around.
[Rewritten for clarity]
ntohl (and ntohs, etc.) is used primarily for moving data from one machine to another. If you're simply manipulating data on one machine, then it's perfectly fine to do bit-shifting without any further ceremony -- bit-shifting (at least in C and C++) is defined in terms of multiplying/dividing by powers of 2, so it works the same whether the machine is big-endian or little-endian.
When/if you need to (at least potentially) move data from one machine to another, it's typically sensible to use htonl before you send it, and ntohl when you receive it. This may be entirely nops (in the case of BE to BE), two identical transformations that cancel each other out (LE to LE), or actually result in swapping bytes around (LE to BE or vice versa).
FWIW, I think a lot of what has been said here is correct. However, if the programmer has coded with endianness in mind, say using masks for bitwise inspection and manipulation, then cross-platform results could be unexpected.
You can determine 'endianness' at runtime as follows:
#define LITTLE_ENDIAN 0
#define BIG_ENDIAN 1
int endian() {
int i = 1;
char *p = (char *)&i;
if (p[0] == 1)
return LITTLE_ENDIAN;
else
return BIG_ENDIAN;
}
... and proceed accordingly.
I borrowed the code snippet from here: http://www.ibm.com/developerworks/aix/library/au-endianc/index.html?ca=drs- where there is also an excellent discussion of these issues.
hth -
Perry
Related
Is fwrite portable? I'm not really faced to the problem described below but I'd like to understand the fundamentals of C.
Lest assume we have two machines A8 (byte = 8bits) and B16 (byte = 16 bits).
Will the following code produce the same output on both machines ?
unsigned char[10] chars;
...
fwrite(chars,sizeof(unsigned char),10,mystream);
I guess A8 will produce 80 bits (10 octets) and B16 will produce 160 bits (20 octets).
Am I wrong?
This problem won't appear if only uintN_t types were used as their lengths in bits are independent of the size of the byte. But maybe uint8_t won't exist on B16.
What is the solution to this problem?
I guess building an array of uint32_t, putting my bytes in this array (with smart shifts and masks depending on machine's architecture) and writing this array will solve the problem. But this not really satisfactory.There is again an assumption that uint32_t exists on all platforms.The filling of this array will be very dependant on the current machine's architecture.
Thanks for any response.
fwrite() is a standard library function. So it must be portable for each C compiler.
That is it must be defined in C standard library of that compiler to support your machine.
So machine of 8bit, 16 bit, 32 bit give you same high level operation.
But if you want to design those library function then you have to consider machine architecture, memory organization of that machine.
As a C compiler user you should not bother about internal behavior.
I think you just want to use those C library function. So no difference in behavior of the function for different machine.
A byte is on almost every modern computer 8 bits. But there is an other reason fwrite isn't portable:
A file which was written on a Little Endian machine can't be readed by a big endian machine and other way.
In C, char is defined as "smallest addressable unit of the machine". That is, char is not necessarily 8 bits.
In most cases, it's safe enough to rely on a fact that char is 8 bits, and not to deal with some extreme cases.
To speak generally, you probably won't be able to write "half of a byte" to a file on a storage. Additionally, there will be issues with portability on hardware level between devices which are designed to work with different byte size machines. If you are dealing with other devices (such as telecom or stuff), you will have to implement bit streams.
In class I've been tasked with writing a C program that decompresses a text file and prints out the characters it contains. Each character in the file is represented by 2 bits (4 possible characters).
I've recently been informed that a byte is not necessarily 8 bits on all systems, and a char is not necessarily 1 byte. This then makes me wonder how on earth I'm supposed to know how many bits got loaded from a file when I loaded 1 byte. Also how am I supposed to keep the loaded data in memory when there are no data types that can guarantee a set amount of bits.
How do I work with bit data in C?
A byte is not necessarily 8 bits. That much is certainly true. A char, on the other hand, is defined to be a byte - C does not differentiate between the two things.
However, the systems you will write for will almost certainly have 8-bit bytes. Bytes of different sizes are essentially non-existant outside of really, really old systems, or certain embedded systems.
If you have to write your code to work for multiple platforms, and one or more of those have differently sized chars, then you write code specifically to handle that platform - using e.g. CHAR_BIT to determine how many bits each byte contains.
Given that this is for a class, assume 8-bit bytes, unless told otherwise. The point is not going to be extreme platform independence, the point is to teach you something about bit fiddling (or possibly bit fields, but that depends on what you've covered in class).
This then makes me wonder how on earth I'm supposed to know how many
bits got loaded from a file when I loaded 1 byte.
You'll be hard pressed to find a platform where a byte is not 8 bits. (though as noted above CHAR_BIT can be used to verify that). Also clarify the portability requirements with your instructor or state your assumptions.
Usually bits are extracted using shifts and bitwise operations, e.g. (x & 3) are the rightmost 2 bits of x. ((x>>2) & 3) are the next two bits. Pick the right data type for the platforms you are targettiing or as others say use something like uint8_t if available for your compiler.
Also see:
Type to use to represent a byte in ANSI (C89/90) C?
I would recommend not using bit fields. Also see here:
When is it worthwhile to use bit fields?
You can use bit fields in C. These indices explicitly let you specify the number of bits in each part of the field, if you are truly concerned about width. This page gives a discussion: http://msdn.microsoft.com/en-us/library/yszfawxh(v=vs.80).aspx
As an example, check out the ieee754.h for usage in the context of implementing IEEE754 floats
In c I am little worried with the concept of endianess. Suppose I declare an integer (2 bytes) on a little endian machine as
int a = 1;
and it is stored as:
Address value
1000 1
1001 0
On big endian it should be stored as vice-versa. Now if I do &a then I should get 1000 on on both machines.
If this is true then if I store int a=1 then I should get a=1 on little endian whereas 2^15 on big endian. Is this correct?
It should not matter to you how the data is represented as long as you don't transfer it between platforms (or access it through assembly code).
If you only use standard C - it's hidden from you and you shouldn't bother yourself with it. If you pass data around between unknown machines (for example you have a client and a server application that communicate through some network) - use hton and ntoh functions that convert from local to network and from network to local endianess, to avoid problems.
If you access memory directly, then the problem would not only be endian but also packing, so be careful with it.
In both little endian and big endian, the address of the "first" byte is returned. Here by "first" I mean 1000 in your example and not "first" in the sense of most-significant or least significant byte.
Now if I do &a then I should get 1000 on little endian and 1001 on big endian.
No; on both machines you will get the address 1000.
If this is true then if I store int a=1 then I should get a=1 on little endian whereas 2^15 on big endian. Is this correct?
No; because the preliminary condition is false. Also, if you assign 1 to a variable and then read the value back again, you get the result 1 back. Anything else would be a major cause of confusion. (There are occasions when this (reading back the stored value) is not guaranteed - for example, if multiple threads could be modifying the variable, or if it is an I/O register on an embedded system. However, you'd be aware of these if you needed to know about it.)
For the most part, you do not need to worry about endianness. There are a few places where you do. The socket networking functions are one such; writing binary data to a file which will be transferred between machines of different endianness is another. Pretty much all the rest of the time, you don't have to worry about it.
This is correct, but it's not only an issue in c, but any program that reads and writes binary data. If the data stays on a single computer, then it will be fine.
Also, if you are reading/writing from text files this won't be an issue.
I have a couple of questions that are all inter-related. Basically, in the algorithm I am implementing a word w is defined as four bytes, so it can be contained whole in a uint32_t.
However, during the operation of the algorithm I often need to access the various parts of the word. Now, I can do this in two ways:
uint32_t w = 0x11223344;
uint8_t a = (w & 0xff000000) >> 24;
uint8_t b = (w & 0x00ff0000) >> 16;
uint8_t b = (w & 0x0000ff00) >> 8;
uint8_t d = (w & 0x000000ff);
However, part of me thinks that isn't particularly efficient. I thought a better way would be to use union representation like so:
typedef union
{
struct
{
uint8_t d;
uint8_t c;
uint8_t b;
uint8_t a;
};
uint32_t n;
} word32;
Using this method I can assign word32 w = 0x11223344; then I can access the various
parts as I require (w.a=11 in little endian).
However, at this stage I come up against endianness issues, namely, in big endian systems my struct is defined incorrectly so I need to re-order the word prior to it being passed in.
This I can do without too much difficulty. My question is, then, is the first part (various bitwise ands and shifts) efficient compared to the implementation using a union? Is there any difference between the two generally? Which way should I go on a modern, x86_64 processor? Is endianness just a red herring here?
I could inspect the assembly output of course, but my knowledge of compilers is not brilliant. I would have thought a union would be more efficient as it would essentially convert to memory offsets, like so:
mov eax, [r9+8]
Would a compiler realise that is what happening in the bit-shift case above?
If it matters, I'm using C99, specifically my compiler is clang (llvm).
Thanks in advance.
If you need AES, why not use an existing implementation? This can be particularly beneficial on modern Intel processors with hardware support for AES.
The union trick can slow down things due to store-to-load-forwarding (STLF) failures. This may happen, depending on the processor model, if you write data to memory and read it back soon as a different data type (e.g. 32bit vs 8bit).
Such a thing is hard to tell without being able to inspect the real use of these operations in your code:
the shift version will probably do
better if you happen to have all your
variables in registers, anyhow, and
then you do intensive computations on
them. Usually compilers (clang including) are relatively clever in issuing instructions for partial words and stuff like that.
the union version would perhaps be
more efficient if you'd have to load
your bytes from memory most of the
time
In any case I would abstract the access operation into a macro, such that you can modify it easily whence you have a working code.
For my personal taste I would go for the shift version, since it is conceptually simpler, and only go for the union when I'd see that at the end the produced assembler doesn't look satisfactory.
I would guess using a union may be more efficient. Of course, the compiler may be able to optimize the shifts into byte loads since they are known during compilation -- in which case both schemes will yield identical code.
Another option (also byte order dependent) is to cast the word to a byte array and access the bytes directly. I.e., something like the following
uint8_t b = ((uint8_t*)w)[n]
I'm not sure you will see any difference on a real modern 32/64 bit processor, though.
EDIT: It seems like clang produces identical code in both cases.
Given that accessing bits using shift and masking is a common operation I'd expect compilers to be quite smart about it especially if you're using constant shift count and mask.
An option would be to use macros for bit set/get so that you can pick the best strategy at configure time if on a specific platform a compiler happens to be on the dumb side (and wisely chosen names for the macros can also make the code more clear and self explaining).
I'm working with embedded C for the first time. Although my C is rusty, I can read the code but I don't really have a grasp on why certain lines are the way the are. For example, I want to know if a variable is true or false and send it back to another application. Rather than setting the variable to 1 or 0, the original implementor chose 0xFF.
Is he trying to set it to an address space? or else why set a boolean variable to be 255?
0xFF sets all the bits in a char.
The original implementer probably decided that the standard 0 and 1 wasn't good enough and decided that if all bits off is false then all bits on is true.
That works because in C any value other than 0 is true.
Though this will set all bytes in a char, it will also work for any other variable type, since any one bit being set in a variable makes it true.
If you are in desperate need of memory, you might want to store 8 booleans in one byte (or 32 in a long, or whatever)
This can easily be done by using a flag mask:
// FLAGMASK = ..1<<n for n in 0..7...
FLAGMASK = 0x10; // e.g. n=4
flags &= ~FLAGMASK; // clear bit
flags |= FLAGMASK; // set bit
flags ^= FLAGMASK; // flip bit
flags = (flags & ~FLAGMASK) | (booleanFunction() & FLAGMASK); // clear, then maybe set
this only works when booleanFunction() returns 0 (all bits clear) or -1 (all bits set).
0xFF is the hex representation of ~0 (i.e. 11111111)
In, for example, VB and Access, -1 is used as True.
These young guys, what do they know?
In one of the original embedded languages - PL/M (-51 yes as in 8051, -85, -86, -286, -386) - there was no difference between logical operators (!, &&, || in C) and bitwise (~, &, |, ^). Instead PL/M has NOT, AND, OR and XOR taking care of both categories. Are we better off with two categories? I'm not so sure. I miss the logical ^^ operator (xor) in C, though. Still, I guess it would be possible to construct programs in C without having to involve the logical category.
In PL/M False is defined as 0. Booleans are usually represented in byte variables. True is defined as NOT False which will give you 0ffh (PL/M-ese for C's 0xff).
To see how the conversion of the status flag carry took place defore being stored in a byte (boolean wasn't available as a type) variable, PL/M could use the assembly instruction "sbb al,al" before storing. If carry was set al would contain 0ff, if it wasn't it would contain 0h. If the opposite value was required, PL/M would insert a "cmc" before the sbb or append a "not al" after (actually xor - one or the other).
So the 0xff for TRUE is a direct compatibility port from PL/M. Necessary? Probably not, unless you're unsure of your skills (in C) AND playing it super safe.
As I would have.
PL/M-80 (used for the 8080, 8085 and Z80) did not have support for integers or floats, and I suspect it was the same for PL/M-51. PL/M-86 (used for the 8086, 8088, 80188 and 80186) added integers, single precision floating point, segment:offset pointers and the standard memory models small, medium, compact and large. For those so inclined there were special directives to create do-it-yourself hybrid memory models. Microsoft's huge memory model was equivalent to intel's large. MS also sported tiny, small, compact, medium and large models.
Often in embedded systems there is one programmer who writes all the code and his/her idiosyncrasies are throughout the source. Many embedded programmers were HW engineers and had to get a system running as best they could. There was no requirement nor concept of "portability". Another consideration in embedded systems is the compiler is specific for the CPU HW. Refer to the ISA for this CPU and check all uses of the "boolean".
As others have said, it's setting all the bits to 1. And since this is embedded C, you might be storing this into a register where each bit is important for something, so you want to set them all to 1. I know I did similar when writing in assembler.
What's really important to know about this question is the type of "var". You say "boolean", but is that a C++/C99's bool, or is it (highly likely, being an embedded C app), something of a completely different type that's being used as a boolean?
Also adding 1 to 0xff sets it to 0( assuming unsigned char) and the checking might have been in a loop with an increment to break.
Here's a likely reason: 0xff is the binary complement of 0. It may be that on your embedded architecture, storing 0xff into a variable is more efficient than storing, say, 1 which might require extra instructions or a constant stored in memory.
Or perhaps the most efficient way to check the "truth value" of a register in your architecture is with a "check bit set" instruction. With 0xff as the TRUE value, it doesn't matter which bit gets checked... they're all set.
The above is just speculation, of course, without knowing what kind of embedded processor you're using. 8-bit, 16-bit, 32-bit? PIC, AVR, ARM, x86???
(As others have pointed out, any integer value other than zero is considered TRUE for the purposes of boolean expressions in C.)