I have a code which is compiled into a library (dll, static library and so). I want the user of this library to use some struct to pass some data as parameters for the library function. I thought about declaring the struct in the API header file.
Is it safe to do so, considering compilation with different compilers, with respect to structure alignment or other things I didn't think about?
Will it require the usage of the same compiler (and flags) for both the library and its user?
Few notes:
I considered giving the user a pointer and set all the struct via functions in the library, but this will make the API really not comfortable to use.
This question is about C, although it would be nice to know if there's a difference in c++.
If it's a regular/static library, the library and application should be compiled using the same compiler. There're a few reasons for this that I can think of:
Different compilers (as in different brands or compilers for different platforms) normally don't understand each other's object and library formats.
You don't want to compile different parts of the same program using different types (e.g. signed vs unsigned char), type sizes (e.g. long = 32 vs 64 bits), alignment and packing and probably some other things, all of which are allowed by the C standard to vary. Mixing and matching those things is usually a bad thing.
You may, however, often use slightly different versions of the same compiler to compile the library and the application using it. Usually, it's OK. Sometimes there're changes that break the code, though.
You may implement some "initialization" function in that header file (declared as static inline) that would ensure that types, type sizes, alignment and packing are the same as expected by the compiled library. The application using this library would have to call this function prior to using any other part of the library. If things aren't the same as expected, the function must fail and cause program termination, possibly with some good textual description of the failure. This won't solve completely the problem of having somewhat incompatible compilers, but it can prevent silent and mysterious malfunctions. Some things can be checked with the preprocessor's #if and #ifdef directives and cause compilation errors with #error.
In addition, structure packing problems can be relieved by inserting explicit padding bytes into structure declarations and forcing tight packing (by e.g. using #pragma pack, which is supported by many compilers). That way if type sizes are the same, it won't matter what the default packing is.
You can apply the same to DLLs as well, but you should really expect that the calling application has been compiled with a different compiler and not depend on the compilers being the same.
All Windows APIs throw structs around like crazy so obviously this is something that is done every day and it works. Of course it doesn't mean that your concerns are not valid :)
I would suggest making your structure's fields have explicit width types (int32_t etc) and maybe specify explicitly that that the packing in a way which would break on any compiler but yours, i.e.
#if defined(_MSC_VER)
#pragma pack(0)
#elif defined ... handle gcc
#else
FAIL // fail compilation on unsupported platform
#endif
Related
There are some architectures which have multiple address spaces, notable examples are true Harvard ones, but for example OpenCL also has this property.
C compilers may provide some solutions to this, one of these is Named Address Spaces, supporting special pointer qualifiers to indicate the pointer's address space, but other solutions might also be present.
For GCC, the corresponding documentation is here: https://gcc.gnu.org/onlinedocs/gcc-4.7.0/gcc/Named-Address-Spaces.html
For IAR targeting the AVR, the corresponding documentation is here: https://www.iar.com/support/tech-notes/compiler/strings-with-iccavr-2.x/ (note that this is earlier than GCC's support, which GCC likely adapted for the 8 bit AVR target).
For SDCC (Small Device C compiler): http://sdcc.sourceforge.net/doc/sdccman.pdf , starts on Page 36. Covers microcontrollers like the 8051, Z80 and 68HC08.
Some information for OpenCL: https://www.khronos.org/registry/OpenCL/sdk/1.1/docs/man/xhtml/local.html and https://software.intel.com/en-us/articles/the-generic-address-space-in-opencl-20
I didn't know about them, and on the architecture I am using (8 bit AVR), there is another solution to deal with the problem, specialized macros (pgmspace.h) to work with data in the ROM. But there are no type checks on these, and they (in my opinion) make code ugly, so it would seem to me that using Named Address Spaces are a superior, and possibly even more portable way to deal with the problem (portable in that one could easily port such a software to a target having a single address space by providing empty definitions for the address space qualifiers).
However in a previous question in which I learned from their availability, solutions suggesting the use of Named Address Spaces got severely downvoted, here: How to make two otherwise identical pointer types incompatible
Downvoters didn't provide any explanation, and I neither found any myself, for me it seems like Named Address Spaces are a good and perfectly functional way of dealing with the problem.
Could anyone provide an explanation? Why Named Address Spaces probably shouldn't be used? (favoring whatever other method available on the target having multiple distinct address spaces)
Another approach is to steal a technique used in things like the Linux kernel and tools like smatch.
Linux has defines like
#define __user
which mean the code can say things like int foo(const __user char *p). The compiler ignores the __user but tools like smatch are then used to make sure that pointers don't accidentally wander between namespaces.
The problem with these is obvious: they only work on the gcc compiler.
And in the embedded systems branch there are lots of different compilers, each offering its own unique, non-portable way to do this. Sometimes that is fine (most embedded projects never get ported to different compilers) but from a generic point-of-view, it is not.
(The very same issue also exists with extended addresses - if you would for example use a 8 or 16 bit MCU with more than 64kib addressable memory. Compilers then use various non-standard extensions such as near and far.)
One solution to these problems is to make a "wrapper" around the compiler-specific behavior, by making a hardware abstraction layer (HAL) where you specify that the type used for storing data in flash is flash_byte_t or some such, then from your HAL include a compiler-specific header-file containing the actual typedef, such as typedef const __flash uint8_t flash_byte_t;. For example, the application includes "compiler.h" and this one in turn includes "gcc.h". That way you only need to re-write one small header file when you switch compiler.
Also as it turns out, C allows const flash_byte_t just fine even though this was already typedef'd as const. There's a special odd rule in C saying that you can add the same qualifier as many times in a declaration as you like. So const const int x is equivalent to const int x. This means that if the user would put on extra const-qualification, that's fine.
Note that it's mostly AVR being a special exception here, because of its weird Harvard model.
Otherwise, there's an industry de facto standard convention used by most compilers: all const qualified variables with static storage duration should be allocated in flash. Of course the C standard makes no guarantees of this (it is out of scope of the standard), but most embedded compilers behave like that.
The memory layout of a struct is up to the compiler. So what happens when some code compiled by one compiler uses a struct generated by code compiled by another compiler?
For example, say I have a header file that declares a struct somestruct, and a function that returns the struct. One source file defines that function and is compiled by compiler A. Another source file uses than function and is compiled by compiler B and links against the binary of the other source file.
If the two compilers create two different layouts for somestruct, then what's the layout of the variable returned by the function? Does it defer to one compiler's layout, or will there be a memory bug when the second source file tries to access elements of the struct returned by the first source file? Is it an error at compile time or link time?
The function will return a structure as specified by the ABI of the compiler of the function. The callee compiler, will just treat the function as if it conforms to the ABI of itself.
Assuming the two compilers use a similar ABI, in most cases, no errors will be reported during compile-time or link time or even during runtime. For some compatible compilers like Clang, GCC, and Intel C Compiler on OS X and Linux, no errors should result (if there are errors then it's a bug of the compiler). However in real world it is usually difficult to find fully compatible compilers (in most cases their ABIs are similar but not exactly the same; such ABI errors will be even harder to track down because your app would appear normal and crashes under some really weird circumstances are encountered during runtime).
Just as Basile said, name mangling for C++ poses an additional difference in ABI, but such differences are more easily caught during compile time as the linker literally can't find the symbol of the function, rather than finding a function that is not compatible.
Also, passing structures is another headache in terms of ABI because there are multiple structure-packing ABIs, sometimes even different in "compatible" compilers like GCC/MinGW and MSVC. (See also the -m[no-]ms-bitfields option in GCC, which forces GCC to use the MSVC ABI for structures.) I have also seen some cases where passing structures by pointer is more reliable than passing structures by value.
The layout of data (e.g. structures etc...), and the call protocol (how are call done at the processor level) are defined in a (processor and operating system specific) document called Application Binary Interface. If both compilers are following the same ABI (for the same processor and the same operating system) their generated code should be interoperable.
See e.g. the wikipage for x86 calling conventions and the x86-64 ABI specification.
Name mangling, notably for C++, might also be an issue.
Read also Levine's book on Linkers and Loaders
Assuming the following header file corresponding to, for example, a shared library. The exported function takes a pointer to a custom structure defined in this header:
// lib.h
typedef struct {
char c;
double d;
int i;
} A;
DLL_EXPORT void f(A* p);
If the shared library is built using one compiler and then is used from C code built with another compiler it might not work because of a different memory alignment, as Memory alignment in C-structs suggests. So, is there a way to make my structure definition portable across different compilers on the same platform?
I am interested specifically in Windows platform (apparently it does not have a well-defined ABI), though would be curious to learn about other platforms as well.
TL;DR in practice you should be fine.
The C standard does not define this but a platform ABI generally does. That is, for a given CPU architecture and operating system, there can be a definition for how C maps to assembly that allows different compilers to interoperate.
Struct alignment isn't the only thing that a platform ABI has to define, you also have function calling conventions and stuff like that.
C++ makes it even more complex and the ABI has to specify vtables, exceptions, name mangling, etc.
On Windows I think there are multiple C++ ABIs depending on compiler but C is mostly compatible across compilers. I could be wrong, not a Windows expert.
Some links:
what is an ABI? http://gcc.gnu.org/ml/libstdc++/2001-11/msg00063.html
things an ABI has to define C++ ABI issues list
example C++ ABI spec http://sourcery.mentor.com/public/cxx-abi/abi.html
how the ABI evolved on Solaris http://developers.sun.com/solaris/articles/CC_abi/CC_abi_content.html
Anyway the bottom line is that you're looking for your guarantee in the platform/compiler ABI spec, not the C standard.
The only way to know for sure is to consult the documentation of the compilers in question. However, it is usually the case that C struct layout (except, as you say, for bitfields) is defined by an ABI description for the environment you're using, and C compilers will tend to follow the native ABI.
Not only that it is not guarantied, but even if you use the same compiler there might be differences due to different compiler switches used in the build, or if you use different versions of the same compiler and same switches (happened in an embedded compiler I worked on).
You need to make make sure the structs are represented exactly the same, use switches, #pragmas, whatever the compilers gives you.
My advice - to stay way from this altogether. Pass your arguments in the function, not wrapped within a struct.
And even in this simple form, if you deal with two compilers, it's not trivial. You need to make sure that an int takes the same number of bytes, for example. Also calling conevntion - arguments order - from left to right or from right to left - can differ between compiler.
The C standard does not allow certain optimisations of structures: for example, rearrangement of fields, merging fields, discarding fields that are never read from, hoisting fields out of the structure if they can be turned into auto variables, etc. This is needed for various reasons, including consistent structure layouts across compilation units and allowing cast-compatible structures.
Do any modern compilers (e.g. gcc, clang, Visual C) support extensions that allow me to tell it that it is okay to do these optimisations?
Naturally, they'd only make sense for definitions that were local to a single compilation unit, so that the compiler could see all possible uses of the structure; and certain things (like the aforesaid cast-compatible structure definitions) would become unusable. But for certain tasks this could be a very valuable optimisation.
I do know that gcc used to have a -fipa-struct-reorg option to allow precisely this, but it never worked very well and bit rotted, and was eventually taken out. But I don't know if it's been replaced by anything. And I haven't been able to find anything in clang, which surprises me because I would think that this is precisely the kind of optimisation that clang would be all over...
No. There is no reason for such a thing to be supplied.
You can't do it where the structure's address is taken and sent anywhere, as it might be aliased anyway. That pretty much rules out anything outside of a single function.
If you can go through and do the analysis required to flag structure members that "this can be optimised away if not used" (beware funky offset calculating macros) then you can see for yourself if it is needed or not, and take it out yourself.
If unsure, just comment it out and see if you get a compile error.
I've worked on a number of different embedded systems. They have all used typedefs (or #defines) for types such as UINT32.
This is a good technique as it drives home the size of the type to the programmer and makes you more conscious of chances for overflow etc.
But on some systems you know that the compiler and processor won't change for the life of the project.
So what should influence your decision to create and enforce project-specific types?
EDIT
I think I managed to lose the gist of my question, and maybe it's really two.
With embedded programming you may need types of specific size for interfaces and also to cope with restricted resources such as RAM. This can't be avoided, but you can choose to use the basic types from the compiler.
For everything else the types have less importance.
You need to be careful not to cause overflow and may need to watch out for register and stack usage. Which may lead you to UINT16, UCHAR.
Using types such as UCHAR can add compiler 'fluff' however. Because registers are typically larger, some compilers may add code to force the result into the type.
i++;
can become
ADD REG,1
AND REG, 0xFF
which is unecessary.
So I think my question should have been :-
given the constraints of embedded software what is the best policy to set for a project which will have many people working on it - not all of whom will be of the same level of experience.
I use type abstraction very rarely. Here are my arguments, sorted in increasing order of subjectivity:
Local variables are different from struct members and arrays in the sense that you want them to fit in a register. On a 32b/64b target, a local int16_t can make code slower compared to a local int since the compiler will have to add operations to /force/ overflow according to the semantics of int16_t. While C99 defines an intfast_t typedef, AFAIK a plain int will fit in a register just as well, and it sure is a shorter name.
Organizations which like these typedefs almost invariably end up with several of them (INT32, int32_t, INT32_T, ad infinitum). Organizations using built-in types are thus better off, in a way, having just one set of names. I wish people used the typedefs from stdint.h or windows.h or anything existing; and when a target doesn't have that .h file, how hard is it to add one?
The typedefs can theoretically aid portability, but I, for one, never gained a thing from them. Is there a useful system you can port from a 32b target to a 16b one? Is there a 16b system that isn't trivial to port to a 32b target? Moreover, if most vars are ints, you'll actually gain something from the 32 bits on the new target, but if they are int16_t, you won't. And the places which are hard to port tend to require manual inspection anyway; before you try a port, you don't know where they are. Now, if someone thinks it's so easy to port things if you have typedefs all over the place - when time comes to port, which happens to few systems, write a script converting all names in the code base. This should work according to the "no manual inspection required" logic, and it postpones the effort to the point in time where it actually gives benefit.
Now if portability may be a theoretical benefit of the typedefs, readability sure goes down the drain. Just look at stdint.h: {int,uint}{max,fast,least}{8,16,32,64}_t. Lots of types. A program has lots of variables; is it really that easy to understand which need to be int_fast16_t and which need to be uint_least32_t? How many times are we silently converting between them, making them entirely pointless? (I particularly like BOOL/Bool/eBool/boolean/bool/int conversions. Every program written by an orderly organization mandating typedefs is littered with that).
Of course in C++ we could make the type system more strict, by wrapping numbers in template class instantiations with overloaded operators and stuff. This means that you'll now get error messages of the form "class Number<int,Least,32> has no operator+ overload for argument of type class Number<unsigned long long,Fast,64>, candidates are..." I don't call this "readability", either. Your chances of implementing these wrapper classes correctly are microscopic, and most of the time you'll wait for the innumerable template instantiations to compile.
The C99 standard has a number of standard sized-integer types. If you can use a compiler that supports C99 (gcc does), you'll find these in <stdint.h> and you can just use them in your projects.
Also, it can be especially important in embedded projects to use types as a sort of "safety net" for things like unit conversions. If you can use C++, I understand that there are some "unit" libraries out there that let you work in physical units that are defined by the C++ type system (via templates) that are compiled as operations on the underlying scalar types. For example, these libraries won't let you add a distance_t to a mass_t because the units don't line up; you'll actually get a compiler error.
Even if you can't work in C++ or another language that lets you write code that way, you can at least use the C type system to help you catch errors like that by eye. (That was actually the original intent of Simonyi's Hungarian notation.) Just because the compiler won't yell at you for adding a meter_t to a gram_t doesn't mean you shouldn't use types like that. Code reviews will be much more productive at discovering unit errors then.
My opinion is if you are depending on a minimum/maximum/specific size don't just assume that (say) an unsigned int is 32 bytes - use uint32_t instead (assuming your compiler supports C99).
I like using stdint.h types for defining system APIs specifically because they explicitly say how large items are. Back in the old days of Palm OS, the system APIs were defined using a bunch of wishy-washy types like "Word" and "SWord" that were inherited from very classic Mac OS. They did a cleanup to instead say Int16 and it made the API easier for newcomers to understand, especially with the weird 16-bit pointer issues on that system. When they were designing Palm OS Cobalt, they changed those names again to match stdint.h's names, making it even more clear and reducing the amount of typedefs they had to manage.
I believe that MISRA standards suggest (require?) the use of typedefs.
From a personal perspective, using typedefs leaves no confusion as to the size (in bits / bytes) of certain types. I have seen lead developers attempt both ways of developing by using standard types e.g. int and using custom types e.g. UINT32.
If the code isn't portable there is little real benefit in using typedefs, however , if like me then you work on both types of software (portable and fixed environment) then it can be useful to keep a standard and use the cutomised types. At the very least like you say, the programmer is then very much aware of how much memory they are using. Another factor to consider is how 'sure' are you that the code will not be ported to another environment? Ive seen processor specific code have to be translated as a hardware engieer has suddenly had to change a board, this is not a nice situation to be in but due to the custom typedefs it could have been a lot worse!
Consistency, convenience and readability. "UINT32" is much more readable and writeable than "unsigned long long", which is the equivalent for some systems.
Also, the compiler and processor may be fixed for the life of a project, but the code from that project may find new life in another project. In this case, having consistent data types is very convenient.
If your embedded systems is somehow a safety critical system (or similar), it's strongly advised (if not required) to use typedefs over plain types.
As TK. has said before, MISRA-C has an (advisory) rule to do so:
Rule 6.3 (advisory): typedefs that indicate size and signedness should be used in place of the basic numerical types.
(from MISRA-C 2004; it's Rule #13 (adv) of MISRA-C 1998)
Same also applies to C++ in this area; eg. JSF C++ coding standards:
AV Rule 209 A UniversalTypes file will be created to define all sta
ndard types for developers to use. The types include: [uint16, int16, uint32_t etc.]
Using <stdint.h> makes your code more portable for unit testing on a pc.
It can bite you pretty hard when you have tests for everything but it still breaks on your target system because an int is suddenly only 16 bit long.
Maybe I'm weird, but I use ub, ui, ul, sb, si, and sl for my integer types. Perhaps the "i" for 16 bits seems a bit dated, but I like the look of ui/si better than uw/sw.