Assuming the following header file corresponding to, for example, a shared library. The exported function takes a pointer to a custom structure defined in this header:
// lib.h
typedef struct {
char c;
double d;
int i;
} A;
DLL_EXPORT void f(A* p);
If the shared library is built using one compiler and then is used from C code built with another compiler it might not work because of a different memory alignment, as Memory alignment in C-structs suggests. So, is there a way to make my structure definition portable across different compilers on the same platform?
I am interested specifically in Windows platform (apparently it does not have a well-defined ABI), though would be curious to learn about other platforms as well.
TL;DR in practice you should be fine.
The C standard does not define this but a platform ABI generally does. That is, for a given CPU architecture and operating system, there can be a definition for how C maps to assembly that allows different compilers to interoperate.
Struct alignment isn't the only thing that a platform ABI has to define, you also have function calling conventions and stuff like that.
C++ makes it even more complex and the ABI has to specify vtables, exceptions, name mangling, etc.
On Windows I think there are multiple C++ ABIs depending on compiler but C is mostly compatible across compilers. I could be wrong, not a Windows expert.
Some links:
what is an ABI? http://gcc.gnu.org/ml/libstdc++/2001-11/msg00063.html
things an ABI has to define C++ ABI issues list
example C++ ABI spec http://sourcery.mentor.com/public/cxx-abi/abi.html
how the ABI evolved on Solaris http://developers.sun.com/solaris/articles/CC_abi/CC_abi_content.html
Anyway the bottom line is that you're looking for your guarantee in the platform/compiler ABI spec, not the C standard.
The only way to know for sure is to consult the documentation of the compilers in question. However, it is usually the case that C struct layout (except, as you say, for bitfields) is defined by an ABI description for the environment you're using, and C compilers will tend to follow the native ABI.
Not only that it is not guarantied, but even if you use the same compiler there might be differences due to different compiler switches used in the build, or if you use different versions of the same compiler and same switches (happened in an embedded compiler I worked on).
You need to make make sure the structs are represented exactly the same, use switches, #pragmas, whatever the compilers gives you.
My advice - to stay way from this altogether. Pass your arguments in the function, not wrapped within a struct.
And even in this simple form, if you deal with two compilers, it's not trivial. You need to make sure that an int takes the same number of bytes, for example. Also calling conevntion - arguments order - from left to right or from right to left - can differ between compiler.
Related
AFAIK in c-language structs get laid out, aligned and padded as the compiler sees fit. That is why you cannot rely on one c-program to use structs from another c-program. E.g. you cannot save a struct as a binary file that another c program will read and cast to that same struct. You may be able to use packed structs like that but that's not really good practice.
So I was surprised to learn that .so and DLL files have c functions that take complicated struct's (references thereof) as their parameters. At least my company's products do this.
Is this reliable, is it good practice? Is there some new standard for struct layout with sizes, alignment and padding all being the same?
I know a 64-bit program cannot call a 32-bit library, but still I thought struct layout can vary amongst compilers of the same bits.
For a given processor type and a given operating system, there is usually a standard ABI (application binary interface) which specifies things such as:
The width and endianness of integer types.
The width and representation of floating-point types.
Alignment constraints, which dictate the presence of padding in structures.
How parameters are passed to functions (in registers and on the stack).
For example, on the x86_64 processor architecture (i.e. the processor architecture of 64-bit PC), there are two popular ABIs: the Microsoft x64 calling convention, used on Windows, and the System V amd64 ABI (PDF — the part you're asking about specifically is in §3.1 “Machine interface”), used everywhere else. 32-bit x86 historically had more fragmentation.
So generally, if you use different compilers for the same platform (processor and operating system) in their default mode, they'll produce the same layouts for structs, and they'll generate function calling code that's compatible with how functions compiled by other compilers read their arguments. Problems arise when you mix different platforms, for example writing a struct on an embedded device and trying to read it on a PC.
I am wondering how mixing C and assembly can be possible as compilers generate code in different ways, for example many C compilers will use registers rather than pushing to the stack while making a function call, These functions will then move those registers into the appropiate memory locations because of this what if you write assembly code or link with an object file created by a different compiler that will call the C function but instead push the arguments to the stack rather than set the registers.
My guess is the C compiler assembly output has done it in such a clever way that it doesn't make a difference and it will still work but I can't be sure looking at the assembly code it doesn't appear it would work.
Can anyone answer my question as I am writing a compiler and need to know this so I don't make any mistakes should I want to link with a C module in the future.
The conventions that are used for calling functions are part of what's called the "application binary interface" (ABI). If this interface is specified, then all code that follows the specification can be linked together.
There is no standard ABI for C. However, most popular platforms have one prevailing C compiler that effectively produces a de-facto standard ABI (e.g. there's one for Windows, one for Linux on x86 (32 and 64 bit), one for Linux on ARM, etc.). ABIs may specify a large number of separate "calling conventions", and your C compiler will typically let you specify the desired convention at the point of function declaration using some vendor extension.
Conversely, if there is no documented ABI for your C compiler, or for an existing bit of object code, then you cannot in general link (or otherwise interact) with it successfully.
The memory layout of a struct is up to the compiler. So what happens when some code compiled by one compiler uses a struct generated by code compiled by another compiler?
For example, say I have a header file that declares a struct somestruct, and a function that returns the struct. One source file defines that function and is compiled by compiler A. Another source file uses than function and is compiled by compiler B and links against the binary of the other source file.
If the two compilers create two different layouts for somestruct, then what's the layout of the variable returned by the function? Does it defer to one compiler's layout, or will there be a memory bug when the second source file tries to access elements of the struct returned by the first source file? Is it an error at compile time or link time?
The function will return a structure as specified by the ABI of the compiler of the function. The callee compiler, will just treat the function as if it conforms to the ABI of itself.
Assuming the two compilers use a similar ABI, in most cases, no errors will be reported during compile-time or link time or even during runtime. For some compatible compilers like Clang, GCC, and Intel C Compiler on OS X and Linux, no errors should result (if there are errors then it's a bug of the compiler). However in real world it is usually difficult to find fully compatible compilers (in most cases their ABIs are similar but not exactly the same; such ABI errors will be even harder to track down because your app would appear normal and crashes under some really weird circumstances are encountered during runtime).
Just as Basile said, name mangling for C++ poses an additional difference in ABI, but such differences are more easily caught during compile time as the linker literally can't find the symbol of the function, rather than finding a function that is not compatible.
Also, passing structures is another headache in terms of ABI because there are multiple structure-packing ABIs, sometimes even different in "compatible" compilers like GCC/MinGW and MSVC. (See also the -m[no-]ms-bitfields option in GCC, which forces GCC to use the MSVC ABI for structures.) I have also seen some cases where passing structures by pointer is more reliable than passing structures by value.
The layout of data (e.g. structures etc...), and the call protocol (how are call done at the processor level) are defined in a (processor and operating system specific) document called Application Binary Interface. If both compilers are following the same ABI (for the same processor and the same operating system) their generated code should be interoperable.
See e.g. the wikipage for x86 calling conventions and the x86-64 ABI specification.
Name mangling, notably for C++, might also be an issue.
Read also Levine's book on Linkers and Loaders
I have a code which is compiled into a library (dll, static library and so). I want the user of this library to use some struct to pass some data as parameters for the library function. I thought about declaring the struct in the API header file.
Is it safe to do so, considering compilation with different compilers, with respect to structure alignment or other things I didn't think about?
Will it require the usage of the same compiler (and flags) for both the library and its user?
Few notes:
I considered giving the user a pointer and set all the struct via functions in the library, but this will make the API really not comfortable to use.
This question is about C, although it would be nice to know if there's a difference in c++.
If it's a regular/static library, the library and application should be compiled using the same compiler. There're a few reasons for this that I can think of:
Different compilers (as in different brands or compilers for different platforms) normally don't understand each other's object and library formats.
You don't want to compile different parts of the same program using different types (e.g. signed vs unsigned char), type sizes (e.g. long = 32 vs 64 bits), alignment and packing and probably some other things, all of which are allowed by the C standard to vary. Mixing and matching those things is usually a bad thing.
You may, however, often use slightly different versions of the same compiler to compile the library and the application using it. Usually, it's OK. Sometimes there're changes that break the code, though.
You may implement some "initialization" function in that header file (declared as static inline) that would ensure that types, type sizes, alignment and packing are the same as expected by the compiled library. The application using this library would have to call this function prior to using any other part of the library. If things aren't the same as expected, the function must fail and cause program termination, possibly with some good textual description of the failure. This won't solve completely the problem of having somewhat incompatible compilers, but it can prevent silent and mysterious malfunctions. Some things can be checked with the preprocessor's #if and #ifdef directives and cause compilation errors with #error.
In addition, structure packing problems can be relieved by inserting explicit padding bytes into structure declarations and forcing tight packing (by e.g. using #pragma pack, which is supported by many compilers). That way if type sizes are the same, it won't matter what the default packing is.
You can apply the same to DLLs as well, but you should really expect that the calling application has been compiled with a different compiler and not depend on the compilers being the same.
All Windows APIs throw structs around like crazy so obviously this is something that is done every day and it works. Of course it doesn't mean that your concerns are not valid :)
I would suggest making your structure's fields have explicit width types (int32_t etc) and maybe specify explicitly that that the packing in a way which would break on any compiler but yours, i.e.
#if defined(_MSC_VER)
#pragma pack(0)
#elif defined ... handle gcc
#else
FAIL // fail compilation on unsupported platform
#endif
From a discussion somewhere else:
C++ has no standard ABI (Application Binary Interface)
But neither does C, right?
On any given platform it pretty much does. It wouldn't be useful as the lingua franca for inter-language communication if it lacked one.
What's your take on this?
C defines no ABI. In fact, it bends over backwards to avoid defining an ABI. Those people, who like me, who have spent most of their programming lives programming in C on 16/32/64 bit architectures with 8 bit bytes, 2's complement arithmetic and flat address spaces, will usually be quite surprised on reading the convoluted language of the current C standard.
For example, read the stuff about pointers. The standard doesn't say anything so simple as "a pointer is an address" for that would be making an assumption about the ABI. In particular, it allows for pointers being in different address spaces and having varying width.
An ABI is a mapping from the execution model of the language to a particular machine/operating system/compiler combination. It makes no sense to define one in the language specification because that runs the risk of excluding C implementations on some architectures.
C has no standard ABI in principle, but in practice, this rarely matters: You do what your OS-vendor does.
Take the calling conventions on x86 Windows, for example: The Windows API uses the so-called 'standard' calling convention (stdcall). Thus, any compiler which wants to interface with the OS needs to implement it. However, stdcall doesn't support all C90 language features (eg calling functions without prototypes, variadic functions). As Microsoft provided a C compiler, a second calling convention was necessary, called the 'C' calling convention (cdecl). Most C compilers on Windows use this as their default calling convention, and thus are interoperable.
In principle, the same could have happened with C++, but as the C++ ABI (including the calling convention) is necessarily far more elaborate, compiler vendors did not agree on a single ABI, but could still interoperate by falling back to extern "C".
The ABI for C is platform specific - it covers issues such as register allocation and calling conventions, which are obviously specific to a particular processor. Here are some examples:
The ARM ABI (includes C++)
The PowerPC Embedded ABI
The several ABIs of x86
x86 has had many calling conventions, which extensions under Windows to declare which one is used. Platform ABIs for embedded Linux have also changed over time, leading to incompatible user space. See some history of the ARM Linux port here, which shows the problems in the transition to a newer ABI.
Although several attempts have been
made at defining a single ABI for a
given architecture across multiple
operating systems (Particularly for
i386 on Unix Systems), the efforts
have not met with such success.
Instead, operating systems tend to
define their own ABIs ...
Quoting ... Linux System Programming page 4.
An ABI, even for C, has parts which are quite platform independent, parts which depend on the processor (which registers should be saved, which are used for passing parameters,...) and parts which depend on the OS (more or less the same factors as for the processor as some choices are not imposed by the architecture but are the result of trade-offs, plus some OS's have a language independent notion of exception and so a compiler for any language has to generate the right thing to handle those, handling of threads may also impose things on the ABI -- if a register points to TLS, you can't use it for what you want).
In theory, every compiler may have its own ABI. But usually, for a couple processor/OS, the ABI is fixed by the OS vendor which often also provide a C compiler and common libraries which use that ABI and competitors prefer to be compatible. (I'd not be surprised if there are exceptions for some OS for which C isn't a major programming language).
But the OS vendor may switch ABI for one reason or the other (new versions of processors may have features that you want to use in the ABI for one - for instance some have asked for a 32bit ABI for x86_64 allowing to use all the registers). During the migration phase - which may be for a very long time - you may have to handle two ABI.
neither does C, right?Right
On any given platform it pretty much does. It wouldn't be useful as the lingua franca for inter-language communication if it lacked one.Pretty much might refer to architecture-specific defaults chosen by C compiler vendors being adapted within other languages. So if Keil's ARM C compiler will use left to right little endian parameter ordering and stack to pass arguments and some predetermined register for return value, then extern "C" from other compilers will assume compatibility with such scheme.
While such agreement maybe considered part of ABI, unlike managed execution context such as JVM browser sandbox, this is far from being complete standard ABI by itself.
C does not have a standard ABI. This is easily illustrated by all the calling conventions (cdecl, fastcall and stdcall) that are used out there. Each is a different ABI.
There's no standard ABI because C has always been about maximum runtime performance and the ABI with the highest performance depends on the underlying hardware. As a result, the ABI may use only stack or prefer registers for passing function call arguments and return values as needed for any given hardware.
For example, even amd64 (a.k.a x86-64) has two calling conventions: Microsoft x64 and System V AMD64 ABI. The former puts 4 first arguments to registers and the rest into the stack. The latter puts 6 first arguments to registers and the rest into the stack. I have no idea why Microsoft created non-compatible calling convention for amd64 hardware. For all I know, the Microsoft variant has a slightly worse performance and was created later.
For more information, see https://en.wikipedia.org/wiki/X86_calling_conventions
Prior to the C89 Standard, C compilers for many platforms used essentially the same ABI, save for variations in data sizes. For machines whose stack grows downward, code which calls a function would push the arguments on the stack in order from right to left and then call the function (pushing the return address in the process). A called function would leave its arguments on the stack, and the caller would at its leisure adjust the stack pointer to remove them [or, on some architectures, might adjust the stacked values in place]. While <stdarg.h> made it unnecessary for most programs to rely upon that convention, it remained in use for many years because it was simple and worked pretty well. While there was no "official" document establishing that as a cross-platform "standard", most compilers targeting machines with downward-growing stacks worked that way, leading to a greater level of consistency than exists today.