We need to integrate a closed-source C library into our project. The other party can compile the library in any mode we need, but we support a lot of different Windows and Linux compilers.
I know a DLL will be compatible with all major Windows compilers, so we could only use one library for Windows. But I’m not familiar with Linux dynamic libraries (.so). Are these compatible across all Linux environments in a similar way?
Thanks
You may have some issues w.r.t. the version of the GNU libc against which that shared library libotherparty.so has been linked (and compiled).
You certainly need to know the version of the libc, and you might require that party to ship several variants of the binary library.
I would strongly suggest to read Drepper's paper: How To Write Shared Libraries. See also this question and Levine's linker & loader book.
If it is C code, the compiler does not matter much. What matters more is the ABI. Learn also about symbol versioning.
Be prepared to have some trouble.
You'll learn by experience that free software is preferable.
Library versions should be specified for shared objects if the function interfaces are expected to change (C++ public/protected class definitions), more or fewer functions are included in the library, the function prototype changes (return data type (int, const int, ...) or argument list changes) or data type changes (object definitions: class data members, inheritance, virtual functions, ...).
The library version can be specified when the shared object library is created. If the library is expected to be updated, then a library version should be specified. This is especially important for shared object libraries which are dynamically linked. This also avoids the Microsoft "DLL hell" problem of conflicting libraries where a system upgrade which changes a standard library breaks an older application expecting an older version of the the shared object function.
Versioning occurs with the GNU C/C++ libraries as well. This often make binaries compiled with one version of the GNU tools incompatible with binaries compiled with other versions unless those versions also reside on the system. Multiple versions of the same library can reside on the same system due to versioning. The version of the library is included in the symbol name so the linker knows which version to link with.
One can look at the symbol version used: nm csub1.o
00000000 T ctest1
No version is specified in object code by default.
Look ld and object file layout
There is one GNU C/C++ compiler flag that explicitly deals with symbol versioning. Specify the version script to use at compile time with the flag: --version-script=your-version-script-file
Note: This is only useful when creating shared libraries. It is assumed that the programmer knows which libraries to link with when static linking. Runtime linking allows opportunity for library incompatibility.
Also look here regarding ABI compatibility
Related
My host application took over the ownership of e.g. a FILE object which came from a dynamic library. Can I call fclose() on this object safely even though my host application and the dynamic library are compiled with different versions of clang / gcc?
Background
On Windows (with different VS runtimes) it would be illegal and I have to first extract the fclose() function from the runtime library which is used by the dynamic library since all runtimes have their own pools and internal structures for file or memory objects.
An illustration for the situation in Windows would look like this:
Does this restriction apply for Linux and macOS as well?
The issue is not whether your application and the dynamic libraries were compiled with different versions of clang and/or gcc. The issue is whether, ultimately, there's one underlying C library that manipulates one kind of FILE * object and has one, compatible implementation of fclose().
Under MacOS and Linux, at least, the answer to all these questions is likely to be "yes". In my experience it's hard to get two different, incompatible C libraries into the mix; you'd have to really work at it.
Addendum: I suppose I should admit, however, that my experience may be getting dated. In my experience, on any Unix-like system, there's exactly one C library, generally /lib/libc.{a,so}. But I gather that "modern" compilers are tending to access their own compiler- and version-specific libraries off in special places, meaning that the scenario you're worried about could be a problem. To me, it seems, this way lies madness, but then again, it seems that more and more of the world seems to be embracing dependency hell, rather than trying to eliminate it.
It is not generally safe to use a library designed for one compiler with code compiled by a different compiler. A compiler may generate code that implements the nominal functions in the standard library using internal routines or interfaces, and those routines or interfaces may be different or missing in the library designed for another compiler.
Nor is it safe to take any pointer to some internal data structure from one library and use it with another library.
If the sources are just compiled with different versions of one compiler (e.g., clang 73 and clang 89), not different compilers (e.g., Apple clang versus GCC), the compiler might offer some guarantee about library compatibility. You would have to check its documentation. Or, if the compiler is intended to use the library provided with the operating system, that could work. Again, you would have to check its documentation.
On Linux, if both your code and the other library dynamically link to the same library (such as libc.so.6), both will get the same version and implementation of that library at runtime. You can check which libraries a given dynamic library links to with ldd.
If you were linking to a library that statically linked in a supporting library, you would need to be careful to pass any structures to or from it against the same version of the library. But this is more likely to come up in practice with libc++ and libstdc++ than with libc.
So, don't statically link your library to another and then pass a data structure that requires client code to separately link to the same library.
I'm compiling a Linux toolchain based on Newlib for a toy project.
Newlib's official page reports:
Newlib is a C library intended for use on embedded systems.
but without providing any particular reason.
What I'm trying to figure out is:
Why newlib consider itself embedded "only"?
What are the downsides to use it on desktop or server environments?
Unlike Glibc, which has very specific code to make sure that it is replaceable with later API compatible versions, the Newlib isn't so; nor does it support dynamic linking anyway. And it doesn't make much sense to statically link in the C library in every possible executable in a desktop environment. Therefore, Newlib is mostly suitable for embedded targets with small number of statically linked executables.
Newlib also fulfils only the parts of the C standard library and a a minimal part of the POSIX C library extensions. Specifically it doesn't concern itself with networking at all. It is somewhat debatable if anyone in their right mind, would want to build a desktop system without any networking at all, in 2017.
Another thing to note is the non-technical aspect of licensing. Glibc uses the LGPL license, which does allow linking against proprietary programs, provided that (note that IANAL) the user is able to replace the LGPL-licenced library with another one. In practice this means that either the library is dynamically linked in, or, in case of a statically linked library, the user is provided with object files that they can use to link against the replacement library to produce an executable. This means that the license itself might not be suitable for small embedded systems with proprietary software. Newlib doesn't contain any LGPL code, unless it is built for Linux targets.
I took over a fairly large C code. There are lots of legacy binaries that are requiring old version shared libraries. The server has never versions of those exact libraries. I could recompile or setup symbolic links that will connect older versions to new. Setting up symbolic links will take some time - is there any standard or smart way to do this? I am new to this and would appreciate any tips. This is all C and FreeBSD environment.
Thanks.
In general when updating legacy code with new libraries, it is best to perform a check by recompiling the source code against the new libraries and their includes. This will allow you to use the compiler to check for inconsistencies between the old and new libraries in areas such as data types, function signatures, etc.
By recompiling you also are able to check that the new libraries provide all of the dependencies that you need.
Finally, doing a recompile will help you check that you are in fact able to recompile and link everything and have all of the necessary components.
I would feel uncomfortable tying to take a short cut such as using symbolic links.
The shared-library version number is only supposed to be changed when the ABI changes. (Old versions of FreeBSD didn't quite get this right, and it's fixed in more recent versions but only for system libraries!) So the only way to make those applications work properly is to either recompile them, or supply the exact version of the shared library that they were linked against. For programs that only depend on old versions of the FreeBSD system libraries, you can installes the compat[45678]x packages, which provide the versions of the libraries supplied with the specified version of the OS -- but there are significant pitfalls:
1) If some of the libraries your application depends on are linked against newer versions of the standard libraries than your application itself is, the dynamic linker will give you two incompatible copies of the standard library, and things are not likely to work.
2) If your application loads external modules or plug-ins using dlopen(), all bets are off, because these modules are not versioned.
FreeBSD 8 and newer use symbol versioning for the C library and some other important system libraries, so those libraries should never change library version again and ABI compatibility will be preserved. Many third-party developers are not so careful, and will both break ABI without changing the library version, and change the library version without breaking the ABI, so you can't win. (Some developers don't read the documentation and think that the shared-library version number should be the same as the product's version number.)
I need to verify something for which I have doubts. If a shared library ( .dll) is written in C, with the C99 standard and compiled under a compiler. Say MinGw. Then in my experience it is binary compatible and hence useable from any other compiler. Say MS Visual Studio. I say in my experience because I have tried it successfully more than once. But I need to verify if this is a rule.
And in addition I would like to ask if it is indeed so, then why libraries written completely in C, like openCV for example don't provide compiled binaries for every different OS? I know that the obvious reason would be to set all the compile-time parameters, but other than that there is none right?
EDIT: I am adding an additional question which I see as a logical extension to the original. Isn't this how one would go and create a closed source library? Since the option of giving source goes out of the window there, giving binaries is the only choice. And in that case providing binaries for as many architectures as possible is the desired result, with C being an obvious choice for having the best portability between systems and compilers. Right?
In the specific case of C compilers (MSVC and GCC/MinGW) in the Windows world, you are correct in the assumption of binary compatibility. One can link a C interface DLL compiled by GCC to a program in Visual Studio. This is the way C99 projects like ffmpeg allow developers to write application wiht Visual Studio. One only needs to create the import library with lib.exe found in the Microsoft toolchain from the DLL. Or vice versa, using mingw.org's pexports or better, mingw-w64's gendef tool, one can create a GCC import lib for a MSVC produced DLL.
This handy interoperability breaks down when you enter the C++ interface world, where the ABI of MSVC and GCC is different and incompatible. It may work, it may not, no guarantees are made and no effort is (currently) being done in changing that. Also, debugging info is obviously different, until someone writes a debug information generator/writer in GCC that is compatible to MSVC's debugger (along with gdb support of course).
I don't think C99 specifically changes anything to function declarations or the way arguments are handled in symbol definitions, so there should be no problem here either.
Note that as Vijay said, there is still the architecture difference, so a x86 library can't be used when linking to an AMD64 library.
To also answer your additional question about closed source binaries and distributing a version for all available compilers/architectures.
This is exactly the way you would create a closed source binary. In addition to the import library, it is also very important to hide exports from the DLL, making the DLL itself useless for linking (if you don't want client code to use private functions in the library, see for example the output of dumpbin /exports on a MSOffice DLL, lots of hidden stuff there). You can achieve the same thing with GCC (I believe, never used or tried it) using things like __attribute(hidden) etc...
Some compiler specific points:
MSVC comes with four (well, actually only three remaining in newer versions) different runtime libraries through /MT, /MD, and /LD. On top of this, you would have to provide a build for each version of Visual Studio (including Service Packs) to assure compatibility. But that is closed source binary and Windows for you...
GCC does not have this problem; MinGW always links to msvcrt.dll provided by Windows (since Windows 98), equivalent with /MD (and maybe also a debug library equivalent with /MDd). But I there are two versions of MinGW (mingw.org and mingw-w64) which do not guarantee binary compatibility. THe latter is more complete as it provides 64-bit options as well as 32-bit, and provides a more complete header/library set (including a substantial part of DirectX and DDK).
The general rule is that IF your OS/CPU combination has a standard ABI, and IF that ABI is powerful enough for your language, most compilers will follow that ABI and as a result will be binary compatible, allowing you to link libraries (shared or static) compiled with different compilers to programs compiled with other compilers just fine.
The problem is that most ABIs are fairly weak -- they're designed around low-level languages like C and FORTRAN and date back to the days before object oriented languages like C++. So they tend to lack support for things like function overloading, user-defined operators, exceptions, global contructors and destructors, virtual functions, inheritance, and such that are needed by C++.
This lack was recognized when C++ was designed which is why C++ has extern "C" -- which causes the compiler to limit itself to the standard ABI for certain functions, while disabling all the extra C++ features that the ABIs generally don't support.
A shared library or dll compiled to a particular architecture can be linked to applications compiled by other compilers that target the same architecture. (By architecture, I mean a processor/OS combination). But it is not practical for a library developer to compile against all possible architectures. Moreover, when a library is distributed in source form, users can build binaries optimized to their specific requirements.
Is it possible to have glibc and uClibc based applications running side-by-side on one system?
Background: We have binary gcc based cross-compiler configured to link with uClibc. We have cross-compiled glibc with it. Now we want to build some applications so they will link with the glibc rather than uClibc. We don't want to rebuild the compiler.
There's no problem with glibc and uClibc living side-by-side with some programs linking to one and other programs linking to the other. However, there is a problem with additional libraries. Each shared library on your system will be built against either glibc or uClibc (using the corresponding headers, which define distinct ABIs for the standard library functions), so for example if both a glibc program and a uClibc program need ncurses, you'll need to have two versions of ncurses built, and have a way of ensuring that the correct one for the given program gets loaded at runtime. Alternatively, you could choose to only use one set of shared libraries, and use static libraries for programs linked to the other libc, but you'd still need to build your 2 sets of libraries.
Yes, it should be perfectly possible, but you might have to play around with LD_PRELOAD_PATH. If you are linking statically, change to dynamic linking.
It is nearly impossible to mix them in the same FHS, as the ABI and include dir are incompatible. However, you could install either of them in an directory offset, by tweaking dynamic-linker field in ELF and exploiting sysroot feature in gcc/binutils. An on going experiment is in Gentoo community[1], known as Prefix/libc.
http://wiki.gentoo.org/wiki/Prefix/libc