I am trying to create a dynamic library which is meant to be linked and loaded into a host environment at runtime (e.g. similar to how class loading works in Java). As such, I want the dynamic library to be left with a few "dangling" references, which I expect it to pick up from its host environment when it is loaded into that environment.
My problem is that I cannot figure out how to create the dynamic library without explicitly linking it to existing symbols. I am hoping to produce a dynamic library that does not depend on a specific host executable (or host library), rather one that is able to be loaded (e.g. by dlopen) in any host as long as the host makes a couple symbols available for use.
Right now, any linking command I've tried results in a complaint of missing symbols. I'd like it to allow symbols to be missing (ideally, just particularly specified symbols).
For example, here's a transcript with the error on OS X:
$ cat frotz.c
void blort(void);
void run(void) {
blort();
}
$ cc -c -o frotz.o frotz.c
$ cc -dynamiclib -o libfrotz.dylib frotz.o
Undefined symbols for architecture x86_64:
"_blort", referenced from:
_run in frotz.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
If I do the same thing using a GNU toolchain (on Linux), it helpfully tells me:
$ gcc -shared -o libfrotz.so frotz.o
/usr/bin/ld: frotz.o: relocation R_X86_64_PC32 against undefined symbol `blort'
can not be used when making a shared object; recompile with -fPIC
and indeed, adding -fPIC to the C compile command seems to fix the problem in that environment. However, it doesn't seem to have any effect in OS X.
All the other dynamic-linking questions I could find on SO seem to be about the more usual arrangement of libraries, where a library is being built to be linked into an executable before that executable runs, rather than the other way around. The closest related question I found was this:
Can an executable be linked to a dynamic library after its built?
which unfortunately has very little info, none of it relevant to the question I'm asking here.
UPDATE: I distilled the info from the answer along with everything else I'd figured
out, and put together this example:
https://github.com/danfuzz/dl-example
As far as my knowledge goes, you want to use weak linkage:
// mark function as weakly-linked
extern void foo() __attribute__((weak));
// inform the linker about that too
clang -dynamiclib -o bar.dylib bar.o -flat_namespace -undefined dynamic_lookup
If a weak function can be resolved at runtime, it will then be resolved. If it can't, it will be NULL, instead of generating a runtime (or, obviously, link-time) error.
Related
I am trying to compile a program within a docker container built from the Alpine 3.7 base image. The program uses argp.h, and includes it as #include <argp.h>. I have installed argp-standalone and verified that it is making it onto the image. The file argp.h is located in usr/include, however when I compile my program using the following commands:
gcc -W -Wall -Wextra -I/usr/include -c -o progname.o progname.c
gcc -largp -o progname progname.o
I get the following error:
progname.o: In function `parse_opt':
progname.c:(.text+0x4c9): undefined reference to `argp_failure'
progname.c:(.text+0x50f): undefined reference to `argp_failure'
progname.c:(.text+0x555): undefined reference to `argp_failure'
progname.c:(.text+0x59b): undefined reference to `argp_failure'
progname.c:(.text+0x5ce): undefined reference to `argp_error'
progname.c:(.text+0x5f4): undefined reference to `argp_error'
progname.o: In function `main':
progname.c:(.text+0x1397): undefined reference to `argp_parse'
collect2: error: ld returned 1 exit status
make: *** [Makefile:9: progname] Error 1
I have:
Ensured that the version of argp.h which is on the image does in fact include the argp_failure, argp_parse, and argp_error functions.
Tried moving argp.h into different locations on the machine (e.g. into the same directory where compilation is taking place, into /usr/lib)
Tried compiling with -l and -L.
The relevant packages also installed in the image are build-base, make, and gcc.
When compiling on an Ubuntu image these same commands work fine, even without the -largp and -I/usr/include flags. What could be happening differently within an Alpine image which would cause this not to work?
Edit
As per #Pablo's comment, I'm now compiling it as follows:
gcc -W -Wall -Wextra -I/usr/include -L/usr/lib -c -o progname.o progname.c
gcc -largp -o progname progname.o
After having verified that the static library, libargp.a, is located in /usr/lib. However, the same problem still persists.
Edit 2
Compiling as follows (and once again as per #Pablo's suggestion) has resolved the error I was having:
gcc -W -Wall -Wextra -I/usr/include -L/usr/lib -c -o progname.o progname.c
gcc -o progname progname.o /usr/lib/libargp.a
However, I am still curious why, using the exact same library and instructions, this would fail to compile in an Alpine image while compiling without issue in an Ubuntu image.
I am still curious why, using the exact same library and instructions, this would fail to compile in an Alpine image while compiling without issue in an Ubuntu image.
The reason for the linking error on Alpine may be kind of surprising, and is actually not specific to Alpine.
While this fails to link:
gcc -largp -o progname progname.o
This works:
gcc -o progname progname.o -largp
The reason is the order of parameters passed to the linker, and it related to the linking algorithm. Typically, in the linking command line objects are specified first (and possibly user's static libraries in any), then libraries using -l. The standard linker algorithm is explained perfectly in Eli Bendersky's article, Library order in static linking:
Object files and libraries are provided in a certain order on the command-line, from left to right. This is the linking order. Here's what the linker does:
The linker maintains a symbol table. This symbol table does a bunch of things, but among them is keeping two lists:
A list of symbols exported by all the objects and libraries encountered so far.
A list of undefined symbols that the encountered objects and libraries requested to import and were not found yet.
When the linker encounters a new object file, it looks at:
The symbols it exports: these are added to the list of exported symbols mentioned above. If any symbol is in the undefined list, it's removed from there because it has now been found. If any symbol has already been in the exported list, we get a "multiple definition" error: two different objects export the same symbol and the linker is confused.
The symbols it imports: these are added to the list of undefined symbols, unless they can be found in the list of exported symbols.
When the linker encounters a new library, things are a bit more interesting. The linker goes over all the objects in the library. For each one, it first looks at the symbols it exports.
If any of the symbols it exports are on the undefined list, the object is added to the link and the next step is executed. Otherwise, the next step is skipped.
If the object has been added to the link, it's treated as described above - its undefined and exported symbols get added to the symbol table.
Finally, if any of the objects in the library has been included in the link, the library is rescanned again - it's possible that symbols imported by the included object can be found in other objects within the same library.
When -largp appears first, the linker does not include any of its objects in the linking procedure, since it doesn't have any undefined symbols yet. If the static library is provided by path, and not with -l, then all of its objects are added to the linking procedure.
I'm using cmake to generate a gstreamer library. In the end cmake uses the following command for linking:
/usr/bin/cc -fPIC -shared -Wl,-soname,libmacq-gstmelpi.so -o libmacq-gstmelpi.so <OBJECT_FILES> -lmacq-melpi -lmacq-icar-tools -lmacq-gstmecimeta -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 -lgstvideo-1.0
Note the -lgstvideo-1.0 flag at the end. The command runs fine, no error is produced, and the resulting library is created just fine. However when I use the library, I get undefined symbol error. So I use ldd to check; and amongst all the output of ldd; libgstvideo-1.0.so is not to be found.
This problem occurs on Ubuntu 14.04 on a armhf architecture. The problem does not occur on opensuse 13.1 (i586) nor on opensuse 13.1 (armv7hl), since in that case ldd libmacq-gstmelpi.so | grep gstvideo gives:
libgstvideo-1.0.so.0 => /usr/lib/libgstvideo-1.0.so.0 (0xb715f000)
EDIT :
I have another library, very similar where a very similar command works just fine; the resulting library is correctly linked to libgstvideo-1.0.so
/usr/bin/cc -fPIC -shared -Wl,-soname,libmacq-gstplugins.so -o libmacq-gstplugins.so <OBJECT_FILES> -lmacq-icar-tools -lmacq-gstmecimeta -lgstapp-1.0 -lgstbase-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 -lgstvideo-1.0
Some remarks on what I have checked and tried:
/usr/lib/arm-linux-gnueabihf/libgstvideo-1.0.so exists. (the other libraries in /usr/lib/arm-linux-gnueabihf/ are found without problem, libmacq-gstmelpi.so is linked to /usr/lib/arm-linux-gnueabihf/libgstreamer-1.0.so.0 without problem)
I tried changing the order, and putting the -lgstvideo-1.0 flag before all other -l flags; no succes.
replacing cc with c++ or gcc; the commands work, but output is the same
removing -lgstvideo-1.0 on a system where the build worked. The resulting library builds (links) without error; yet upon execution I have the same undefined symbol error as on Ubuntu. This proves that the missing symbol is in libgstvideo-1.0.so, and that I need it.
Are you actually using symbols from that particular library or do you just want to link to it to avoid linking it in the application that uses the library later on?
It could be a default compiler behavior that it skips linking of libraries when no symbol from these are actually used.
Try -Wl,--no-as-needed to your flags. In this case the library should get linked - not checking whether its symbols are actually used or not.
EDIT:
After chatting it turned out that the actual desired symbols are in gstbase-1.0 and not gstvideo-1.0. Since gstvideo-1.0 pulls in gstbase-1.0 as a dependency this worked but would cause problem as the linker may remove this dependency since no symbols from this particular library are being used. Linking directly to gstbase-1.0 seemed to solve all issues.
I am writing an application that has multiple subdirectories. I have created a Makefile structure such that the subdirectories compile the file and do "ar rvs" and "ranlib libxxx.a" to create archives into the parent directory for linking.
However the "ld" command is running into the following problem.
ld: warning: ignoring file ./libxxx.a, file was built for archive which is not
the architecture being linked (x86_64):
./libxxx.a Undefined symbols for architecture x86_64:
I am using gcc on Mac OS X 10.10.1
I read many posts on this. I tried "gcc -arch i386", then I encounter the same error for i386
Undefined symbols for architecture i386:
I installed gcc-4.9.2 and tried using it instead of the default gcc, with no luck. I tried using x86_64-apple-darwin14.0.0-g++-4.9.2 and that did not help either.
The errors you are seeing mean that you have a mixture of i386 and x86_64 code in the build, and you need to be consistent. Unless there's a compelling reason to do otherwise (I'd be curious to know what it is), you should be compiling everything for 64-bit only. Using gcc on Mac, that is normally the default, but you can force it by adding a -m64 flag to the compilations. One way of doing that is to set CC="gcc -m64" on the make command line; there are other better ways too, but the details depend on your makefile contents.
To resolve: first, remove all the libraries and object code you've built in your project area (maybe make clean will do it — if you wrote a clean target). Then, fettle the value of CC or its flags (CFLAGS ultimately, but how CFLAGS is built depends on the makefile) so that 64-bit compilation is enforced. Then make sure you're using that in all compilations. If you don't see the -m64, you've got a problem.
If you must use 32-bit, substitute -m32 for -m64.
The discussion above assumes you are using gcc to run the ld command. If you are not, then you're on your own until you use gcc to run the ld command. In my view, you've got better things to do with your life than work out how to run the ld command correctly; I certainly have.
My build environment is CentOS 5. I have a third party library called libcunit. I installed it with autotools and it generates both libcunit.a and libcunit.so. I have my own application that links with a bunch of shared libraries. libcunit.a is in the current directory and libcunit.so and other shared libraries are in /usr/local/lib/. When I compile like:
gcc -o test test.c -L. libcunit.a -L/usr/local/lib -labc -lyz
I get a linkage error:
libcunit.a(Util.o): In function `CU_trim_left':
Util.c:(.text+0x346): undefined reference to `__ctype_b'
libcunit.a(Util.o): In function `CU_trim_right':
Util.c:(.text+0x3fd): undefined reference to `__ctype_b'
But when I compile with .so like:
gcc -o test test.c -L/usr/local/lib -lcunit -labc -lyz
it compiles fine and runs fine too.
Why is it giving error when linked statically with libcunit.a?
Why is it giving error when linked statically with libcunit.a
The problem is that your libcunit.a was built on an ancient Linux system, and depends on symbols which have been removed from libc (these symbols were used in glibc-2.2, and were removed from glibc-2.3 over 10 years ago). More exactly, these symbols have been hidden. They are made available for dynamic linking to old binaries (such as libcunit.so) but no new code can statically link to them (you can't create a new executable or shared library that references them).
You can observe this like so:
readelf -Ws /lib/x86_64-linux-gnu/libc.so.6 | egrep '\W__ctype_b\W'
769: 00000000003b9130 8 OBJECT GLOBAL DEFAULT 31 __ctype_b#GLIBC_2.2.5
readelf -Ws /usr/lib/x86_64-linux-gnu/libc.a | egrep '\W__ctype_b\W'
# no output
Didn't notice that the libcunit.a is actually found in your case and the problem with linakge is rather in the CUnit library itself. Employed Russian is absolutely right, and he's not talking about precompiled binary here. We understand that you've built it yourself. However, CUinit itself seems to be relying on the symbol from glibc which is not available for static linking anymore. As a result you have only 2 options:
File a report to the developers of CUnit about this and ask them to fix it;
Use dynamic linking.
Nevertheless, my recommendation about your style of static linkage still applies. -L. is in general bad practice. CUnit is a 3rd party library and should not be placed into the directory containing source files of your project. It should be rather installed in the same way as the dynamic version, i.e. like you have libcunit.so in /usr/local/lib. If you'd supply prefix to Autotools on the configure stage of CUnit, then Autotools would install everything properly. So if you want to link statically with CUnit, consider doing it in the following form:
gcc -o test test.c -L/usr/local/lib -Wl,-Bstatic -lcunit -Wl,-Bdynamic -labc -lyz
I'm attempting to do a release of some software and am currently working through a script for the build process. I'm stuck on something I never thought I would be, statically linking LAPACK on x86_64 linux. During configuration AC_SEARCH_LIB([main],[lapack]) works, but compilation of the lapack units do not work, for example undefiend reference to 'dsyev_' --no lapack/blas routine goes unnoticed.
I've confirmed I have the libraries installed and even compiled them myself with the appropriate options to make them static with the same results.
Here is an example I had used in my first experience with LAPACK a few years ago that works dynamically, but not statically: http://pastebin.com/cMm3wcwF
The two methods I'm using to compile are the following,
gcc -llapack -o eigen eigen.c
gcc -static -llapack -o eigen eigen.c
Your linking order is wrong. Link libraries after the code that requires them, not before. Like this:
gcc -o eigen eigen.c -llapack
gcc -static -o eigen eigen.c -llapack
That should resolve the linkage problems.
To answer the subsequent question why this works, the GNU ld documentation say this:
It makes a difference where in the command you write this option; the
linker searches and processes libraries and object files in the order
they are specified. Thus, foo.o -lz bar.o' searches libraryz' after
file foo.o but before bar.o. If bar.o refers to functions in `z',
those functions may not be loaded.
........
Normally the files found this way are library files—archive files
whose members are object files. The linker handles an archive file by
scanning through it for members which define symbols that have so far
been referenced but not defined. But if the file that is found is an
ordinary object file, it is linked in the usual fashion.
ie. the linker is going to make one pass through a file looking for unresolved symbols, and it follows files in the order you provide them (ie. "left to right"). If you have not yet specified a dependency when a file is read, the linker will not be able to satisfy the dependency. Every object in the link list is parsed only once.
Note also that GNU ld can do reordering in cases where circular dependencies are detected when linking shared libraries or object files. But static libraries are only parsed for unknown symbols once.