I have a specialized shared library that is loaded dynamically during the execution of an executable. This library provides the function b(). This function, in turn, calls a function a().
The a function is defined in the executable, not the library itself, so I would like to be able to call some of my executable's code from the library. I've managed to use the -undefined dynamic_lookup flag to make clang leave these symbols for lazy binding, but dyld still complains when I try to run the executable:
dyld: lazy symbol binding failed: Symbol not found: _a
Referenced from: /usr/local/lib/myLib.dylib
Expected in: flat namespace
How can I get this symbol linked?
As it turns out, this was caused by Xcode not exporting the symbol by default. You can fix this by changing the "Symbols Hidden By Default" option under the LLVM Code Generation build settings. If you're not using Xcode, this controls whether or not to use the -fvisiblity=hidden flag on the command line.
Alternatively, you can leave this option turned on and selectively export symbols by adding the __attribute__((visibility("default"))) attribute to the functions you want to export.
Related
System spec: Ubuntu 16.04, QtCreator 3.5.1 based on Qt5.5.1
In a C project I'm using dlopen and friends to load a shared library. The library uses some machinery from the main application.
Now, if I compile manually from terminal, everything works just fine. When I use QtCreator to build the project, calls from the library to the main application fail with the following error:
/Programming/C/emme_1/emme_1: symbol lookup error: /Programming/C/emme_1/modtest/test.so: undefined symbol: pop. Here pop is a function in the main application which is named emme_1. Please note that the error happens only if the library's function calls another function inside the main application. If the library's function is empty and is called from the application, it works.
This is my very simple .pro file:
QMAKE_CFLAGS += -DTRACE_ASM
QMAKE_CFLAGS += -rdynamic
QMAKE_CFLAGS += -ldl
LIBS += /usr/lib/x86_64-linux-gnu/libdl.so
I believe it contains all it's needed for the requested functionality, but maybe I'm missing something.
I believe it contains all it's needed for the requested functionality, but maybe I'm missing something.
You must be missing something since the setup doesn't work.
Note that -rdynamic and -ldl are linker flags, not compile flags. You'll want to move them to QMAKE_LFLAGS.
You can look at the final link command, and verify that -rdynamic is not in it. Once you manage to get it there, things will work again.
I'm compiling a C++ static library using g++ via Cmake. I want to remove symbols relating to the internal implementation so they don't show up in nm. (See here and here for the same with shared libraries.)
This answer tells you how to do it on iOS, and I'm trying to understand what happens under the hood so I can replicate on Linux. They invoke ld with:
-r/--relocatable to Generate relocatable output---i.e., generate an output file that can in turn serve as input to ld.
-x/--discard-all: Delete all local symbols.
AFAICS the -r glues all the modules into one module, and then the -x removes symbols only used inside that module. Is that right?
It's not clear how the linker 'knows' which symbols will be exported externally? Does it rely on __attribute__((visibility("hidden/default"))) as in the .so case?
Edit: clearly I'm confused... I thought cmake invoked ld to link the .os into .a. Googled + clarified above.
Question still stands: how do I modify the build process to exclude most symbols?
A colleague gave me a modified version of a shared library where he added a GTK widget.
When inspecting the shared library file I see that the new widget functions are defined as local and not global.
I have tried to set the visibility attribute of GCC on the function (after the declaration itself, before the semicolon), it has G_BEGIN_DECLS around it and the same common headers and defines as other files in the library that are exported properly.
Is there a linker command line option I may be missing? A list of files that "can" export that is used by gcc, perhaps another definition for exported functions?
When inspecting the shared library file I see that the new widget functions are defined as local and not global.
By default, all symbols in a shared library are exported (unless you compile with -fvisibility=hidden or protected.
Since observe that your symbols are LOCAL, it is a good bet that your link command uses a linker version script to control symbol visibility (to hide all symbols except ones that are explicitly exported), and that you have not modified that version script to add your functions to the export list.
Look for -Wl,--version-script=... on your link command line, and modify the version script appropriately.
See also this answer.
I've found out that the library uses a regular expression to filter exports (the -export-symbols-regex switch), adding another regular expression made the symbols properly exported, now I everything is linking properly.
My question is fairly OS X on x86-64 specific but a universal solution that works on other POSIX OSes is even more appreciated.
Given a list of symbol names of some shared library (called original library in the following) and I want my shared library to re-export these symbols. Re-export as in if someone tries to resolve the symbol against my library I either provide my version of this symbol or (if my library doesn't have this symbol) forward to the original library's symbol.
I don't know the types of the symbols, I only know whether they are functions (type T in nm output) or other symbols (type S in nm output).
For functions, I already have a solution: For every function I want to re-export I generate an assembly stub that does dynamically resolve the symbol (using dlsym()) and then jumps into the resolved function with the very same environment (registers rdi, rsi, rdx, rcx, r8, r9, stack pointer, ...). I'm basically generating universal proxy functions. Using some macro trickery that can be generated fairly easy without writing code for each and every symbol.
For non-function symbols the problem seems to be harder because I cannot generate this universal proxy function, because the resolving party does never call a function.
Using a constructor function static void init(void) __attribute__((constructor)); I can execute code whenever someone loads my library, that would be a good point to resolve and re-export all non-function symbols if that's possible.
In other words, I'd like to write the symbol table of my library to point to the respective symbols of another shared library. Doing the rewriting at compile or run time is okay (run time preferred). Or put yet another way, the behaviour of DYLD_INSERT_LIBRARIES (LD_PRELOAD) is exactly what I need but I don't want to insert a new library, I want to replace one (in the file system). EDIT: The reason I don't want/can't use DYLD_INSERT_LIBRARIES or any other environment variable of the DYLD_* family is that they are ignored for code signed, restricted, ... binaries.
I'm aware of the -reexport-l, -reexport_library and -reexported_symbols_list linker flags but I could not get them to work, especially when my library is a "replacement" for frameworks that are part of umbrella frameworks (example: /System/Library/Frameworks/CoreServices.framework/Frameworks/SearchKit.framework/SearchKit) because ld forbids to link directly against parts of umbrella frameworks.
EDIT: Because I explained it somewhat ambiguously: I can't change the way the actual program is linked. The goal is to produce a shared library that is a replacement for the original library. (Apparently called filter library.)
Found it out now (OS X specific): clang -o replacement-lib.dylib ... -Xlinker -reexport_library PATH_TO_ORIGINAL_LIB does the trick. PATH_TO_ORIGINAL_LIB could for example be /System/Library/Frameworks/CoreServices.framework/Frameworks/SearchKit.framework/Versions/Current/SearchKit.
If PATH_TO_ORIGINAL_LIB is a library that is part of an umbrella framework (as in the example above), then replace PATH_TO_ORIGINAL_LIB by the path of some other lib (I created a lib empty.dylib for that) and as a second step do
install_name_tool -change /usr/local/lib/empty.dylib PATH_TO_ORIGINAL_LIB replacement-lib.dylib
To see if the actual reexporting worked use:
otool -l replacement-lib.dylib | grep -A2 LC_REEXPORT_DYLIB
The output should look like
cmd LC_REEXPORT_DYLIB
cmdsize XX
name empty.dylib (offset YY)
After launching the install_name_tool it could be
cmd LC_REEXPORT_DYLIB
cmdsize XX
name /System/Library/Frameworks/CoreServices.framework/Frameworks/SearchKit.framework/Versions/Current/SearchKit (offset YY)
You could link against both libraries and use the link order to make sure to link against the right symbols. This works on both OS X and Linux:
cc -o executable -lmylib -loriglib
Where origlib is the original library and mylib contains symbols that are supposed to overwrite symbols in origlib. Then the executable will be linked against your symbols from mylib first and all unresolved symbols will be linked against origlib.
This works in the same way when linking against OS X frameworks. Just link against your library that replaces symbols first and against the framework after.
cc -o executable -lmylib -framework SomeFramework
Edit: If you just want to replace symbols at runtime then you can use LD_PRELOAD in the same way:
cc -o executable -framework SomeFramework
LD_PRELOAD=libmylib.dylib ./executable
I'm pretty new to working with libraries and I'm in the process of trying to understand some specifics regarding static libraries and object files.
Summary
The behavior I'm noticing is that I can link several objects to make an executable with no problem, but if I take an intermediate step of combining those objects into static libraries, I cannot link those static libraries to make an executable without additionally specifying the needed C Run-time library in the link command.
Also, or the record, I'm doing the compiling/linking with Visual Studio 2010 from the command line. More details of the process I'm following are below.
First, let's say I have four source files in a project: main.c, util1.c, util2.c, and util3.c.
What works
I can compile these sources with the following command:cl -c main.c util1.c util2.c util3.cAs a result, I now have four object files: main.obj, util1.obj, util2.obj, and util3.obj. These object files each contain a DEFAULTLIB statement intended to inform the linker that it should additionally check the static C Run-time library libcmt.lib for any unresolved external dependencies in these object files when linking them.
I can create an executable named "app_objs.exe" by linking these objects with the following command:
link -out:app_objs.exe main.obj util1.obj util2.obj util3.obj
As mentioned in step 1, the linker used the runtime library due to the compiler's step of adding a default library statement to the objects.
Where I'm confused
Let's say I want to have an intermediate step of combining these objects into static libraries, and then linking those resulting LIB files to create my executable. First, I can create these libraries with the following commands:
link -lib -out:main.lib main.obj
link -lib -out:util.lib util1.obj util2.obj util3.obj
Now, my original thought was that I could simply link these libraries and have the same executable that I created in step 2 of "What works". I tried the following command and received linker error LNK1561, which states that an entry point needs to be specified:
link -out:app_libs.exe main.lib util.lib
From Microsoft's documentation, it is evident that linking libraries without any object files may require entry points to be specified, so I modified the command to set the subsystem as "console" to specify that the executable in intended to be a console application (which seems to imply certain entry points, thereby resolving that error):link -out:app_libs.exe -subsystem:console main.lib util.libUnfortunately, now I get a linker error stating that mainCRTStartup is an unresolved external symbol. I understand that this is defined in the C runtime library, so I can resolve this issue by manually specifying that I want to link against libcmt.lib, and this gives me a functioning executable:link -out:app_libs.exe -subsystem:console main.lib util.lib libcmt.lib
What I'm not understanding is why the default library info that the compiler placed in each object file couldn't be used to resolve the dependency on libcmt.lib. If I can link object files without explicitly stating I want libcmt.lib, and I created static libraries that are containers for the object files, why can't I link those static libraries without having to explicitly state that I want libcmt.lib? Is this just the way things are, or is there some way I could create the static libraries so that the linker will know to check for unresolved symbols in the runtime library?
Thanks for your help. If I have some fundamentally incorrect ideas here, I'd love suggestions on good references to learn all of this correctly.
Well the answer to your misunderstanding is that .lib files are often a product in themselves, and the compiler can't make those assumptions safely. That's what "external" is for.
If I produce binaries for someone's platform because its users are totally helpless, and they want/need static linkage, I have to give them foo.h and libfoo.lib without tying them to a specific runtime entry point. They may very well have defined their own entry point already for their final product, whether DLL or EXE.
You either want the runtime, or you want your own .obj that contains your entry point. Be warned that declaring and defining mainCRTStartup on your own may mean you're not executing important instructions for the target platform.