How to link a binary in a manner to be compatible with two existing version of a library that have conflicting SONAME ?
Those two versions don't share same SONAME prefix. One is libcapi10.so.3 and the other is libcapi10.so.4.
I can't recompile them since i get them as binary, and since those are certified crypto libraries i can't request new one with right SONAME. Of course i would not have faced any problem if one was libcap10.so.3 and the other libcap10.so.3.1 since i would just need to link over the first to be compatible with the second.
Those two libraries are told to be binary compatible ( i should trust this info ).
I searched but didn't find any good way to do , either with linker options or using objcopy. I would like to avoid patching binary by hand to use it at compilation linking time.
So back to my initial question : How to specify SONAME to be ( in this case libcap10.so ) used for link ?
( I already searched, and my current findings are just that it is a no go, but unfortunately this is a requirement ... ).
Update:
i patched .so library using a binary sed-like tool replacing libcapi10.so.6\0 with libcapi10.so\0, what works since new name is shorter than previous and that elf structure for SONAME is a C-string ended with a 0 and that elf checksum is not used during gcc linking. i used that patched library only at compilation time, then i can use either one or the other original library on my target system with the same binary.
patchelf is your friend. You can do something like: patchelf --replace-needed libcapi10.so.3 libcapi10.so.4 <your_thing>.
Patchelf is useful for a variety of other things such as changing RPATH, too. Check out the manpage. Very nifty toy.
My current best answer allowing to fullfill my need.
I patched .so library using a binary sed-like tool replacing libcapi10.so.6\0 with libcapi10.so\0, what works since new name is shorter than previous and that elf structure for SONAME is a C-string ended with a 0 and that elf checksum is not used during gcc linking. i used that patched library only at compilation time, then i can use either one or the other original library on my target system with the same binary.
best hints in stackverflow found there :
How can I change the filename of a shared library after building a program that depends on it?
Related
nokogori gem comes with its own version of libxml2. Moreover it warns about libxml2.so of a different version being loaded before it was required:
if compiled_parser_version != loaded_parser_version
["Nokogiri was built against LibXML version #{compiled_parser_version}, but has dynamically loaded #{loaded_parser_version}"]
It basically compares LIBXML_DOTTED_VERSION macro and xmlParserVersion global variable:
rb_const_set( mNokogiri,
rb_intern("LIBXML_VERSION"),
NOKOGIRI_STR_NEW2(LIBXML_DOTTED_VERSION)
);
rb_const_set( mNokogiri,
rb_intern("LIBXML_PARSER_VERSION"),
NOKOGIRI_STR_NEW2(xmlParserVersion)
);
And I'm experiencing it firsthand. When rmagick (which dynamically links to libxml2.so, ldd confirms that) is required before nokogiri, the latter complains.
From what I can see nokogiri is linked to libxml2 statically. First that is the default (supposedly). Then when rmagick is not required I can't see libxml2.so in /proc/PID/maps. I neither can see another version of libxml2.so. ldd doesn't list libxml2.so as a nokogiri.so's dependency. objdump lists xmlReadMemory (and friends) as a nokogori.so's symbol (probably a sign that it was linked statically).
So how come can nokogiri access libxml2.so's variables? Does that mean that loading libxml2.so overrides any statically linked versions? Can that happen in the middle of code execution?
So how come can nokogiri access libxml2.so's variables?
This is by design (and due to the fact that nokogiri was built incorrectly).
UNIX shared libraries are designed to emulate archive libraries. This means that the first ELF image to export a given symbol wins (there are some complications for libraries linked with -Bstatic flag, but we'll ignore them for now).
Does that mean that loading libxml2.so overrides any statically linked versions?
Yes, if statically linked version is also exported and calls to it go though PLT.
Can that happen in the middle of code execution?
With lazy symbol resolution (which is default, except when LD_BIND_NOW or -z now are in effect) it will always happen in the middle of code execution.
Now, the problem is that if nokogiri links in a static copy of libxml.a, it should hide that fact by localizing that copy within itself and not exporting any of its symbols. That would prevent end users from having to deal with symbol conflicts.
Your best bet is to either build your own nokogiri compiling and linking it against the same version of libxml, or to contact nokogiri maintainers and ask them to fix their builds.
Basically, I want to get a list of libraries a binary might load.
The unreliable way I came up with that seems to work (with possible false-positives):
comm -13 <(ldd elf_file | sed 's|\s*\([^ ]*\)\s.*|\1|'| sort -u) <(strings -a elf_file | egrep '^(|.*/)lib[^:/]*\.so(|\.[0-9]+)$' | sort -u)
This is not reliable. But it gives useful information, even if the binary was stripped.
Is there a reliable way to get this information without possible false-positives?
EDIT: More context.
Firefox is transitioning from using gstreamer to using ffmpeg.
I was wondering what versions of libavcodec.so will work.
libxul.so uses dlopen() for many optional features.
And the library names are hard-coded. So, the above command helps
in this case.
I also have a general interest in package management and binary dependencies.
I know you can get direct dependencies with readelf -d, dependencies of
dependencies with ldd. And I was wondering about optional dependencies, hence the question.
ldd tells you the libraries your binary has been linked against. These are not those that the program could open with dlopen.
The signature for dlopen is
void *dlopen(const char *filename, int flag);
So you could, still unreliably, run strings on the binary, but this could still fail if the library name is not a static string, but built or read from somewhere during program execution -- and this last situation means that the answer to your question is "no"... Not reliably. (The name of the library file could be read from the network, from a Unix socket, or even uncompressed on the fly, for example. Anything is possible! -- although I wouldn't recommend any of these ideas myself...)
edit: also, as John Bollinger mentioned, the library names could be read from a config file.
edit: you could also try substituting the dlopen system call with one of yours (this is done by the Boehm garbage collector with malloc, for example), so it would open the library, but also log its name somewhere. But if the program didn't open a specific library during execution, you still won't know about it.
(I am focusing on Linux; I guess that most of my answer fits for every POSIX systems; but on MacOSX dlopen wants .dylib dynamic library files, not .so shared objects)
A program could even emit some C code in some temporary file /tmp/foo1234.c, fork a compilation of that /tmp/foo1234.c into a shared library /tmp/foo1234.so by some gcc -O -shared -fPIC /tmp/foo1234.c -o /tmp/foo1234.so command -generated and executed at runtime of your program-, perhaps remove the /tmp/foo1234.c file -since it is not needed any more-, and dlopen that /tmp/foo1234.so (and perhaps even remove /tmp/foo1234.so after dlopen), all that in the same process. My GCC MELT plugin for gcc does exactly this, and so does Bigloo, and the GCCJIT library is doing something close.
So in general, your quest is impossible and even has no sense.
Is there a reliable way to get this information without possible false-positives?
No, there is no reliable way to get such information without false positives (you could prove that equivalent to the halting problem, or to some other undecidable problem). See also Rice's theorem.
In practice, most dlopen happens on plugins provided by some configuration. There might not be exactly named as such in a configuration file (e.g. some Foo programs might have a convention like a plugin named bar in some foo.conf configuration file is provided by foo-bar.so plugin).
However, you might find some heuristic approximation. Most programs doing some dlopen have some plugin convention requesting some particular symbol names in the plugin. You could search for shared objects defining these names. Of course you'll get false positives.
For example, the zsh shell accepts plugins called zsh modules. the example module shows that enables_,
boot_, features_ etc... functions are expected in zsh modules. You could use nm -D to find *.so files providing these (hence finding the plugins likely to be perhaps loadable by zsh)
(I am not convinced that such an approach is worthwhile; in fact you should usually know which plugins are useful on your system by which applications)
BTW, you could use strace(1) on the execution of some command to understand the syscalls it is doing, hence the plugins it is loading. You might also use ltrace(1), or pmap(1) (on some given process), or simply -for a process 1234- use cat /proc/1234/maps to understand its virtual address space, hence the plugins it has already loaded. See proc(5).
Notice that strace, ltrace, pmap exist on Linux, but many POSIX systems have similar programs.
Also, a program could generate some machine code at runtime and execute it (SBCL does that at every REPL interaction!). Your program could also use some JIT techniques (e.g. with libjit, llvm, asmjit, GCCJIT or with hand-written code...) to do likewise. So plugin-like behavior can happen without dlopen (and you might mimic dlopen with mmap calls and some ELF relocation processing).
Addenda:
If you are installing firefox from its packaged version (e.g. the iceweasel package on Debian), its package is likely to handle the dependencies
I have an autotools C project.
How do I compile a binary which works with both libcrypto.so.0.9.8 and libcryto.so.1.0.0? (i.e. Ubuntu 9.10 and 12.04)
Depending on the age of the OS on which I do the build, the binary requires one version or the other.
Is there a way of making it not care, or are the differences between the libcryto versions insurmountable?
In my opinion, you want to use some function of libcrypto.so.0.9.8 and some from libcryto.so.1.0.0. If most of the functions are required from 1.0.0 or is preferred choice then link with libcrypto.so.1.0.0.
And you may need some function from libcrypto.so.0.9.8 or you may have other good reasons to use libcrypto.so.0.9.8.
In my view, if you link from both the library, you will get linker error (duplicate symbols as both of the library contains same symbols).
If you need to use 0.9.8, then load it dynamically using dlopen and get the function callback which you want use with dlsym.
This can be accomplished as follows:
void * handle;
/*reqd_callback is the callback of required function.*/
reqd_callback cb;
handle = dlopen ("libcrypto.so.0.9.8", RTLD_LAZY);
cb = (reqd_callback)dlsym(handle, "reqd_function");
//Call the cb
cb (parameters);
//Close the library.
dlclose(handle);
I think this may solve your purpose. If the preference is inverse, invert the library in linking and in loading through program.
Can you make a soft link that will "point" to either libcrypto.so.0.9.8 or libcryto.so.1.0.0.
Give it a generic name and then use that, then whatever version of the library the link "points" to, will be picked up? On app install, you set your soft link to point to the library version available. Your software might be tested up to 1.0.0 if the lib is backward compatible enough, i.e. you don't rely on somthing in 1.0.0 that's not in 0.9.8, your ok.
You can rebuild ssl-0.9.8 (not 1.x because it contains some things that won't work on the older version) and change the line in the makefile where it does the final shared library linkage and embeds the SONAME
recompile it with the SONAME changed from libssl.so.0.9.8 to libssl.so.0
it will look something like: -Wl,-soname,libssl.so.0.9.8
change it to: -Wl,-soname,libssl.so.0
now when you compile against this library the binaries built against it will look for libssl.so.0 (which is included as a symlink in both versions)
I have existing Linux shared object file (shared library) which has been stripped. I want to produce a new version of the library with some additional functions included. I had hoped that something like the following would work, but does not:
ld -o newlib.so newfuncs.o --whole-archive existinglib.so
I do not have the source to the existing library. I could get it but getting a full build environment with the necessary dependencies in place would be a lot of effort for what seems like a simple problem.
You might like to try coming at this from a slightly different angle by loading your object using preloading.
Set LD_PRELOAD to point to your new object
export LD_PRELOAD=/my/newfuncs/dir/newfuncs.o
and specify the existing library in the same way via your LD_LIBRARY_PATH.
This will then instruct the run time linker to search for needed symbols in your object before looking in objects located in your LD_LIBRARY_PATH.
BTW You can put calls in your object to then call the function that would've been called if you hadn't specified an LD_PRELOAD object or objects. This is why this is sometimes called interposing.
This is how many memory allocation analysis tools work. They interpose versions of malloc() and free() that records the calls to alloc() and free() before then calling the actual system alloc and free functions to perform the memory management.
There's plenty of tutorials on the interwebs on using LD_PRELOAD. One of the original and best is still "Building library interposers for fun and profit". Though written nine years ago and written for Solaris it is still an excellent resource.
HTH and good luck.
Completely untested idea:
# mv existinglib.so existinglib-real.so
# ld -o exlistinglib.so -shared newfuncs.o -lexistinglib-real
The dynamic linker, when loading a program that expects to load existinglib.so, will find your version, and also load existinglib-real.so on which it depends. It doesn't quite achieve the stated goal of your question, but should look as if it does to the program loading the library.
Shared libraries are not archives, they are really more like executables. You cannot therefore just stuff extra content into them in the same way you could for a .a static library.
Short answer: exactly what you asked for can't be done.
Longer answer: depending on why you want to do this, and exactly how existinglib.so has been linked, you may get close to desired behavior. In addition to LD_PRELOAD and renaming existinglib.so already mentioned, you could also use a linker script (cat /lib/libc.so to see what I mean).
I'm planning to release some compiled code that shall be linked by client applications on MacOSX.
The distribution is some kind of code library and a set of header files defining the public interface for the library.The code is internally C++ but its public interface (i.e what's being shown in the headers) is completely C.
These are my requirements or atleast what I hope I can accomplish:
I want my library to be as agnostic
as possible for what version of OSX
and GCC the user is running. Having
separate libraries for 64 bit and 32
bit is okay though.
I want my library
to be loadable from languages that
supports loading C libraries such as
python or similar.
I want my
libraries internal symbols to be
isolated from the code it's being
linked into. I don't want to have
duplicate symbol errors because we
happen to name an internal function
in the same way. My C++ code is properly namespaced so this may not be as big of an issue though, but some of the libraries I depend on is C and can be an issue (see next point).
I want my library
dependencies to be safe. My library
depends on some libraries such as
libpng, boost and stl and I don't
want issues because some users don't
necessarily have all of them installed
or get problems because they have
been compiled with other flags or
have different versions than I have.
On Windows I use a DLL with an export library and link all my dependencies statically into the dll. It fulfills all the criteria above and if I can get the same result on OSX it would be great, however I've heard that dynamic libraries tend not to isolate symbols on mac in the same way.
Is there some kind of best practice for this on OSX?
A normal OS X .dylib pretty much satisfies your requirements, with the note that you will want to have an exports file that the linker uses to determine exactly which symbols are exported (to prevent leaking your internal symbols).
In order to make your own library dependencies safe, you will probably need to either include those libraries with yours or link them statically into your library.
edit: To answer your follow-up question of how to apply an exports file to a link command, the man page for ld has the following to say:
-exported_symbols_list filename
The specified filename contains a list of global symbol names
that will remain as global symbols in the output file. All
other global symbols will be treated as if they were marked
as __private_extern__ (aka visibility=hidden) and will not be
global in the output file. The symbol names listed in file-
name must be one per line. Leading and trailing white space
are not part of the symbol name. Lines starting with # are
ignored, as are lines with only white space. Some wildcards
(similar to shell file matching) are supported. The *
matches zero or more characters. The ? matches one charac-
ter. [abc] matches one character which must be an 'a', 'b',
or 'c'. [a-z] matches any single lower case letter from 'a'
to 'z'.
So, if your library had only two functions that you wanted to be public, lets call them foo and bar, and they were C functions (so the symbol names aren't mangled), your exports file (let's call it myLibrary.exports) would contain these two lines:
_foo
_bar
and maybe some comments, etc. When you do the final link step to build the library, you would pass the -exported_symbols_list myLibrary.exports flag to the linker. This has the additional benefit that the link will fail if you don't provide one of the exported symbols; this can catch a lot of "oops, I forgot to include that file in the build" mistakes.
You don't need to use the command-line tools to do all this, of course. In the build settings for a dynamic library in XCode, you will find Exported Symbols File (undefined by default); set it to the path to your exports file there and it will be passed to the linker.
The key term you need is 'framework'. You need to create a 'universal' framework that is self-contained. ('Universal' is Apple-ease for 'compile several times and package into one library.) It's not as straightforward as on Windows in terms of encapsulation, but the necessary linker options are there.