Here's the situation. I have an old legacy library that is broken in many places, but has a lot of important code built in (we do not have the source, just the lib + headers). The functions exposed by this library have to be handled in a "special" way, some post and pre-processing or things go bad. What I'm thinking is to create another library that uses this old library, and exposes a new set of functions that are "safe".
I quickly tried creating this new library, and linked that into the main program. However, it still links to the symbols in the old library that are exposed through the new library.
One thing would obviously be to ask people not to use these functions, but if I could hide them through some way, only exposing the safe functions, that would be even better.
Is it possible? Alternatives?
(it's running on an ARM microcontroller. the file format is ELF, and the OS is an RTOS from Keil, using their compiler)
[update]
Here's what i ended up doing: I created dummy functions within the new library that use the same prototypes as the ones in the old. Linked the new library into the main program, and if the other developers try to use the "bad" functions from the old library it will break the build with a "Symbol abcd multiply defined (by old_lib.o and new_lib.o)." Good enough for government work...
[update2]
I actually found out that i can manually hide components of a library when linking them in through the IDE =P, much better solution. sorry for taking up space here.
If you're using the GNU binutils, objcopy can prefix all symbols with a string of your choice. Just use objcopy --prefix-symbols=brokenlib_ old.so new.so (be careful: omitting new.so will cause old.so to be overwritten!)
Now you use brokenlib_foo() to call the original version of foo().
If you use libtool to compile and link the library instead of ld, you can provide -export-symbols to control the output symbols, but this will only work if your old library can be statically linked. If it is dynamically linked (.so, .dylib, or .dll), this will not be possible.
Related
I'm building an dynamic library, libfoo.so, which depends on libcrypto.so.
Within my autotools Makefile.am file, I have a line like this:
libfoo_la_LIBADD += -L${OPENSSL_DIR}/lib -lcrypto
where $OPENSSL_DIR defaults to /usr but can be overridden by passing --with-openssl-dir=/whatever.
How can I ensure that an executable using libfoo.so uses ${OPENSSL_DIR}/lib/libcrypto.so (only) without the person building or running the executable having to use rpath or fiddle with LD_LIBRARY_PATH?
As things stand, I can build libfoo and pass --with-openssl-dir=/usr/local/openssl-special and it builds fine. But when I run ldd libfoo.so, it just points to the libcrypto.so in /usr/lib.
The only solution I can think of is statically linking libcrypto.a into libfoo.so. Is there any other approach possible?
Details of runtime dynamic linking vary from platform to platform. The Autotools can insulate you from that to an extent, but if you care about the details, which apparently you do, then it probably is not adequate to allow the Autotools to choose for you.
With that said, however, you seem to be ruling out just about all possibilities:
The most reliable way to ensure that at runtime you get the specific implementation you linked against at build time is to link statically. But you say you don't want that.
If you instead use dynamic libraries then you rely on the dynamic linker to associate a library implementation with your executable at run time. In that case, there are two general choices for how you can direct the DL to a specific library implementation:
Via information stored in the program / library binary. You are using terminology that suggests an ELF-based system, and for ELF shared objects, it is the RPATH and / or RUNPATH that convey information about where to look for required libraries. There is no path information associated with individual library requirements; they are identified by SONAME only. But you say you don't want to use RPATH*, and so I suppose not RUNPATH either.
Via static or dynamic configuration of the dynamic linker. This is where LD_LIBRARY_PATH comes in, but you say you don't want to use that. The dynamic linker typically also has a configuration file or files, such as /etc/ld.so.conf. There you can specify library directories to search, and, with a bit of care, the order to search them.
Possibly, then, you can cause your desired library implementation to be linked to your application by updating the dynamic linker's configuration files to cause it to search the wanted path first. This will affect the whole system, however, and it's brittle.
Alternatively, depending on details of the nature of the dependency, you could give your wanted version of libcrypto a distinct SONAME. Effectively, that would make it a different object (e.g. libdjcrypto) as far as the static and dynamic linkers are concerned. But that is risky, because if your library has both direct and indirect dependencies on libcrypto, or if a program using your library depends on libcrypto via another path, then you'll end up at run time (dynamically) linking both libraries, and possibly even using functions from both, depending on the origin of each call.
Note well that the above issue should be a concern for you if you link your library statically, too. If that leaves any indirect dynamic dependencies on libcrypto in your library, or any dynamic dependencies from other sources in programs using your library, then you will end up with multiple versions of libcrypto in use at the same time.
Bottom line
For an executable, the best options are either (1) all-static linkage or (2) (for ELF) RPATH / LD_LIBRARY_PATH / RUNPATH, ensuring that all components require the target library via the same SONAME. I tend to like providing a wrapper script that sets LD_LIBRARY_PATH, so that its effect is narrowly scoped.
For a reusable library, "don't do that" is probably the best alternative. The high potential for ending up with programs simultaneously using two different versions of the other library (libcrypto in your case) makes all available options unattractive. Unless, of course, you're ok with multiple library versions being used by the same program, in which case static linkage and RPATH / RUNPATH (but not LD_LIBRARY_PATH) are your best available alternatives.
*Note that at least some versions of libtool have a habit of adding RPATH entries whether you ask for them or not -- something to watch out for. You may need to patch the libtool scripts installed in your project to avoid that.
I have a binary library and a binary executable using that library, both written in C. I know the C API provided by the library, but neither the source of the library or the executable. I would like to understand how the executable uses the library (compare my previous question How to know which functions of a library get called by a program).
The proposed solutions did not give satisfactory results. A possibility not mentioned seems to be to implement a wrapper library that imitates the known interface of the binary library I am interested in. My idea is to forward all of the calls to the wrapper to the binary library. This should allow me to log all the calls and passed parameters, in other words to instrument the library.
I succeeded in implementing the wrapper library on Linux as a dynamic link library (*.so), together with my own sample application connecting to the wrapper. The wrapper, in turn, uses the original binary library. Both libraries are used with dlopen and dlsym to access the API. However, I am facing the following practical problem: I do not manage to link the original binary executable to my wrapper library. That is related to the fact that the executable expects the library under a certain name. However, if I name my rapper library that way, it conflicts with the original library. Surprisingly (to me) simply renaming the .so-file of that one and linking the wrapper library against it does not work (The result stops without error message when the wrapper library calls dlopen and I do not get more information from the debugger than that it seems to happen in an malloc).
I tried a number of things like using symbolic links to move one of the libraries out of the search path of the run-time linker, to add paths to the LD_LIBRARY_PATH environment variable, different relative locations of the .so-files (and corresponding paths for dlopen), as well as different compiler options, so far without success.
To summarize, I would like
(executable)_orig->(lib.so)->(lib.so)_orig
where (executable)_orig and (lib.so)_orig (both binaries that I cannot influence) are such that
(executable)_orig->(lib.so)_orig
works. I have the sources of (lib.so) and can modify it as I wish. Also, I can modify the Linux host system as I like. The task of (lib.so) is to tell me how (executable)_orig and (lib.so)_orig interact.
I also have
(executable)->(wrapper.so)->(lib.so)_orig
working, which seems to indicate that the issue is related to the naming and loading conventions for the libraries.
This is a separate new question because it deals with the specific practical issue sketched above. Beyond that, some background info on why renaming the file corresponding to (lib.so)_orig to circumvent the issue may fail could also prove useful.
On Windows, it's more or less common to create "proxy DLLs" which take place of the original DLL and forward calls to it (after any additional actions as needed). You can read about it here and here for example.
However, shlib munging culture under Linux is quite different. It starts with the fact that LD_PRELOAD is the builtin feature with ld.so under Linux, which simply injects separate shlib into process and uses any symbols it defines as override. And that "injection" technique seems to define whole direction of thought - here's a typical ELF hacking tool or this question, where gentleman seems to have the same usecase as me, but starts with asking how he can patch existing binaries.
No, thanks. I don't want to inject into or modify something which is nor mine. All I want to do is to make a standalone proxy shlib which will call out to the original. Ideally, there would be a tool which can be fed with the original .so and create a C source code which would just redirect to original's functions, while letting me easily override anything I want. So, where's such tool? ;-) Thanks.
Using LD_PRELOAD doesn't really involve modifying something which isn't yours, and the injection isn't all that different from normal dynamic library loading. The “typical ELF hacking tool” from the ERESI project is unrelated to LD_PRELOAD. You should not be afraid of it. A good introduction to writing LD_PRELOAD-able “proxies” is here.
That being said, if you want to create a system-wide proxy for some library, you might argue that globally setting LD_PRELOAD (and thus loading your proxy into every binary that ever runs on your system) is undesirable. It is commonly used to override functions from glibc by tools such as libeatmydata or socksify, but if you're overriding a function in a library that is bigger and/or less widespread than glibc, it makes sense to try to find another approach, to really create a proxy for just that one library.
One such approach is to use patchelf --replace-needed or --add-needed to hardcode the full pathname of the original library and then make sure the proxy library is found first by setting LD_LIBRARY_PATH¹. So, the complete procedure is:
create an LD_PRELOAD-able library that overrides some functions of the original one (test that it works using only LD_PRELOAD before proceeding further!)
compile and link this library with the original library so that ldd libwrapper-foo.so includes something like:
libfoo.so.0 => /usr/lib/x86_64-linux-gnu/libfoo.so.0 (0x0000deadbeef0000)
hardcode the full path using patchelf:
patchelf --replace-needed libfoo.so.0 /usr/lib/x86_64-linux-gnu/libfoo.so.0 libwrapper-foo.so
symlink libwrapper-foo.so to libfoo.so.0
now LD_LIBRARY_PATH=. ldd $(which program-that-uses-libfoo) should include these lines:
libfoo.so.0 => ./libfoo.so.0 (0x0000dead56780000)
/usr/lib/x86_64-linux-gnu/libfoo.so.0 (0x0000dead1234000000)
set LD_LIBRARY_PATH to full path to the wrapper library in your .bashrc or somewhere
A real-life example of such proxy libary is my wrapper for libpango that enables subpixel positioning for all applications.
¹) It might also be possible to put this proxy library into /usr/local/lib, but ldconfig (the tool that updates shared libraries cache) refuses to use libraries with hardcoded absolute paths.
apitrace is a tool which covers detailed tracing of graphic libs (OpenGL, DirectX) calls for a number of platform. It's probably too detailed and complex for generic solution, but at least provides some reference and affinity.
I have implemented semaphores in Linux last year. But for that I have to use -lpthread.
Now while implementing log10() function in C, I surfed the INTERNET and I saw that I have to use -lm.
I want to know why these kind of command line arguments are necessary in Linux.And Does this rule is compiler oriented?
(In windows Turboc compiler, I never used these kind of arguments.)
You are instructing the compiler to look for certain libraries and use them to try and produce a final object file.
When you were doing your threading code, you used threading primitives. These threading primitives are implemented in a library called pthread, -lpthread tells the linker to use the library pthread, without providing this switch the compiler will not be able to produce a valid object file as it is missing threading code implementation.
On the file system the libraries can be found in /usr/lib and lib (among others) when you look in these directories you will see files start with the lib prefix. for example libpthreadxxxxxx. You will have to do your own research to figure out what the xxxx means.
The development cycle using unix style tools is very granular on the surface, when you use heavyweight IDE's (read: visual studiio for C++), the IDE implicetly links against loads of standard libraries, so often you do not need to supply the name of the libraries you will commonly use. However, when you start doing more advanced programming you will probably have to install and configure your IDE to use external code libraries. If you were to use threading primitives in visual studio, you most likely will not have to provide the compiler with information on where to look for threading primitives, Microsoft considers this a common library and every new project will implicitly link against it.
A little discussion on GCC
GCC is a very diverse compiler producing code for various different usage scenarios. As such they try to be neutral and do not make assumptions. For example pthread is a particular threading primitives implementation. However, even through now on Linux at least it is the defacto standard, it is not the only one. Other Unix implementation have had different implementation. When such choices exist it is not fair for the compiler developers to implicitly link against libraries. They do however implicitly link against standard libraries; for example G++ is just a wrapper command to the internal compiler code, it is a C++ front-end so it implicitly links against an implementation of the C++ standard library. Similarly the C front end links against a the standard C library.
People often do not want to use certain standard library implementation, and instead they might want to use another implementation, in such cases you have to explicetly inform the compiler to use an implementation that you provide. Such use cases are very granular and are surface level issues with G++. In visual studio, you would have to tinker a lot to make such changes generally, since it is not an anticipated use-case anymore.
wikipedia will provide you with more information.
Edit: I'll fix the spelling and Grammatical issues later :D
The option -l indicates to gcc what libraries must be used for linking. -lpthread stands for "use the pthread library", and -lm stands for "use the m library" which is the math library. These commands are relative to gcc, not linux.
Because by default, gcc only links the C library (libc), which contains the well-known functions printf, scanf, and many more.
log10 exists in a different library called libm, and thus you need to explictly tell gcc to link that library, with -lm. The same logic applies for -lpthread.
This is purely a backwards, harmful practice. Separating parts of the standard library into separate .so files does nothing but increase load time and memory usage. Good luck getting anyone to change it though... Just accept that you have to do it (and that POSIX specifically allows, but does not require, that an implementation require -lm for using the math functions and -lpthread for using threads, etc.) and move on to more important things.
Or, go pester Drepper about it on the glibc bug tracker/mailing list. He won't change his mind, but if you enjoy flamewars you can get some kicks...
Is there a cross platform way to selectively export certain functions and structs from a C project which builds a shared library?
I want to do in a way that does not require a specific build system (the visibility should be defined in the code, eg as a macro), and in a way which both GCC and MSVC can understand.
Thank you.
Strictly no, of course, because the toolchains aren't the same.
But people do this. The complexities are that in windows, you need to
specifically tag the declarations of functions you want exported from
a DLL with __declspec(dllexport) in the location in the library where
the function is defined and __declspec(dllimport) in the locations
in client code where the funciton is referenced. Because standard
C practice has only one declaration in a single header file, this
means that you generally need to do some macro work to have a single
prefix that works in both locations. It seems like every project
picks its own standard for this.
On the Unix side, you don't need to tag exports at all, which is nice.
This is because every non-static function is exported by default,
which is not so nice. Often you can get away with this as long as
your non-public/non-static symbols have sane prefixes, which is what
most projects seem to do. If you need finer control over your
exported symbols, you can use a Solaris-style "mapfile" with the GNU
linker's --version-script (-M under solaris) argument to define explicitly which symbols should appear
in the external namespace.
There are a few more gotchas between the platforms, like the way the
per-library global namespace works and the handling of
startup/shutdown code. Basically, it's a rats nest that can't be
sanely explained in a post this short, but as long as you're careful
that your library contains simple functions and your namespace doesn't
pollute, you shouldn't have much trouble. Look to some of the more
popular cross-platform shared libraries (e.g. Qt, Glib/Gtk+, anything
distributed with msys, etc...) for guidance.