When to actually use dlopen()? Does dlopen() means dynamic loading? - linker

I have gone through below link, through which I understood how to create and use shared library.
https://www.cprogramming.com/tutorial/shared-libraries-linux-gcc.html
Step 1: Compiling with Position Independent Code
$ gcc -c -Wall -Werror -fpic foo.c
Step 2: Creating a shared library from an object file
$ gcc -shared -o libfoo.so foo.o
Step 3: Linking with a shared library
$ gcc -L/home/username/foo -Wall -o test main.c -lfoo
Step 4: Making the library available at runtime
$ export LD_LIBRARY_PATH=/home/username/foo:$LD_LIBRARY_PATH
$ ./test
This is a shared library test...
Hello, I am a shared library
However, I have couple of questions:
In the given link, there is no usage of dlopen(), which is required to open shared library. How this code is working without dlopen() calls?
When to actually use dlopen()?
Can I compile program without having .so file available (program has function calls to that shared library)?
Does the dlopen() means dynamic loading and example in above link(step 3) means static loading? If Yes, then in case of dynamic loading, is there any difference in linking step(step 3)?
Thanks in advance.

Dynamic Library
When we link an application against a shared library, the linker leaves some stubs (unresolved symbols) to be filled at application loading time. These stubs need to be filled by a tool called, dynamic linker at run time or at application loading time.
Loading of a shared library is of two types,
Dynamically linked libraries –
Here a program is linked with the shared library and the kernel loads the library (in case it’s not in memory) upon execution.
This is explained in mentioned link.
Dynamically loaded libraries –
Useful for creating a "plug-in" architecture.
As the name indicates, dynamic loading is about loading of library on demand and linked during execution.
The program takes full control by calling functions with the library.
This is done using dlopen(), dlsym(), dlclose().
The dlopen() function opens a library and prepares it for use.
With this system call it is possible to open a shared library and use the functions from it, without having to link with it. Your program just starts, and when it finds out that it needs to use a function from a specific library, it calls dlopen() to open that library. If the library is not available on the system, the function returns NULL and it is up to you, the programmer, to handle that. You can let the program gracefully die.
DL Example:
This example loads the math library and prints the cosine of 2.0, and it checks for errors at every step (recommended):
#include <stdlib.h>
#include <stdio.h>
#include <dlfcn.h>
int main(int argc, char **argv) {
void *handle;
double (*cosine)(double);
char *error;
handle = dlopen ("/lib/libm.so.6", RTLD_LAZY);
if (!handle) {
fputs (dlerror(), stderr);
exit(1);
}
cosine = dlsym(handle, "cos");
if ((error = dlerror()) != NULL) {
fputs(error, stderr);
exit(1);
}
printf ("%f\n", (*cosine)(2.0));
dlclose(handle);
}
Use -rdynamic while compiling DL source code.
Ex. gcc -rdynamic -o progdl progdl.c -ldl

Related

LD_PRELOAD doesn't affect dlopen() with RTLD_NOW

If I use a function from a shared library directly, i.e. by declaring it in my code and linking during compile time, LD_PRELOAD works fine. But if I use dlopen()/dlsym() instead LD_PRELOAD has no effect!
The problem is that I want to debug a program that loads some plugins using dlopen(), and it uses absolute file names at that, so simply using LD_LIBRARY_PATH won't work.
Here's a sample code which illustrates the problem.
./libfoo.so
void foo() {
printf("version 1\n");
}
./preload/libfoo.so
void foo() {
printf("version 2\n");
}
main.c
#include <stdio.h>
#include <dlfcn.h>
void foo();
int main(int argc, char *argv[]) {
void (*pfoo)();
foo(); // call foo() first so we are sure ./preload/libfoo.so is loaded when we call dlopen()
pfoo = dlsym(dlopen("libfoo.so", RTLD_NOW), "foo");
pfoo();
return 0;
}
command line
LD_PRELOAD=preload/libfoo.so LD_LIBRARY_PATH=. ./a.out
output
version 2
version 1
Why doesn't LD_PRELOAD affect dlopen(), and is there any way to redirect dlopen(), especially when using absolute paths?
Specifying LD_PRELOAD will cause the loader to unconditionally load (and initialize) the indicated shared libraries prior to loading the main executable. This makes the symbols defined in the preloaded libraries available prior to linking main, allowing the interposition of symbols. [Note 1]
So, in your example, the call to foo() uses the symbol from the preloaded module, and dlsym would return the same symbol if you had called it with a NULL handle.
However, the call to dlopen does not take into account the symbol you are looking for (for obvious reasons). It just loads the indicated shared object or returns a handle to an already-cached version of the shared object. It does not add the module to a list of modules to load if necessary; it simply loads the module. And when you pass the returned handle to dlsym, dlsym looks in precisely that module in order to resolve the symbol, rather than searching the set of external symbols present in the executable. [Note 2]
As I mentioned, dlopen will not load the "same" shared object more than once, if it already has loaded the object. [Note 3]. However, the shared object in your LD_PRELOAD is called preload/libfoo.so, not libfoo.so. (ELF does not strip directory paths from shared object names, unlike certain other operating systems.) So when you call dlopen("libfoo.so"), the dynamic loader is not going to find any shared object named libfoo.so in the cache of loaded shared objects, and it will therefore look for that object in the filesystem, using the library search paths since the provided filename does not contain a /.
As it turns out, ELF does allow you to specify the name of a shared object. So you can set the name of the preloaded module to the name which you will later dynamically load, and then dlopen will return the handle to the preloaded module.
We start by correcting the version of main.c in the original question:
#include <stdio.h>
#include <dlfcn.h>
void foo();
int main(int argc, char *argv[]) {
const char* soname = argc > 1 ? argv[1] : "libfoo.so";
void (*pfoo)();
pfoo = dlsym(NULL, "foo"); // Find the preloaded symbol, if any.
if (pfoo) pfoo(); else puts("No symbol foo before dlopen.");
void* handle = dlopen(soname, RTLD_NOW);
if (handle) {
pfoo = dlsym(handle, "foo"); // Find the symbol in the loaded SO
if (pfoo) pfoo(); else puts("No symbol foo after dlopen.");
}
else puts("dlopen failed to find the shared object.");
return 0;
}
This can be built without specifying any libraries other than libdl:
gcc -Wall -o main main.c -ldl
If we build the two shared libraries with no specified names, which is probably what you did:
gcc -Wall -o libfoo.so -shared -fPIC libfoo.c
gcc -Wall -o preload/libfoo.so -shared -fPIC preload/libfoo.c
then we observe that the dlopen/dlsym finds the symbol in the loaded module:
$ LD_PRELOAD=preload/libfoo.so LD_LIBRARY_PATH=. ./main libfoo.so
version 2
version 1
However, if we assign the name being looked for to the preloaded shared object, we get a different behaviour:
$ gcc -Wall -o preload/libfoo.so -Wl,--soname=libfoo.so -shared -fPIC preload/libfoo.c
$ LD_PRELOAD=preload/libfoo.so LD_LIBRARY_PATH=. ./main libfoo.so
version 2
version 2
That works because the dlopen is looking for the shared object named libfoo.so. However, it is more likely that an application loading plugins will use a filename rather than using the library search path. And that will cause the preloaded shared object to not be considered, because the names no longer match:
$ LD_PRELOAD=preload/libfoo.so LD_LIBRARY_PATH=. ./main ./libfoo.so
version 2
version 1
As it happens, we can make this work by building the shared library with the name actually being looked for:
$ gcc -Wall -o preload/libfoo.so -Wl,--soname=./libfoo.so -shared -fPIC preload/libfoo.c
$ LD_PRELOAD=preload/libfoo.so LD_LIBRARY_PATH=. ./main libfoo.so
version 2
version 2
That's a bit of a hack, IMHO, but it is acceptable for debugging. [Note 4]
Notes:
Consequently, the comment "call foo() first so we are sure ./preload/libfoo.so is loaded" is incorrect; the preloaded module is preloaded, not added to a list of modules to load if necessary.
If you want dlsym to just look for a symbol, you can pass a NULL handle. In that case, dlsym will search in modules loaded by dlopen (including modules required by the module loaded by dlopen). But that's rarely what you want, since applications which load plugins with dlsym normally specify a particular symbol (or symbols) which the plugin must define, and these symbols will be present in every loaded plugin, making the lookup by symbol name imprecise.
This is not quite correct, but dynamic symbol namespaces are outside the scope of this answer.
Other hacks are possible, of course. You could, for example, interpose your own version of dlopen to override the shared object name lookup. But that's probably a lot more work than necessary.
According to http://linux.die.net/man/3/dlopen
The four functions dlopen(), dlsym(), dlclose(), dlerror() implement
the interface to the dynamic linking loader.
Whereas the LD_PRELOAD only affects the dynamic linker itself -- ie: ld.so (http://linux.die.net/man/8/ld.so). The only way I can think of forcing dlopen to resolve as you want is through chroot.
Followup thought:
Another thought I just had, what if you write a wrapper that FIRST loads the correct *.so THEN invokes the program you are trying to redirect. Does this cause the child process to use the redirected *.so ?
The problem is that I want to debug a program that loads some plugins using dlopen(), and it uses absolute file names at that, so simply using LD_LIBRARY_PATH won't work.
Yep, dlopen does not do LD_LIBRARY_PATH search for paths with slashes.
You can override/superimpose dlopen itself to do that search for those specific plug-in paths.

How to compile ffmpeg.c into shared library?

Is it possible to compile ffmpeg.c into the shared library? I need it in shared library form to link with my C program and call main function in ffmpeg.c.
Yes, it is possible. To do so you'd need to:
Find out how you gonna pass the arguments (if via argc/argv no changes would be needed)
Rebuild the object files linked into ffmpeg shared library with -fpic
Pass the -shared option to the linker

Ubuntu C program loadable module and undefined symbols

Very new to linux in general and trying to build a loadable module for use in zabbix, which works, but trying to build a simple shell program for testing it. That means this module needs to be loaded dynamically.
Sharable module SNMPmath is built with this:
gcc -shared -o SNMPmath.so $(CFLAGS) -I../../../include -I/usr/include/libxml2 -fPIC SNMPmath.c
That works fine for zabbix.
The test program (TestSO.c) uses
lib_handle = dlopen("./SNMPmath.so", RTLD_NOW);
to load this image dynamically, and when it does, it is missing symbols including init_snmp which come from the net-snmp package which is referenced in the SNMPmath loadable module.
My question is both general and specific. What's the right approach -- is this library called by the loadable module supposed to be forced into the loadable module? Is it supposed to be forced into the test program (despite having no compile-time reference to it)? Or is it supposed to be dynamically loaded, itself, in the loadable module (which seems to contradict what I'm seeing in other examples)?
And in either case, how is the GCC command modified to include net-snmp? I've tried variations of whole-archive, no-as-needed, listing what I think is the library (/usr/lib/x86_64-linux-gnu/libsnmp.a) with various compiler options with no effect (or occasionally errors). Also tried linking to the .so version of that (no effect). So a hint as to the proper GCC command to include the library would be very helpful.
Here's one iteration of attempts at linking the main program:
gcc -rdynamic -o TestSO -I../../../include -I/usr/include/libxml2 TestSO.c -ldl -Wl,--no-as-needed /usr/lib/x86_64-linux-gnu/libsnmp.so
I've found numerous examples of loading modules, but they all load a simple routine that does not itself have undefined symbols that need satisfying.
Recap:
TestSO.c
==> Loads with dlopen SNMPmath.c
==> needs to refer to net-snmp routines like init_snmp
Pointers to examples or explanation welcome, I realize I'm missing something fairly obvious.
EDIT AFTER FIRST COMMENTS TO INCLUDE:
I have it sort of working now, but would appreciate a sanity check if this is correct. I pruned it down so as to show the whole code. Here is the code to produce the SO:
#include <net-snmp/net-snmp-config.h>
#include <net-snmp/net-snmp-includes.h>
#include <string.h>
int zbx_module_SNMPmath_avg(int i, int j)
{
init_snmp("snmpapp"); // initialize SNMP library
return 1;
}
Here is how it is compiled (note slight name change):
CFLAGS=-I. `net-snmp-config --cflags`
SNMPmath_small: SNMPmath_small.c
gcc -shared -o SNMPmath.so $(CFLAGS) -I../../../include -I/usr/include/libxml2 -fPIC SNMPmath_small.c -Wl,--no-as-needed,/usr/lib/x86_64-linux-gnu/libsnmp.so
Then here is the main program:
#include <stdio.h>
#include <dlfcn.h>
#include <dlfcn.h>
#include <string.h>
int main(int argc, char **argv)
{
void *lib_handle;
int(*fn)(int req, int ret);
int x;
char *error;
lib_handle = dlopen("./SNMPmath.so", RTLD_NOW);
if (!lib_handle)
{
fprintf(stderr, "Error on open - %s\n", dlerror());
exit(1);
}
else
fprintf(stderr,"Successfully loaded module\n");
fn = dlsym(lib_handle, "zbx_module_SNMPmath_avg");
if ((error = dlerror()) != NULL)
{
fprintf(stderr, "Error on dlsym %s\n", error);
exit(1);
}
else fprintf(stderr,"Successfully called dlsym\n");
// testing
int req, ret;
req=1;
ret=1;
x=(*fn)(req, ret);
printf("Valx=%i\n",x);
dlclose(lib_handle);
return 0;
}
And finally this is built with:
TestSO: TestSO.c
gcc -rdynamic -o TestSO -I../../../include -I/usr/include/libxml2 TestSO.c -ldl
This now will run as expected. I found that linking against the netsnmp library's so file when building my so seemed to work.
But is this the correct sequence? And/or the preferred sequence?
Ps. Off to read the paper in the first proposed answer.
Pointer to explanation: Drepper's paper: Howto write shared libraries
But we need more source code (and the commands used to compile it), and the real error messages (e.g. as given by dlerror() after dlopen call) to help more.
(See also this answer)
You might want to add some libraries like -lsomething (I don't know which, you probably do know!) to your command gcc -shared which is building SNMPmath.so .... You don't want to link a static library like libsnmp.a to it (you should link shared libraries to your SNMPmath.so shared object).

Trying to understand the main function with GCC and Windows

They say that main() is a function like any other function, but "marked" as an entry point inside the binary, an entry point that the operating system may find (Don't know how) and start the program from there. So, I'm trying to find out more about this function. What have I done? I created a simple .C file with this code inside:
int main(int argc, char **argv) {
return (0);
}
I saved the file, installed the GCC compiler (in Windows, MingW environment) and created a batch file like this:
gcc -c test.c -nostartfiles -nodefaultlibs -nostdlib -nostdinc -o test.o
gcc -o test.exe -nostartfiles -nodefaultlibs -nostdlib -nostdinc -s -O2 test.o
#%comspec%
I did this to obtain a very simplistic compiler and linker, no library, no header, just the compiler. So, the compiling goes well but the linking stops with this error:
test.c:(.text+0xa): undefined reference to '___main'
collect2.exe: error: Id returned 1 exit status
I thought that the main function is exported by the linker but I believed that you didn't need any library with additional information about it. But it looks like it does. In my case I supposed that it must be the standard GCC library, so I downloaded the source code of it and opened this file: libgcc2.c
Now, I don't know if that is the file where the main function is constructed to be linked by GCC. In fact, I don't understand how the main function is used by GCC. Why does the linker need the gcc standard libraries? To know what about main? I hope this has made my question quite specific and clear. Thanks!
When gcc puts together all object files (test.o) and libraries to form a binary it also prepends a small object (usually crt0.o or crt1.o), which is responsible for calling your main(). You can see what gcc is doing, when you add -v on the command line:
$ gcc -v -o test.exe test.o
crt0/crt1 does some setup and then calls into main. But the linker is finally responsible for building the executable according to the OS. With -v you can see also an option for the target system. In my case it's for Linux 64 bit: -m elf_x86_64. For your system this will be something like -m windows or -m mingw.
The error happens because you use these two options: -nodefaultlibs -nostdlib
These tell GCC that it should not link your code against libc.a/c.lib which contains the code which really calls main(). In a nutshell, every OS is slightly different and most of them don't care about C and main(). Each has their own special way to start a process and most of them are not compatible with the C API.
So the solution of the C developers was to put "glue code" into the C standard library libc.a which contains the interface which the OS expects, creates the standard C environment (setting up the memory allocation structures so malloc() will map the OS's memory management functions, set up stdio, etc) and eventually calls main()
For C developers, this means they get a libc.a for their OS (along with the compiler binaries) and they don't need to care about how the setup works.
Another source of confusion is the name of the reference. On most systems, the symbolic name of main() is _main (i.e. one underscore) while __main is the name of an internal function called by the setup code which eventually calls the real main()

Delay-Load equivalent in unix based systems

What is the delay load equivalent in unix based system.
I have a code foo.cpp, While compiling with gcc I link it to shared objects(totally three .so files are there.). Each of the .so file for different option.
./foo -v needs libversion.so
./foo -update needs libupdate.so
I need the symbol for those libraries should be resolved only at the run time.
./foo -v should not break even if libupdate.so library is not there.
It is working in windows using the delay load option(in properties of dll). What is its equivalent in unix systems.
Will -lazy option does the same in UNIX? If so, where to include this option: in makefile or with linker ld?
See the reference on your system for dlopen(). You can manually open libraries and resolve external symbols at runtime rather than at link time.
Dug out an example:
int main(int argc, char **argv) {
void *handle=NULL;
double (*myfunc)(double);
char *err=NULL;
handle = dlopen ("/lib/libm.so.1", RTLD_LAZY);
if (!handle) {
err=dlerror();
perror(err);
exit(1);
}
myfunc = dlsym(handle, "sin");
if ((err = dlerror()) != NULL) {
perror(err);
exit(1);
}
printf("sin of 1 is:%f\n", (*myfunc)(1.));
dlclose(handle);
return 0;
}
I know it has been 8 years but still...
Delay load is not supported out out the box on GNU systems but you can mimic it yourself by creating a small static stub which provides all necessary symbols and dlopens real implementation on first call (or even at program startup). Such stub can be written by hand, generated by project-specific script or via Implib.so tool:
# Replace
$ gcc -o foo foo.c -lversion
# with
$ implib-gen.py libversion.so
$ gcc -o foo foo.c libversion.tramp.S libversion.init.c

Resources