I'm doing some C programming on a machine for which I don't have root access. I've compiled some shared libraries that I'm linking to, but because I cannot install the libraries in the typical location (/usr/local/lib) I have to explicitly specify the location of the libraries each time I compile and run. When compiling, this simply means adding the -L flag to the gcc command, but for program execution it's a lot more annoying. Either I must add the non-standard directory to LD_LIBRARY_PATH in each session, or I must add LD_PRELOAD=/path/to/libs to the beginning of the execute command.
Is there a better way to do this on a machine for which I don't have root access?
BTW, the machine is running Red Hat 4.1.
There are several solutions, from better to worse:
Use $ORIGIN, e.g. gcc main.o -L../lib -lfoo -Wl,-rpath='$ORIGIN'/../lib
Use target RPATH, e.g. gcc main.o -L../LIB -lfoo -Wl,-rpath=/home/user/lib
Set LD_LIBRARY_PATH from your .bashrc or .profile
Solution 1 allows you to install the binary anywhere, so long as you move the binary and the libraries together, e.g. my-app/bin/a.out and my-app/lib/{needed-shared-libs}.so. It also allows for multiple versions of the application and their set of shared libs.
Solution 2 works fine if you only need one set of shared libs, and never wish to move them around.
Solution 3 affects every application you run, and may cause some of them to bind to your shared libraries instead of the system ones. That may cause them to crash, fail with unresolved symbols, or cause you other pain. To exacerbate, the problem will only happen to you and nobody else, so you'll have hard time getting help for it.
You can add the environment variables to your .bashrc (or whatever file your shell sources when you log-in).
If you set the environment variable LD_RUN_PATH when you compile and link your program, then that search path will be baked in to the executable, and the dynamic linker will search it at runtime.
Using LD_LIBRARY_PATH or LD_PRELOAD is pretty much how to do this. To fix this, rename your program from myprog to myprog-exe, and create a shell script that looks like this called myprog:
#!/bin/sh
export LD_LIBRARY_PATH=/path/to/libs:$LD_LIBRARY_PATH
`dirname $0`/myprog-exe
This way, when someone runs myprog, it will really run the shell script which then runs myprog.
Related
My question is in relation to .so shared libraries. I am building a project that uses cmake on one ubuntu machine but running the application on another ubuntu machine.
In the CMakeLists.txt file, I have the following lines:
project (clientapp)
add_executable(${PROJECT_NAME} ${SOURCES} ${WAKAAMA_SOURCES} ${SHARED_SOURCES})
LINK_DIRECTORIES(/home/user//mraa-master-built/build/src)
target_link_libraries (clientapp libmraa.so)
target_link_libraries(clientapp m)
These lines add two libraries libmraa.so and the math library to the executable and it runs successfully on the other machine.
My understanding of shared libraries is that they must be present at compile time, and when the application starts. But I do not have the libmraa.so file on the other machine and the application runs ok. I expected it not to work.
Is my assumption correct?
In general, gcc and clang support lazy linking/binding of symbols, but not for entire libraries. This means that all of the shared objects (ie: .so files) should be present at application startup, at a minimum. The one exception to this is if you modified your makefile to not link against these libraries, and you manually call library functions via dlopen()/dlsym(), etc.
The binding of individual symbols within those libraries can be postponed until they are needed, or you can force all the symbols to be resolved at startup, using -z lazy or -z now, respectively.
It is strange that your application runs without libmraa.so being present. The two most likely reasons your application is running in the absence of the library is:
Your application isn't using any symbols defined in the library, so the linker ignores the library at build time (try ldd app_name and see if your library is present in the list of libraries provided by ldd).
Something is amiss in your build script, and you are statically linking against a .a archive of the library.
Edit: In response to how the application knows how to find the library, your linker (ld in this case) will use rpath lookup to decide which directories to use in its search for the appropriate library. You can see how this works by doing something like LD_DEBUG=libs app_name from the command line. You can also add an extra path via LD_LIBRARY_PATH=/some/path app_name.
Is my assumption correct?
Yes.
There are two likely explanations for why the application runs anyway:
You are mistaken, and there is libmraa.so somewhere on the machine (though perhaps not in the place where you looked), or
Your compiler defaults to -Wl,--as-needed by default, and your binary does not in fact depend on libmraa.so despite the fact that it appears on your link line.
You can trivially confirm or disprove either of the above guesses.
To confirm guess 2, do this:
readelf -d clientapp | grep NEED | grep libmraa
# if there is no output, guess 2 is correct
If guess 2 is wrong, to confirm guess 1, do this (on machine without libmrra.so):
ldd clientapp | grep libmraa.so
# if guess 2 is incorrect, and this command produces no output, then
# your dynamic loader is broken, which is very unlikely.
My issue started identical to this one: Python executable not finding libpython shared library
I updated .bashrc with export LD_LIBRARY_PATH=$HOME/local/lib/python/2.7.6/lib and things were fine. Python works, and I installed pip. But now, I'm running into something similar when installing cython with pip. I get this error message when I execute pip install cython:
gcc -pthread -shared build/temp.linux-x86_64-2.7/tmp/pip_build/cython/Cython/Plex/Scanners.o -L. -lpython2.7 -o build/lib.linux-x86_64-2.7/Cython/Plex/Scanners.so
/usr/bin/ld: cannot find -lpython2.7
collect2: ld returned 1 exit status
error: command 'gcc' failed with exit status 1
I cannot add $HOME/local/lib/python/2.7.6/lib to /etc/ld.so.conf and run ldconfig as I do not have root. I was under the impression that setting the LD_LIBRARY_PATH was the way around this, but this appears to not be true for compilation. Is there a way to get the compiler to see this local library without running root commands?
Update:
The LD_LIBRARY_PATH is only used by the dynamic loader at runtime, not at build time, so that is not the issue. The issue is that you forgot to put the -L/path/to/pylib before the -l. I've never had to use LIBRARY_PATH because a build requires path extension that is specific to a given build, so you never set LIBRARY_PATH you just use -L. You would only set if if you are going to regularly do builds that use a specific library, and even then I find it better to use -L because sooner or later this will cause linker to find the wrong lib and by then you will have forgotten that it's because LIBRARY_PATH is set permanently.
There are many ways to set -L values in a build: if you run the compiler from command line you don't need that env var, you just specify as many -L as required as part of the command; if you use a makefile, you edit whatever make variable you are using, such as CFLAGS or other, different platforms have different conventions. So whereas setting -L directly will always work, setting CFLAGS will only work if that is the variable used by the makefile.
Now this is a python installation so where to set this may not be obvious, but I am sure there is another way than setting LIBRARY_PATH. In principle any python package you install, if it involves compilation of C++ modules, could require edit of the setup.py to set library paths. For example
Extension(...,
library_dirs=['/usr/X11R6/lib'],
...)
Since you mention nympy, another place to set this might be in site.cfg (see Supplying NumPy site.cfg arguments to pip).
Old (wrong) answer:
Set your LD_LIBRARY_PATH in your bash console. If this doesn't work then it's because you have the wrong path: check by echoing the environment var.
Once you get that to work, edit your .bashrc or .profile then exit your shell and restart it. Echo the env var to verify that contains the part you added.
Also, ensure that you are appending to the path rather overwriting it:
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/...
export LD_LIBRARY_PATH
Because python lib might depend on .so in other folders, if the linker can't find them it may appear as though it is the python lib that was not found. This is not explained on the page you linked to in your question.
OK after some more digging I found this: LD_LIBRARY_PATH vs LIBRARY_PATH
Setting LIBRARY_PATH to the same path as LD_LIBRARY_PATH made the compiler aware of the python lib. cython/numpy/scipy all built and installed no problem afterwords.
I am using a program that has some shared library that are usually installed to /usr/lib.
However for some reason i have to have these libs locally. So to make my program run (which depends on the former) I need to export LD_LIBRARY_PATH= ... or add my local path permanently. This is ok for me but users of my software don't know this and for them this is too complicated. So my question: is there a way to automatically set the local path to my shared libs which are called by my program at runtime.
is there a way to automatically set the local path to my shared libs
Maybe.
If you are on Linux, and your application is installed together with shared library into say /some/prefix/bin/app and /some/prefix/lib/libsharedlib.so, then linking your application with:
gcc -o app -Wl,--rpath='$ORIGIN/../lib' main.o ... -lsharedlib
will achieve exactly the result you want (note: you can move both app and library into /another/dir, and it will still work so long as both the lib and bin directories are moved together).
Note: single quotes around $ORIGIN are required.
If you are on a platform that doesn't support $ORIGIN, the other common technique is to wrap the application in a shell script, which looks at $0, sets LD_LIBRARY_PATH appropriately, then execs the real application (which is often called app.bin, or app.exe).
I,m using gcc compiler(MinGW) on Windows XP.I created a .dll library libdir.dll than I tried to build a program that is using that library.
I don't want to put that .dll file into System or System32 folder nor to set path to it in PATH variable, what i want is to give that information to the program itself.
I know there is a -R and -rpath switches available so i was gonna link it with one of them.
First -rpath:
gcc -L/path/to/lib -Wl,-rpath,/path/to/lib main.o -ldir -o prog
Than -R:
gcc -L/path/to/lib -Wl,-R,/path/to/lib main.o -ldir -o prog
This links successfully into prog but when i start the program Windows prints message that it cannot find libdir.dll.
So my question is what went wrong, why path to libdir.dll is not known in runtime even when I'm using appropriate switches?
Let's say i have prog1 and prog2 each containing their own copy of libdir.dll and both of them start to run at the same time loading code in the library.What happens in memory is there a two copies loaded or linker figures out that there is a copy and uses that for both programs?
Second question is about how libraries are loaded(any OS).Does linkers always load entire library or just parts needed?For example if program references function foo() which is in the library, does linker maps into memory only that function or entire library first?
There are only two real alternatives: put the DLL in the same folder as the EXE or put it in the working directory for the EXE. The latter being not much of an option since you'd have to create a shortcut to make the default working directory different from the directory that contains the EXE.
Not putting the DLL in the same directory as the EXE only makes sense if you want to share the DLL with other applications. To avoid the inevitable DLL hell this causes, you'd need to store the DLL in the side-by-side cache. The tooling you need to create the manifest and embed it in the EXE and the installer you'd need to deploy the DLL to the target machine are probably hard to come by with your tool chain. It is very rarely done anyway.
Part of this question is a duplicate of this one: Is there a Windows/MSVC equivalent to the -rpath linker flag?
The summary of the answer is that there is no direct equivalent of RPATH on Windows.
Since you precluded placing your DLLs in the default library search path (which on Windows includes the system directories you listed and the directories in the PATH environment variable), you are left with these options:
using batch files
placing all the DLLs and executables in the same directory
making OS-level calls in your program for adding to the DLL search path
I have a collection of dynamic libraries that I want to link to in /usr/local/lib, how can I do this using gcc without setting my LD_LIBRARY_PATH (i have heard it is bad to do this fora number of reasons).
I have spent an hour looking at this on the internet, and fiddling with command line arguments, the only way I got it to work was using the -static -I/usr/local/lib/ flag, but this converts dynamic libraries to static libraries, and the compiler throws up some warnings.
Thanks for your help
Add /usr/local/lib to the loader configuration in /etc/ld.so.conf and run ldconfig.
You can set the system wide search directories for ldd (the dynamic linker) in /etc/ld.so.conf. In many distributions (well, mine) there is a /etc/ld.so.conf.d/ directory, from which the /etc/ld.so.conf includes all *.conf files. You can add the directory directly in ld.so.conf or add a .conf file in the directory.
Of course, you'll need root access to do this.
Oh, yeah: as Ignacio says, run ldconfig after changing these config files.