So I am in a bit of a pickle here and am at my wits end.
Issue:
I have made some lua C module that target another package's C++ shared library. Attempting to require the Lua module fails due to a dependency issue. I can find the module shared library just fine but when the linker attempts to find the library I need to link against, I get an error that looks like:
/usr/bin/lua: error loading module 'mytest' from file '/usr/lib/lua/5.3/mytest.so':
Error loading shared library libmy_test.so: No such file or directory (needed by /usr/lib/lua/5.3/mytest.so)
The directory structure used for linking is
|-usr
|--lib
|---lua
|----5.3
|-----mytest.so
|-mockfs
|--usr
|---lib
|----libmy_test.so
My LUA_CPATH is:
/usr/lib/lua/5.3/?.so;/mockfs/usr/lib/?.so;
My LD_LIBRARY_PATH is:
/mockfs/usr/lib/
What is driving me mad is that when the libmy_test.so is in the same directory as mytest.so, there are no problems. But due to other restrictions I cannot modify the path of libmy_test.so or add a symlink without substantial refactoring and lag time for builds/regressions. I'm unsure of where to even continue here because it is definitely a "searching for the library" problem but I don't know how to get insight into what is going on under the covers to fix the issue.
Adding /mockfs/usr/lib/?.so to LUA_CPATH is not going to help, as it only works for Lua modules loaded with require (which already works in your case for mytest.so) and doesn't work for loading their dependencies (which should be handled using dynamic linking).
It's not clear if LD_LIBRARY_PATH is even used in your case; I'd set LD_DEBUG=all to get LD troubleshooting information, which should tell you what is or is not being loaded and why.
Also, even if you can't put libmy_test.so where mytest.so is located, you can still put mytest.so where libmy_test.so is and configure LUA_CPATH like you have to try making it work that way.
So this will be unrelated to other people's Lua code. This problem description is slightly abstracted from reality as there is a lot of internal tooling from my company in front of it. I followed Paul's answer to try and get more information, LD_DEBUG was working up until I called my Lua script. Then the linker would stop giving me any information even though the require was still failing with the same error.
After a LOT of digging and my personal discovery of strace I managed to identify that the linker being used when calling require was in fact using "LD_PATH" to search for libraries instead of "LD_LIBRARY_PATH". Why this was? I still have no idea other than maybe an intentional design decision to combat some of the limitations of our internal tooling when doing cross compilation.
For anyone who experiences linker issues with Lua, LD_DEBUG and strace are very nice utilities for running down your issues.
Related
Configuration
I use autotools (autoreconf -iv and ./configure) to generate correct makefiles. On my development machine (Fedora) everything works correctly. For make check I use the library libcheck and from autotools I use Libtools. On Fedora the library for check is dynamic: libcheck.so.0.0.0 or something such. It works.
The Issue
When I push the commits to my repo on github and do a pull request, the result is tested on Travis CI which uses Ubuntu as a platform. Now on Ubuntu the libcheck is a static library: libcheck.a and a libcheck_pic.a.
When Travis does a make check I get the following error message:
/usr/bin/ld: /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../..
/../libcheck.a(check.o): relocation R_X86_64_32 against.rodata.str1.1' can
not be used when making a shared object; recompile with -fPIC`
/usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../libcheck.a: could not read
symbols: Bad value
Which means I have to somehow let configure determine what library I need. I suspect I need libcheck_pic.a for Ubuntu and the regular libcheck.so for Fedora.
The question
Does anyone know how to integrate this into configure.ac and test/Makefile.am using libtool? I would prefer to stay in line with autotools way of life.
I couldn't find usable information using google, but there is a lot of questions out there about what the difference is between static and dynamic (which is not what I need).
I would much appreciate if anyone could point me in the right direction, or maybe even solved this already?
I suspect you're right that the library you want to use on the CI system is libcheck_pic.a, for its name suggests that the routines within are compiled as position-independent code, exactly as the error message you receive suggests that you do.
One way to approach the problem, then, would be to use libcheck_pic if it is available, and to fall back to plain libcheck otherwise. That's not too hard to configure your Autotools-based build system to do. You then record the appropriate library name in an output variable, and use that in your (auto)make files.
Autoconf's SEARCH_LIBS macro specifically serves this kind of prioritized library search requirement, but it has the side effect, likely unwanted in this case, of modifying the LIBS variable. You can nevertheless make it work. Something like this might do it, for example:
LIBS_save=$LIBS
AC_SEARCH_LIBS([ck_assert], [check_pic check], [
# Optional: add a test to verify that the chosen lib really provides PIC code.
# Set LIBCHECK to the initial substring of $LIBS up to but excluding the first space.
LIBCHECK=${LIBS%% *}
], [
# or maybe make it a warning, and disable your test suite when libcheck
# is not available.
AC_MSG_ERROR([A PIC version of libcheck is required])
])
AC_OUTPUT([LIBCHECK])
LIBS=$LIBS_save
I presume you know what to do with $(LIBCHECK) on the Make side.
As written, that has the limitation that if there is no PIC version of libcheck available then you won't find out until make or maybe make check. That's undesirable, and you could add Autoconf code to detect that situation if it is undesirable enough.
As an altogether different approach, you could consider building your tests statically (add -static to the appropriate *_LDFLAGS variable). Of course, this has the opposite problem: if a static version of the library is not available, then the build or tests fail. Also, it requires building a static version of your own code if you're not doing so already, and furthermore, it is the static version that will be tested.
For greatest flexibility, you could consider combining those two approaches. You might set up a fallback from one to the other, or you might set up separate targets for testing static code and PIC code, and exercise whichever of them (possibly both) is supported by the libraries available on the build system.
Though this is my first question on the StackOverflow, I consider myself a long time member of the community.
Considering myself as a pre-intermediate programmer, I'll try to be as specific as possible.
I'm writing a Python package that uses a C dll to load image files with ctypes. I'm using Python 3.3.5 32bits version on Windows 8 x64. I had to build the dll from the C code. The dll is stb_image.h which is available in GitHub. I used Code::Blocks version 13.12 with the GCC compiler. The dll seems to have built fine.
By my definition, the dll is must be in the same folder as the py code file that wraps its functions, but when ctypes attempts to load the dll an exception is raised: the [in]famous WindowsError: [Error 126] Module not found.
I've found several similar questions and attempted to glean from their solutions by
adding the dll path via os.environ['PATH']
adding the dll path manually in the Path environment variable
changing the current working directory with os.chdir()
using the file module attribute to locate the library
none of these solved the issue
The stb_image library itself has no dependencies so I don't understand why windows can't find it since the dll is where it should be and the path addresses it's exact location. And, of course, there's a lot of other ways to get the job done: PyGame, pySFML (which also uses stb_image), PIL, PyPng, PySDL, you name it, but I'm doing this mostly for learning purposes.
At the writing of this question, something popped up in my head (which may or may not have any relation with the problem): whenever I compile and link programs with GCC and try to run them, I got a message box telling me that some dll (libgcc.dll or something like that) was not found. Could that be the reason Windows can't load stb_image.dll?
I tried everything my experience allowed me to do to solve it but it was not enough. Would you guys, please, give me a light on this?
PS.: Sorry for any bad english. I'm natural from Brazil.
I've been trying to compile balsa the email reader on ubuntu (i'm new to the whole process of compiling an open source project directly from sources.
so i did the whole configure and make but then I ran into this problem:
error: unknown type name 'gpgme_decrypt_result_t
after doing a little research on the error code.. it turns out that it belongs to gpgme a scary looking encryption library..
what do I do in this case? i hope i'm not expected to go in the code and debug stuff am i?
update:
this question is related to this other question here on the ubuntu forum
Unfortunately, you may be required to delve into the code to figure out why something doesn't build!
In this case however the answer is probably quite simple. The main reasons I would expect a symbol from an external library to not have been defined are:
The library header files do not exist on your system
You have the wrong version of the library header files on your system
The headers exist, but the compiler is not looking in the correct place
Clearly the solutions are similar: you will need to install the correct version of gpgme's header files for this software in the right place. Note that having the library itself installed is not the same as having the headers. In Ubuntu this is evident due to the presence of -dev packages which are the package's header files, and if you look through the repose you will find libgpgme11-dev. I suggest you start by installing this, which will eliminate point 1 above. If it still doesn't work, you'll need to start investigating the other two possibilities.
I'm getting crazy while trying to understand how to produce a binary in OCaml that doesn't need anything too specific (eg. having anything as OCaml build tools installed).
I'm using OMake and everything seems enough cool while working on my machine, also linking libs with
OCAMLPACKS[] = str unix batteries
works quite well, but if I try to distribute the executable it gives me the following error:
Fatal error: exception Sys_error("/usr/godi/share/camomile/database/general_category.mar: No such file or directory")
yes I'm using GODI but I don't see any apparent reason for this error, can anyone enlighten me? In addition, if you have any tricks regarding building stand-alone applications please tell me (I'm already arguing with GODI that compiled everylib just in its x64 version and now I'm unable to produce x86 executables)
Thanks in advance
The camomile library requires runtime configuration. See README.txt for informations about how to set it up.
In particular, it requires runtime files (the missing .mar you see) to correctly work.
I don't use OMake, neither batteries, neither GODI nor Camomile. But from what I see the problem seems to come from the fact that you link against Batteries which I guess must use Camomile internally and Camomile relies on data stored in the file system to provide some of its service. It fails to load if that data is not present on the final host.
Note that shipping purely static excutables is something that is fairly simple to do with OCaml. But for that you need to consider which libraries you use (e.g. if you use camomille you will have to ship the support files with).
I would like to have control over the type of the libraries that get found/linked with my binaries in CMake. The final goal is, to generate binaries "as static as possible" that is to link statically against every library that does have a static version available. This is important as would enable portability of binaries across different systems during testing.
ATM this seems to be quite difficult to achieve as the FindXXX.cmake packages, or more precisely the find_library command always picks up the dynamic libraries whenever both static and dynamic are available.
Tips on how to implement this functionality - preferably in an elegant way - would be very welcome!
I did some investigation and although I could not find a satisfying solution to the problem, I did find a half-solution.
The problem of static builds boils down to 3 things:
Building and linking the project's internal libraries.
Pretty simple, one just has to flip the BUILD_SHARED_LIBS switch OFF.
Finding static versions of external libraries.
The only way seems to be setting CMAKE_FIND_LIBRARY_SUFFIXES to contain the desired file suffix(es) (it's a priority list).
This solution is quite a "dirty" one and very much against CMake's cross-platform aspirations. IMHO this should be handled behind the scenes by CMake, but as far as I understood, because of the ".lib" confusion on Windows, it seems that the CMake developers prefer the current implementation.
Linking statically against system libraries.
CMake provides an option LINK_SEARCH_END_STATIC which based on the documentation: "End a link line such that static system libraries are used."
One would think, this is it, the problem is solved. However, it seems that the current implementation is not up to the task. If the option is turned on, CMake generates a implicit linker call with an argument list that ends with the options passed to the linker, including -Wl,-Bstatic. However, this is not enough. Only instructing the linker to link statically results in an error, in my case: /usr/bin/ld: cannot find -lgcc_s. What is missing is telling gcc as well that we need static linking through the -static argument which is not generated to the linker call by CMake. I think this is a bug, but I haven't managed to get a confirmation from the developers yet.
Finally, I think all this could and should be done by CMake behind the scenes, after all it's not so complicated, except that it's impossible on Windows - if that count as complicated...
A well made FindXXX.cmake file will include something for this. If you look in FindBoost.cmake, you can set the Boost_USE_STATIC_LIBS variable to control whether or not it finds static or shared libraries. Unfortunately, a majority of packages do not implement this.
If a module uses the find_library command (most do), then you can change CMake's behavior through CMAKE_FIND_LIBRARY_SUFFIXES variable. Here's the relevant CMake code from FindBoost.cmake to use this:
IF(WIN32)
SET(CMAKE_FIND_LIBRARY_SUFFIXES .lib .a ${CMAKE_FIND_LIBRARY_SUFFIXES})
ELSE(WIN32)
SET(CMAKE_FIND_LIBRARY_SUFFIXES .a ${CMAKE_FIND_LIBRARY_SUFFIXES})
ENDIF(WIN32)
You can either put this before calling find_package, or, better, you can modify the .cmake files themselves and contribute back to the community.
For the .cmake files I use in my project, I keep all of them in their own folder within source control. I did this because I found that having the correct .cmake file for some libraries was inconsistent and keeping my own copy allowed me to make modifications and ensure that everyone who checked out the code would have the same build system files.