I have a simple common lisp server program, that uses the osicat library to interface with the posix filesystem. I need to do this because the system creates symbolic links to files, and uses the POSIX stat metadata, and neither of those things are straightforward to do in portable lisp.
I am managing the dependencies with quicklisp, and I have all of this pinned to a working distribution. The app is portable between CCL and SBCL, and I tend to build it in the first and deploy it using the latter. I declare the dependencies for the app with an asdf defsystem, and I can use quicklisp to load it for easy development from local projects.
For deployment I was just using some ansible playbooks that replicated a developer environment on a remote (.e. setting up quicklisp, pushing code into local projects, running out of a user home directory) which was hacky, but mostly ok. More recently, as it's becoming more stable I have been compiling it using sb-ext:save-lisp-and-die, using a simple compile script. This means I get an executable that I can run more like a server, with service management scripts, and an anonymous user account.
This has been working very well, and so I recently moved this step to the next level, and I'm building .deb packages with my compile script, so I can bundle up everything into a relocatable binary. This also sort of works, but the resultant binaries are not relocatable from the original build host. They refuse to start up, and it appears that they try to dynamically load a shared library for osicat
Unhandled SIMPLE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING
Mar 15 12:47:14 annie [479]: {10005C05B3}>:
Mar 15 12:47:14 annie [479]: Error opening shared object "libosicat.so":
Mar 15 12:47:14 annie [479]: libosicat.so: cannot open shared object file: No such file or directory.
it looks like the image expects to find this in the original build tree's quicklisp archives
(ERROR "Error opening ~:[runtime~;shared object ~:*~S~]:~% ~A." "/home/builder/buil...quicklisp/dists/quicklisp/software/osicat-20180228-git/posix/libosicat.so
(SB-SYS:DLOPEN-OR-LOSE #S(SB-ALIEN::SHARED-OBJECT :PATHNAME #P"
so poking around the source, I realise that when quicklisp fetches osicat and exercise its build operation, it compiles this DLL to wrap it's interface with the system libaries, rather than just ffi to them directly - possibly because it's using cffi groveller, I don't really know much about cffi (yet). This is fine, but rather than linking to a .so using the system linker it's trying to dlopen it from a fixed path, which is not very portable, and kind of breaks the usefulness of save-image
I'm a bit stumped at this point, but before I go diving any much further into QL and cffi builds, I wondered if there's some build or compile configuration I'm missing that would make it bootstrap in a more 'static' fashion or influence the production of the wrapped library. Ideally I just want a single blob I can wrap in an installer, and link it against system libraries, but if I have to deploy some additional artefacts that's probably alright. I don't know how to make the autogenerated shared objects occur at a more controlled path.
At that point though, I may as well write a .so for my posix calls and distribute this alongside the app and try and FFI to it more directly. That would be a bit of a pain, so I would prefer to not do this.
You're right, when a dumped image is starting up, it is trying to reload the shared libraries. Which, as you are experiencing, is not working if the image is not starting on the machine it was dumped on.
This is almost what static-program-op set out to solve. A simple system definition like this should help you compile a static program:
(defsystem "foo"
:defsystem-depends-on ("cffi-grovel")
:build-operation "static-program-op" ; "asdf" package is implied
:build-pathname "foo" ; path of the generated binary
:entry-point "foo:main" ; function to use as the entry point
;; ... everything else ...
)
If your system depends on grovel files (defined by :cffi-wrapper-file, :c-file or :o-file), such as the ones provided by osicat, then it will statically link those to your dumped image.
However, it's not perfect.
Essentially, there are still some issues. Some are being fixed upstream by CFFI itself (e.g. not reloading shared libraries of the statically embedded libraries), others are a bit harder. (E.g. SBCL default compilation options don't let you use static-program-op by default. This is being fixed in Debian builds of SBCL, but other distributions are being less responsive.)
This is obviously a problem that the community at large has met, and there are several libraries that are around to help:
The first one, that has been around for a while, is Deploy. The approach it takes is that it embeds the dumped image and the libraries into an archive, and rearranges things for the binary to load them from wherever it is extracted to.
The second one, which I am biased towards because I made it, is linux-packaging. It takes the approach of fixing static-program-op by extending it, but requires you to build a custom SBCL. However, it generates distribution packages like .deb and .rpm, in order to be able to specify dependencies for system shared libraries (e.g. if you depend on sqlite, it will figure out which package provides it and add it as a dependency in the .deb). I highly recommend looking at the .gitlab-ci.yml for examples.
I recommend reading the webpages of both of those libraries to make your choice, they both have their advantages and drawbacks. <joke>Obviously, linux-packaging is superior.</joke>
Maybe you can use sb-posix:symlink and sb-posix:fstat on SBCL instead, removing the osicat dependency by feature toggle.
Related
I have a C-based DLL that I wrote years ago for a project and it exports a set of functions that define an API. Now I need to re-write this DLL's internals but keep the API exactly the same.
The user of the DLL used static linking and they do not want to or are unable to recompile their executable.
I've noticed that the RVAs of the exported functions are different. My understanding is that means the executable won't be able to find the functions unless it is re-linked with the updated lib file.
Is there a way in VS2017 to force an exported function to use a specific RVA? I checked the Microsoft LINK DEF file format and I didn't see an option in there.
Even if it is possible, is fixing the RVAs enough to ensure the old executable will be able to use the updated DLL or are there additional complications that make this a non-starter?
Thanks.
When you statically link an EXE module against a DLL, you do indeed link against the the DLL's import library (a .LIB) created alongside the DLL when the DLL was built. This is not the same thing as linking against a static library which is confusing because those are also .LIB files.
The first thing you should do is figure out if your EXE module has an import entry for said DLL using a tool like Dependency Walker, Dumpbin, pelook or your favorite PE analyzer tool. If there is no DLL import entry, you have have probably linked the EXE against a static library as described by #HAL9000 's answer. Short of reverse-engineering the EXE, your best bet would be to rebuild the module as suggested if possible.
Otherwise, if you do find an import for said DLL, then yes you can swap out a newly-built DLL provided you have the same export (function) names and/or ordinal values as the original. DLLs find function by export names and/or ordinal values, not RVAs which in this case are only an internal detail. Whether the DLL is implicitly loaded (from being statically-linked) during process (EXE) initialization (before the EXE's entry point is called) or explicitly loaded (via code using LoadLibrary, etc.) the whole point of being a DLL is that it is a module is designed to be dynamically replaced and Windows was designed around this concept. The internal RVAs both within the EXE (referencing the DLL) and the DLL itself do not need to match an old DLL's values; this bookkeeping is automatically handled by the Windows loader during a process also known as runtime linking.
In the event the EXE is linked against said DLL and ALSO specifies hard-coded addresses (RVAs) for the DLL's exported functions (a process known as static binding), Windows will still verify the addresses still internally reflect the correct values in the DLL that is actually loaded which may be a different, updated DLL. This is done via a timestamp check in the import section for the DLL. If there is a mismatch, the Windows loader tosses-out all of the static RVAs and updates them with the current values incurring a slight performance penalty, but the program will still load. FWIW the bind.exe tool to do this static binding no longer ships with the Visual C++ toolset as the performance gain in modern versions of Windows is minimal. This optimization used to be be common practice to speed up load times, especially in OS-supplied system DLLs, but shouldn't affect what you are trying to do one way or the other.
If the user has statically linked in your library, then it is not a DLL, and making a drop in replacement without relinking is not possible. At least not without some ugly hacks. The old library functions have been copied into the executable, so there are no way around editing the executable. If you can't recompile or relink, then it is probably easier to rewrite the executable from scratch.
Mucking around with adresses of functions in your new DLL, if possible, can't have any effect if the executable doesn't have any code to load a DLL at all.
On our RHEL 6.6 machines we have the following two packages installed
hdf5-1.8.5.patch1-7.el6.x86_64 (provides /usr/lib64/libhdf5*)
hdf5-openmpi-1.8.5.patch1-7.el6.x86_64 (provides /usr/lib64/openmpi/lib/libhdf5*)
These seemingly provide what I would think are duplicate libraries (i.e. libhdf5.so.6.0.4), but doing an md5sum reveals that they are not identical.
1) Is this a bad practice / actual problem? One of our users claims having such duplicate libraries creates a dependency nightmare for him.
2) Assuming it is a problem, how do we "fix" it? Removing one or the other might break things for other people who are depending on the package we delete.
It shouldn't be a problem. If you are writing parallel code, you link to the parallel/OpenMPI version.
This Fedora page notes that they are built from the same source, so that strongly implies they have been tested in the configuration presented and shouldn't conflict.
These are not duplicate libraries and it is neither a bad practice nor an actual problem. HDF5 could be built with or without support for MPI. When built with MPI support, the HDF5 library can only be linked with applications that are also built against the same MPI library. That's why there are separate HDF5 packages:
hdf5-1.8.5 - non-MPI-enabled build to be used within non-MPI applications
hdf5-openmpi-1.8.5 - MPI-enabled build, uses Open MPI
hdf5-mpich-1.8.5 - MPI-enabled build, uses MPICH
The actual shared objects are installed in different places so that they could co-exist on the same system.
I am currently looking for ways to expose the location of a shared library on Linux such that it can be picked up easily by any program installed separately. I want to make this location configurable so it can point to different possible installations of the same library. Examples of similar cases I can think of would be Qt5 and Java.
To make a long story short, I am developing FreeRDS, a FreeRDP-based Remote Desktop Services stack. Server-side RDS-aware applications link to libwinpr-wtsapi, a stub library that exposes the Microsoft Windows Terminal Services API interface, but does not implement it. This enables applications to link to libwinpr-wtsapi without having to link directly to a specific RDS implementation. On the first call to any of the WTSAPI functions, the real implementation is loaded dynamically by libwinpr-wtsapi. However, the location of the dynamic library implementing the WTSAPI (here, FreeRDS) needs to be known.
Right now, I am achieving this by setting an environment variable with the full path to the library:
export WTSAPI_LIBRARY=/opt/freerds/lib/x86_64-linux-gnu/libfreerds-fdsapi.so
However, this is not very practical, as this environment variable would need to be set for every program using the WTSAPI. In this case, I have my installation of FreeRDS in /opt/freerds.
I am thinking I could probably simplify this by using a single environment variable to expose the installation prefix of FreeRDS on the system, with something similar to JAVA_HOME:
export FREERDS_HOME=/opt/freerds
However, I then need to know the proper library subdirectory. It is also important to know that it would be possible in the future to offer both a 32-bit and a 64-bit version of the library offering the FreeRDS WTSAPI. This library basically performs RPC with the FreeRDS session manager, so that would be definitely possible.
Let's say we have FREERDS_HOME properly set, or that FreeRDS is installed in the default installation prefix of the system, which files would be "standard" to offer some additional installation configuration information? Here I'm thinking I could have an equivalent of Qt5's qt.conf that would specific installation subdirectories, like the 64-bit installation subdir, the 32-bit installation subdir, etc. However, I don't know where I should be putting that file. Should it be in <prefix>/etc/freerds/freerds.conf?
Ideas, anyone? Thank you!
some (many? all?) Linux distributions today include environment-modules, which aim is exactly to make available many different versions of the same software by customizing the environment (and eventually, shell aliases/functions) with easy front-end commands.
You can find all the needed information here.
Thanks for the multiple answers, here is the solution I finally opted for that satisfies my needs:
As explained earlier, there could be more than one installation of FreeRDS on the same system, but only one of them running at once. We can also assume FreeRDS is supposed to be running before we can attempt to interact with it. Knowing this, I modified FreeRDS to write a simple configuration file in /var/run/freerds.instance with the install prefix and installation subdirectories. This is very similar to having a .pid file, except we're exposing installation paths.
The freerds.instance file is using the .ini format, which is fairly common in configuration files. All that libwinpr-wtsapi has to do is parse /var/run/freerds.instance to find the installation prefix of the current FreeRDS instance, along with the library subdir, so we can find the correct libfreerds-fdsapi.so.
Here is what a sample freerds.instance file looks like:
[FreeRDS]
prefix="/opt/freerds"
bindir="bin"
sbindir="sbin"
libdir="lib/x86_64-linux-gnu"
datarootdir="share"
localstatedir="var"
sysconfdir="etc"
I prefer this solution because it requires literally no special configuration, setting of environment variables, etc. No matter what, we always find the proper FreeRDS installation wherever it is on the system.
You can add a $ORIGIN rpath to your executable, that makes it load libraries relative to the directory the executable is in. (See "ld: Using -rpath,$ORIGIN inside a shared library (recursive)"). This probably applies to dlopen() too.?
$ gcc ... -Wl,-rpath,'$ORIGIN/../lib/dir' -lsomething
I've also found you can run the dynamic linker directly to get some debug facility:
$ /lib/ld-linux.so.2
Usage: ld.so [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...]
...
--list list all dependencies and how they are resolved
export LD_LIBRARY_PATH=/yourso.so
We have a project that is going to require linking against libcurl and libxml2, among other libraries. We seem to have essentially two strategies for managing these depencies:
Ask each developer to install those libraries under the "usual" locations, e.g. /usr/lib, or
Include the sources to these libraries under a dedicated folder in the project's source tree.
Approach 1 requires everyone to make sure those libraries are installed on their system, but appears to be the approach used by many open source projects. On such projects, the build will detect that those libraries are missing and will fail.
Approach 2 might make the project tree unmanageably large in some instances and make the compilation time much longer. In addition, this approach can obviously be taken too far. I wouldn't put the compiler under the project tree, for instance (right?).
What are the best practices wrt external dependencies? Can/should one require of every developer to have certain libraries installed to build the project? Or is considered better to include all the dependencies in the project tree?
Don't bother about their exact location in your code. Locating them should be handled by the used compiler/linker (or the user by setting variables) if they're common. For very uncommon dependencies (or ones with customized/modified files) you might want to include them in your source (if possible due to licensing etc.).
If you'd like it more convenient, you should use some script (e.g. configure or CMake) to setup/create the build files being used. CMake for example allows you to set different packages (libcurl and libxml2 in your example) as optional and required. When building the project it will try to locate those, if that fails it will ask the user. This IS an additional step and might make building a bit more cumbersome but it will also make downloading faster (smaller source) as well as updating easier (as all you have to do is rebuild your program).
So in general I'd follow approach 1, if there's special/rare/customized stuff being used, approach 2.
The normal way is to have the respective dependencies and have the developer install them. Later, if the project is packeted into .deb or .rpm, these packets will require the respective libraries to be installed, the source packets will have the -devel packets as dependencies.
Best practice is not to include the external libraries in your source tree - instead, include a text file called INSTALL in your project root, which gives instructions on building the project and includes a list of the library dependencies, including minimum versions.
The term has several definition according to Wikipedia, however what I'm really interested in is creating a program that has all its needed dependencies included within the source folder, so the end user doesn't need to install additional libraries for the app to install. For example, how Mac apps has all its dependencies all within the program itself already...
or is there a function that autotools does this? I'm programming in the Linux environment...
Are you talking about the source code of your application, or about your application binary?
The answer I'd give for both the cases depends on what libraries you're using.
If you're using libraries that you can find anywhere, that are somehow standard and/or that are quite big, you shouldn't attach them to your application, just require them both to build and to run your application.
Anyway don't be much concerned about your source code: little people will build your application, and they probably know something about programming and how a Linux system works; it won't be a big deal to require many (also not-so-common) dependences to build your application.
For what concerns the binary version it could be a little more problematic, since it will be used by end users who often don't know anything about libraries and programming stuff: you could choose to statically link the smallest and most uncommon libraries to your binary, in order to have less dependences.
You could do it, if you link statically, but it'd be somewhat unusual, and depending on what your program is supposed to do, you might be limiting yourself.
The alternative, if this is not just a one-off project, is to create a Linux Standard Base compatible RPM package and restrict yourself to linking against the libraries and symbols that LSB defines.
Run ldd on your program to discover all dependencies, then copy these to your directory, and add a program-wrapper script that issues
#!/bin/sh
LD_LIBRARY_PATH="${0##*/}:$LD_LIBRARY_PATH" exec "${0##*/}/real-program" "$#";
Duplicating the Mac OS X .app behavior on a plain POSIX system is difficult because it is very hard to guarantee that a process can find it's own executable (there are several way that will almost always work...). Mac OS X provides a OS service for this, but Linux (for instance) does not.
Once you've accomplished that feat, this becomes possible. Though, as others have mentioned, it loses the ability to share resource demands (disk space, RAM space, cache space) with other programs that use the same libraries because you'd be using static copies, or dynamically loading your own copy from the .app-like bundle.