Must R Packages Unload Dynamic Libraries When They Unload? - c

From Hadley's C best practices:
Like with C++, whenever you use C code in your package, you should unload the DLL when the package is unloaded:
.onUnload <- function (libpath) {
library.dynam.unload("mypackage", libpath)
}
Writing R Extensions on the other hand doesn't even mention this. I can see how it would be polite to unload the dlls, but doing so seems to cause some weird problems for me with packages that are loaded/unloaded/reloaded (see example further down). Additionally, there are some mentions that suggest maybe unloading isn't required. From ?library.dynam:
Note that whether or not it is possible to unload a DLL and then reload a revised version of the same file is OS-dependent: see the ‘Value’ section of the help for dyn.unload.
though this shouldn't affect objects that are not modified. Then there is this comment from Brian Ripley in R-devel:
Having said all that, my experience is that unloading the DLL often does not help if you need to load it again (and that is why e.g. tcltk does not unload its DLL).
So is it acceptable to leave the C libraries loaded? I would prefer not to have to dig into why stuff like the below is happening (did not happen before I started unloading libraries).
R version 3.1.1 (2014-07-10)
Platform: x86_64-apple-darwin13.1.0 (64-bit)
> library(alike) # install_github("brodieg/alike", ref="fdaa578e"), if you're curious
> library(data.table)
data.table 1.9.2 For help type: help("data.table")
> detach("package:data.table", unload=T)
> detach("package:alike", unload=T)
> library(alike)
> library(data.table)
Error : .onLoad failed in loadNamespace() for 'data.table', details:
call: address(x)
error: object 'Caddress' not found
In addition: Warning messages:
1: In FUN(X[[9L]], ...) :
failed to assign RegisteredNativeSymbol for alike to alike since alike is already defined in the ‘data.table’ namespace
2: In FUN(X[[9L]], ...) :
failed to assign RegisteredNativeSymbol for typeof to typeof since typeof is already defined in the ‘data.table’ namespace
3: In FUN(X[[9L]], ...) :
failed to assign RegisteredNativeSymbol for type_alike to type_alike since type_alike is already defined in the ‘data.table’ namespace
Error: package or namespace load failed for ‘data.table’
The warnings are all related to alike functions. alike did not use to unload its dynamic libraries, and the above errors did not happen. After I implemented unloading the errors started happening. Note that data.table 1.9.2 did not unload its DLLs, though other packages that also don't unload DLLs didn't cause this problems. data.table 1.9.4 works fine.

Normally unloading a DLL would be a good idea. The resources it owns, would be completely freed, and re-loading would not be an issue.
In R, there is the complication of the R environment, because even if a DLL is unloaded, there may be some knowledge left behind in the R runtime. In this case, the result may be that the re-loaded DLL library does not share the same inferred state as the R variables which are intended to understand the DLL state, and thus errors occur.
I think it would be possible for an R package (DLL and R code) to be safely unloaded, but it would be easier for you to leave the DLLs loaded, unless you find particularly heavy resource usage.

Related

Correct way to use the iterate package in Common Lisp

On my Windows XP box with sbcl-1.4.14 I've installed the ASDF using
(load "C:\\Program Files\\clisp-2.49\\asdf\\asdf.lisp")
(require :asdf)
(push "C:\\Documents and Settings\\mayhem\\lisp\\iterate\\" asdf:*central-registry*)
On SLIME
(require :iterate)
(iterate (for i from 1 to 5) (collect (* i i)))
gives The variable I is unbound error
If I do (in-package :iterate), the code above works fine but this time familiar functions such as exit and other functions which I've defined in .sbclrc cease to work, they give The function ITERATE::EXIT is undefined type of errors, for example.
If I do (use-package :iterate), then it gives [Condition of type NAME-CONFLICT] error.
So I began to use the package like this:
(iterate:iterate (iterate:for i from 1 to 5) (iterate:collect (* i i)))
But I think you'll agree that it's a bad style.
How to use the iterate correctly?
Note: I've seen the post about the very similar problem but it didn't help. There aren't many posts or articles about this particular problem.
You need to say (use-package :iterate) before you try to refer to unqualified symbols from the iterate package.
What has happened in your case is this.
You've loaded the iterate system into the running Lisp, creating a package called "ITERATE".
In the "CL-USER" package you've typed (iterate ...), and the reader has worked out that it needs to find or create a symbol whose name is "ITERATE" and which is accessible in the "CL-USER" package.
There is no such symbol, so it creates a new one, CL-USER::ITERATE.
This is not ITERATE:ITERATE so you get an error from the evaluator as it's trying to evaluate the arguments to a function (which doesn't exist, but it doesn't know that yet). In fact the error you're getting is while it's evaluating the first argument in the (for i ...) subform.
Now you say (use-package :iterate) to tell the system to add the "ITERATE" package to "CL-USER"'s search list.
Now there's a conflict: should iterate refer to the existing CL-USER::ITERATE or the newly-accessible ITERATE::ITERATE? (And there are some other conflicts too, probably).
So the system signals an error, and there should be some useful ways to proceed from that, one of which is probably 'unintern all the conflicting "CL-USER" symbols', but you didn't take that option, I suppose.
So now everything is messed up.
And the answer is: use the packages you want to refer to unqualified symbols from before you try to refer to those symbols unqualified.
(Also: Windows XP? I'm impressed by your retroness.)

Shows Missing library : ff while building echidna using stack install

I am trying to build echidna using stack install.
https://github.com/crytic/echidna
I am continuously getting the missing library error while installing at a progress state of 171/178.
I believe it to be more of stack error than library error. Error shown is as follows
I have tried installing the library but not getting any C library of such sort.ye
SO, i've installed every library using
sudo apt-get install lib*ff*-dev
isn't working yet.
-- While building package hevm-0.31 using:
/home/aman/.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_2.2.0.1_ghc-8.4.3 --builddir=.stack-work/dist/x86_64-linux/Cabal-2.2.0.1 configure --with-ghc=/home/aman/.stack/programs/x86_64-linux/ghc-8.4.3/bin/ghc --with-ghc-pkg=/home/aman/.stack/programs/x86_64-linux/ghc-8.4.3/bin/ghc-pkg --user --package-db=clear --package-db=global --package-db=/home/aman/.stack/snapshots/x86_64-linux/lts-12.10/8.4.3/pkgdb --package-db=/home/aman/Desktop/new/echidna2/echidna/.stack-work/install/x86_64-linux/lts-12.10/8.4.3/pkgdb --libdir=/home/aman/Desktop/new/echidna2/echidna/.stack-work/install/x86_64-linux/lts-12.10/8.4.3/lib --bindir=/home/aman/Desktop/new/echidna2/echidna/.stack-work/install/x86_64-linux/lts-12.10/8.4.3/bin --datadir=/home/aman/Desktop/new/echidna2/echidna/.stack-work/install/x86_64-linux/lts-12.10/8.4.3/share --libexecdir=/home/aman/Desktop/new/echidna2/echidna/.stack-work/install/x86_64-linux/lts-12.10/8.4.3/libexec --sysconfdir=/home/aman/Desktop/new/echidna2/echidna/.stack-work/install/x86_64-linux/lts-12.10/8.4.3/etc --docdir=/home/aman/Desktop/new/echidna2/echidna/.stack-work/install/x86_64-linux/lts-12.10/8.4.3/doc/hevm-0.31 --htmldir=/home/aman/Desktop/new/echidna2/echidna/.stack-work/install/x86_64-linux/lts-12.10/8.4.3/doc/hevm-0.31 --haddockdir=/home/aman/Desktop/new/echidna2/echidna/.stack-work/install/x86_64-linux/lts-12.10/8.4.3/doc/hevm-0.31 --dependency=QuickCheck=QuickCheck-2.11.3-3XHQpBYb83U2mMvNz2AjQX --dependency=abstract-par=abstract-par-0.3.3-Ie3MD7O3orK6ZR8i7FuBEv --dependency=aeson=aeson-1.3.1.1-7JlrwYHW7OR9ca4RRZ9oOf --dependency=ansi-wl-pprint=ansi-wl-pprint-0.6.8.2-9fh9v74MJQDJeSHE7X3Co6 --dependency=async=async-2.2.1-7DQr5PBcpiwJNOuf7ZiSPa --dependency=base=base-4.11.1.0 --dependency=base16-bytestring=base16-bytestring-0.1.1.6-I0igvRcEwRNBMqqPC2yQBh --dependency=base64-bytestring=base64-bytestring-1.0.0.1-4OCIe2BZn8jKI191JIXI37 --dependency=binary=binary-0.8.5.1 --dependency=brick=brick-0.37.2-HmDqAExuwtV4o98FRmS9eK --dependency=bytestring=bytestring-0.10.8.2 --dependency=cereal=cereal-0.5.7.0-ILaYAmVTyR1IcEsGXXUCfI --dependency=containers=containers-0.5.11.0 --dependency=cryptonite=cryptonite-0.25-GgyZs9E1viv2owjaLxA3vq --dependency=data-dword=data-dword-0.3.1.2-CMzWV6RCGtK8L6wsVSErKS --dependency=deepseq=deepseq-1.4.3.0 --dependency=directory=directory-1.3.1.5 --dependency=fgl=fgl-5.6.0.0-E3fRSNebqkULRqplV3ljSc --dependency=filepath=filepath-1.4.2 --dependency=ghci-pretty=ghci-pretty-0.0.2-I5PJbL8ge6MChp3KxZbHqH --dependency=haskeline=haskeline-0.7.4.3-5EI2v7Zmtnz57a7mmnEGOS --dependency=lens=lens-4.16.1-B0f4CyKKTUGAKBzzY965AX --dependency=lens-aeson=lens-aeson-1.0.2-Jly9eqrxMbj6GutGwOKn69 --dependency=megaparsec=megaparsec-6.5.0-G48ltiRFbVUHATB1lTnNRx --dependency=memory=memory-0.14.16-GTCi0eCrvrnI3inLDBWVMK --dependency=monad-par=monad-par-0.3.4.8-Jjk0JT5qYVK4xEO13GFUu8 --dependency=mtl=mtl-2.2.2 --dependency=multiset=multiset-0.3.4.1-DOPR5uXspss8vZY4jdOdOo --dependency=operational=operational-0.2.3.5-4jEcCWo4nKu1T4LZlKqe58 --dependency=optparse-generic=optparse-generic-1.3.0-1jcIewFKPXBLcRetZisV2l --dependency=process=process-1.6.3.0 --dependency=quickcheck-text=quickcheck-text-0.1.2.1-2lR7Kay7WBV5AMKAdKDpZZ --dependency=regex-tdfa=regex-tdfa-1.2.3.1-9PkkapJrmiKFVG38JA02jN --dependency=restless-git=restless-git-0.7-83nBoDMPYLtJIx01bvXzKs --dependency=rosezipper=rosezipper-0.2-GAlMD5j8Qb83GzjCYQknnx --dependency=s-cargot=s-cargot-0.1.4.0-J9AhWfrUxDQ6YMTyzXkHth --dependency=scientific=scientific-0.3.6.2-5di0UflhS5I1xpiiCPzjKz --dependency=semver-range=semver-range-0.2.7-dBvW3ofcsgDQf0zazTsJd --dependency=temporary=temporary-1.3-5Z6bOFbSCb7VhnH5UnI2r --dependency=text=text-1.2.3.0 --dependency=text-format=text-format-0.3.2-Fd261TSu6ptAAzilVN6BFx --dependency=time=time-1.8.0.2 --dependency=transformers=transformers-0.5.5.0 --dependency=tree-view=tree-view-0.5-Kkrk0dCM0oj2Q4xwPbd7Gg --dependency=unordered-containers=unordered-containers-0.2.9.0-IkfpzvG0LzrHAbTzfMidvY --dependency=vector=vector-0.12.0.1-4awQG9XUvVEBfJgKGHBhOb --dependency=vty=vty-5.21-A2OCwk39Wv3J3RjR3BvHey --dependency=witherable=witherable-0.2-2RYbFmOnVolGAqiDDS1CLe --dependency=wreq=wreq-0.5.2.1-IjuXB0jwsxA7O3uIVjcJXu --extra-include-dirs=/home/aman/Downloads/Compressed/ff13c/source/ --extra-include-dirs=/usr/local/opt/readline/include --extra-lib-dirs=/usr/local/opt/readline/lib
Process exited with code: ExitFailure 1
Logs have been written to: /home/aman/Desktop/new/echidna2/echidna/.stack-work/logs/hevm-0.31.log
Configuring hevm-0.31...
Cabal-simple_mPHDZzAJ_2.2.0.1_ghc-8.4.3: Missing dependency on a foreign
library:
* Missing (or bad) C library: ff
This problem can usually be solved by installing the system package that
provides this library (you may need the "-dev" version). If the library is
already installed but in a non-standard location then you can use the flags
--extra-include-dirs= and --extra-lib-dirs= to specify where it is.If the
library file does exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.
The stack install must find this library and end up successfully installing the library I'm using.
So, it is clear from the error provided above, that the installation was not able to find the libff file.
The best solution is to build the libff library manually and copy the required builds at the place it should be.
The best way to debug the problem is to look at the executed command closely and try to find the possible places where the libff should be...
Looking into the very 2nd error line, i.e. the complete command it was trying to execute
here, I sighted the following options being executed
--extra-include-dirs=/usr/local/opt/readline/include --extra-lib-dirs=/usr/local/opt/readline/lib
In order to solve this, I looked for the required libff library and cloned it from GitHub. I cloned the required dependencies and places them in the directory it should be.
https://github.com/scipr-lab/libff
I then followed the "Build guide" to build the required library.
Then with the superuser on, I created the directories, /usr/local/opt/readline/lib and /usr/local/opt/readline/include, and it was solved. And pasted the required builds in it.
Used stack install again, and it worked.
Note:
I thought many beginners may have similar problem while installing the required tools for their use. These small ideas may help them.

Gracefully handle a failed lazy symbol binding in C?

In short, I am looking for a way of handling this without the process crashing:
dyld: lazy symbol binding failed: Symbol not found: _evalstring
Referenced from: /Users/nrser/src/gh/nrser/my_first_ext/ext/my_first_ext/my_first_ext.bundle
Expected in: flat namespace
dyld: Symbol not found: _evalstring
Referenced from: /Users/nrser/src/gh/nrser/my_first_ext/ext/my_first_ext/my_first_ext.bundle
Expected in: flat namespace
./bin/console: line 2: 69989 Abort trap: 6 bundle exec pry -f -r my_first_ext
Is there a way to tell if evalstring is present before making the call, or trap the error in the function and return something else?
So far the resources I've been able to find focus on stopping the error form happening, not handling it when it does.
For context, I have a some C that runs both inside a Bash "loadable", where Bash's evalstring is available, and also outside of it, where evalstring is not.
It is an error to call functions that depend on evalstring unless inside the loadable, but I'd rather do something nicer than crash the process if users do.
Details
The evalstring function is declared as
extern int evalstring __P((char *, const char *, int));
in <BASH_DIR>/include/bash/builtins/common.h, at least as far as what my IDE thinks.
I understand that portability may be an issue. I'm targeting Linux and macOS/OSX at this time.
At least on Linux the only way to achieve this is to provide your own wrapper which intercepts calls to evalstring and then redirects them to actual implementation obtained via dlsym or fails in app-specific way. Such wrappers can be implemented manually or (if you need to achieve this for all functions in some library) via script (for example see Implib.so wrapper generator).

Compiling a DLL using C

I am trying to compile a simple DLL using strictly C. The code for the entire test library is shown below:
#include <stdio.h>
__declspec(dllexport) void hello(void) {
printf("Hello, World!\n");
}
The library is meant to be a simple proof of concept which is built using CMake as provided by CLion with the following CMakeLists.txt:
cmake_minimum_required(VERSION 3.12)
project(test_lib C)
set(CMAKE_C_STANDARD 99)
add_library(test_lib SHARED library.c)
The DLL was built so I tried testing it with node-ffi and got Error: Dynamic Linking Error: Win32 error 126. Taking a step back from node-ffi, I put my newly created DLL in the apparently unanimously recommended debugging tool for DLLs: Dependency Walker. And it appears that the DLL I built has errors. Specifically:
Error: At least one required implicit or forwarded dependency was not found.
Warning: At least one delay-load dependency module was not found.
Dependency Walker did however find that I am exporting the symbol hello. I am basing this on the fact that hello is listed with an entry point when I examine the root DLL in Dependency Walker. It shows up in the bottom of the two right panes in Dependency Walker with nothing in the top most pane. The only item in Dependency Walker's bottom pane for symbol exploration looks something like:
E | Ordinal | Hint | Function | Entry Point
-----+------------+------------+----------+------------
[C ] | 1 (0x0001) | 0 (0x0000) | hello | 0x000010C8
Where the [C ] is a grayish color (if this means anything to anyone).
I am not sure what I am missing to cause this DLL to have faulty exported dependencies/symbols.
As it turns out, I was building for the wrong architecture. I was building for x86 with previous attempts (with my machine using an Intel processor). After all these attempts failing, I tried using this DLL in Python using ctypes. The error ctypes provided after loading in the DLL via CDLL('lib_test.dll') was much more elaborate, not only explaining the mismatched architecture, but which one my machine was expecting. In my case, it was x86-amd64.
Makes sense now that my DLL could not find those libraries. The DLL was attempting to fetch MSVC++ runtime libraries not installed since they are for a completely different architecture.

How to resolve mutual references between two shared library?

First, I know the title of the question is not the best. Likely, these concepts are well-known. Hence, let's try to be clear now.
I have a shared library(native Java agent) which can load additional shared libraries as plugins. In short, libCore.so does at load time:
H = dlopen("libPlugin1.so", RTLD_LAZY)
routine = dlsym("registration_function", H)
routine(struct Plugin)
That part is working, my problem is in the interaction between plugins and the core.
For instance, the core has a function with the name getDataFromTag. When the plugins tries to call such a function I get the following error:
java: symbol lookup error: libplugin1.so: undefined symbol: getDataFromTag
Take into account that the problem is with native code, there is no Java-Code involved in the interaction.
Now, it seems to me that the linker is not fixing the imported symbols in libPlugin1.so when I perform the dlopen. My questions are:
Do I need to declare the imported symbols in libPlugin1.so? How?
Do I need to declare the exported symbols in libCore.so? How?
Do I have to link the plugin with a dependencies to the core? How? Isn't that weird? Sort of cyclic dependencies?
Is this relate to the option -z defs? How can use that? How it does work?
Additional points, every plugin has a function with the name registration_function. So, I cannot use RTLD_GLOBAL.
I found a solution. I don't know if it is the best. Here it goes.
Do I have to link the plugin with a dependencies to the core? How? Isn't that weird? Sort of cyclic dependencies?
That's exactly what I did. I included in plugins the following linker switches:
LDFLAGS +=-L$(PATH_TO_CORE)
# Libraries we are dependent on
LIBRARIES+=-lCore
In the core library I am using RTLD_LAZY for performance and RTLD_LOCAL to avoid symbols' redefinition.

Resources