C code structure evokes linking cycles with CMake - c

I am trying to build bozorth3 from NIST Biometric Image Software using CMake but having troubles with the linking. This software has on the one side an executable:
//bin/bozorth3.c:
int min_comp_minutiae = MIN_COMP_BOZORTH_MINUTIAE; // defines what libbozorth needs
//...
getopt_spec = malloc_or_exit( (int)strlen(default_getopt_spec) + 1,
"getopt() string" ); // uses a function from libbozorth
...on the other side a library:
//lib/bozorth3.c:
if ( pstruct->nrows < min_computable_minutiae ) { //uses a variable defined in bin
//lib/bz_alloc.c:
char * malloc_or_exit( int nbytes,
const char * what ) { // implements a function used in bin
//code
}
...and a common header:
//include/bozorth3.h
extern int min_comp_minutiae;
extern char *malloc_or_exit(int, const char *);
With the following filesets (exceprt from CMakeLists.txt)
set(LIB_SOURCE_FILES
src/lib/bozorth3/bozorth3.c
src/lib/bozorth3/bz_alloc.c #[[etc...]])
add_library(libbozorth3 ${LIB_SOURCE_FILES})
add_executable(bozorth3 src/bin/bozorth3/bozorth3.c)
Since the executable uses the library I have to add a link: target_link_libraries(bozorth3 libbozorth3). But this is results in an error:
Linking C shared library ..\..\bin\liblibbozorth3.dll
CMakeFiles\libbozorth3.dir/objects.a(bozorth3.c.obj):bozorth3.c:(.rdata$.refptr.min_computable_minutiae[.refptr.min_computable_minutiae]+0x0): undefined reference to `min_computable_minutiae'
It forces me to create a link vice-versa: target_link_libraries(libbozorth3 bozorth3) and overriding ENABLE_EXPORTS, which of course results in an error as well:
Linking C executable ..\..\bin\bozorth3.exe
CMakeFiles\bozorth3.dir/objects.a(bozorth3.c.obj): In function `main':
D:/git/ba-phmf/NBIS/bozorth3/src/bin/bozorth3/bozorth3.c:174: undefined reference to `malloc_or_exit'
And I cant have both because it obviously results in a cycle:
CMake Error: The inter-target dependency graph contains the following strongly connected component (cycle):
"libbozorth3" of type SHARED_LIBRARY
depends on "bozorth3" (weak)
"bozorth3" of type EXECUTABLE
depends on "libbozorth3" (weak)
At least one of these targets is not a STATIC_LIBRARY. Cyclic dependencies are allowed only among static libraries.
I can compile the package using the original makefiles but need to "translate" it into CMake in order to have integration into my project. I tried to analyze the makefiles but couldn't find a solution. The whole package can be found at github.
I am using CLion 2016.2.3, CMake 3.7.0-rc2, Msys2 20160921 with GCC 6.2.0 and ld 2.27 on Windows 7 Professional N x64

Solved it like this:
add_library(libbozorth3 STATIC ${LIB_SOURCE_FILES})
set_property(TARGET libbozorth3 PROPERTY OUTPUT_NAME bozorth3)
target_include_directories(libbozorth3 PRIVATE
include)
add_executable(bozorth3 ${BIN_SOURCE_FILEs})
target_include_directories(bozorth3 PRIVATE
include)
target_link_libraries(bozorth3
PRIVATE
libbozorth3
commonnbis)

Related

How can I use wasm-bindgen from a program compiled with Emscripten?

I am trying to link a Rust library containing code generated by wasm-bindgen against a program written in C which I would like to compile with Emscripten. My MRE is as follows:
On the Rust side, I have Cargo.toml:
[package]
name = "rust_project"
version = "0.1.0"
edition = "2021"
[lib]
crate-type = ["staticlib"]
[dependencies]
wasm-bindgen="0.2"
and in lib.rs I have:
use wasm_bindgen::prelude::*;
#[wasm_bindgen]
extern "C" {
#[wasm_bindgen(js_namespace = console)]
fn log(s: &str);
}
#[no_mangle]
pub extern "C" fn call_from_c() {
log("Hello, World!");
}
As a first step, I compile this with cargo build --target wasm32-unknown-unknown which produces a librust_project.a. I then set up the following C project with main.c:
/* forward declare the function from Rust */
void call_from_c();
/* call the function from main */
int main() {
call_from_c();
return 0;
}
and CMakeLists.txt:
cmake_minimum_required(VERSION 3.5)
project(c_project)
add_executable(c_project main.c)
target_link_libraries(c_project /path/to/librust_project.a)
Finally, I attempt to put it all together using the Emscripten toolchain as follows:
cmake -DCMAKE_TOOLCHAIN_FILE=path/to/Emscripten.cmake ../
make
Which is where something appears to go wrong at the linking stage with emcc reporting that __wbg_log_941ab916ed5a24bd is an undefined symbol. I suspect that this symbol (among others) is being stripped out as part of an optimization effort but I am not sure at what stage or how I can disable this optimization.
Adding the following linker options in CMake results in compilation with a warning about the undefined symbol:
target_link_libraries(c_project
path/to/librust_project.a
"-s EXPORTED_FUNCTIONS=[\"_main\",\"___wbg_log_941ab916ed5a24bd\"]"
"-s ERROR_ON_UNDEFINED_SYMBOLS=0")
but I believe these missing symbols are problematic and when I run wasm-bindgen (the CLI tool) over c_project.wasm I get the following error:
import of `__wbg_log_941ab916ed5a24bd` doesn't have an adapter listed
How can I prevent the wasm-bindgen imported/exported functions from being stripped during this process?
This is a work-in-progress answer, which hopefully I will be able to turn into a complete answer in the near future. If not perhaps, it will at least serve as a starting point for others who have come down this path.
I have been working on making my MWE even more minimal by removing CMake and Emscripten and just compiling with Clang directly. This is sufficient since I don't need to worry about the standard library in this MWE.
My command for compiling then becomes:
clang -Wall --target=wasm32-unknown-unknown --no-standard-libraries \
-Wl,--export-all \
-Wl,--no-entry \
-Wl,-L/path/to/librust_project_a
-Wl,-lrust_project
-o main.wasm main.c
Worthwhile noting is that I can trigger the following wasm-bindgen error by adding/removing the --export-all linker argument. Suggesting that the LLVM linker was responsible for removing this section before it could be processed with wasm-bindgen.
import of `__wbg_log_941ab916ed5a24bd` doesn't have an adapter listed

Can you have multiple DLLs with common exports/ordinals and exchange at run-time?

I am trying to create a couple of Win32 64-bit DLLs (Windows 10) which have different implementations but consistent symbol exports. The aim for this is that one would link with whichever one at build time but have the option at deployment to install either DLL and correctly run with that. I have achieved this straightforwardly on Linux where I am much more comfortable and familiar with run-time linking. But on Windows, I have not yet managed this and I am wondering if this is possible at all. I am trying this using both VS2010 and VS2019.
Suppose I have two libraries blah_legacy.dll and blah_modern.dll. They both export 6 symbols which are the interface to using the library, e.g. blah_open, blah_read, blah_write, blah_close, blah_control, blah_status.
I can link with the import library for either blah implementation and a test program calling each symbol loads and executes correctly with the corresponding blah DLL.
However, I cannot yet switch the DLLs at run time. For example, should I actually be able to link with blah-legacy.lib and then run with blah-modern.dll if I rename it to blah-legacy.dll? (Or vice-versa.)
I already got around basic file-naming issues and ensured the DLL needed can actually be found. I still got the application failed to start (0x22).
I used "objdump -xs" on the DLLs and noticed the order of symbols and their ordinals are different. So I created a .def file and ensured that the exported symbols match in number, names and in ordinals. Still nothing - the same error occurs.
There's still something to this I clearly have not figured out and would appreciate some guidance. Is this actually possible? Where do I start to look (which tools) to figure out what step to take next.
Yes.
I don't use Visual Studio much, but this is the kind of thing that happens all the time if you use MSYS2, and install some MinGW packages, and update them.
Here's what I mean by that: MSYS2 is an open source software distribution for Windows that, among other things, provides a bunch of native Windows software packages. The package manager (pacman) let's you choose which packages to have in your system, and it downloads DLLs and EXEs that were created by the MSYS2 developers. When an MSYS2 developer updates a library, you can download the updated library package, and all the other packages using that library will automatically start using the new DLL. Usually there is no issue with that because the new library version will be ABI-compatible with the old library version.
You do not need to use LoadLibrary or otherwise mess up your source code; the linker and the operating system should be able to take care of this for you.
Example
Here is a minimal example I threw together with MSYS2 showing how this can work.
The file foo_legacy.c represents your legacy DLL. I added some extra symbols so it wouldn't be too similar to the modern DLL.
__declspec(dllexport) int eoo() {
return 0;
}
__declspec(dllexport) const char * foo_name() {
return "legacy";
}
__declspec(dllexport) int foo_version() {
return 1;
}
__declspec(dllexport) int goo() {
return 0;
}
The file foo_modern.c represents the modern implementation:
__declspec(dllexport) const char * foo_name(void);
__declspec(dllexport) int foo_version(void);
int foo_version() {
return 2;
}
const char * foo_name() {
return "modern";
}
The file main.c represents an application using the foo API:
#include <stdio.h>
__declspec(dllimport) const char * foo_name(void);
__declspec(dllimport) int foo_version(void);
int main()
{
printf("%s %d\n", foo_name(), foo_version());
}
My build.sh file is a Bash script that builds and tests everything:
#!/usr/bin/bash
set -uex
gcc -Wall foo_legacy.c -shared -o foo_legacy.dll
gcc -Wall foo_modern.c -shared -o foo_modern.dll
gcc -Wall -c main.c -I. -o main.o
gcc main.o foo_legacy.dll -o main.exe
./main.exe # output: "legacy 1"
mv foo_modern.dll foo_legacy.dll
./main.exe # output: "modern 2"
rm foo_legacy.dll
./main.exe # fails because foo_legacy.dll is not found
The build script runs main.exe three different times, showing that it can either use the legacy DLL, or use the modern DLL, or fail, depending on what was installed in foo_legacy.dll.

Expose c-library functionality to node

I have been working on using a c library in my node project. After little bit investigation I found node-gyp.
I was successfully able to execute example but when I am trying to using third party c library functions in the code it was giving me linking error on run time.
Library can be found here http://bibutils.refbase.org/bibutils_3.40_src.tgz
I compiled the library independently to have *.a objects
I am using following example
https://github.com/nodejs/node-addon-examples/tree/master/5_function_factory/node_0.12
So I have following questions as I can infer
Shall I convert bibutils from make to gyp?
Shall I convert each source file to work with V8? I don't know how to do this.
How can I easily link this project to work with node-gyp with less noise?
Details related to script can be found below. bibutils folder is placed along with addon.cc
binding.gyp looks like
{
"targets": [
{
"target_name": "addon",
"sources": [ "addon.cc" ],
"include_dirs": ["bibutils/lib"],
"library_dirs": ["bibutils/lib/libbibutil.a","bibutils/lib/libbibprogs.a"]
}
]
}
modified addon.cc
#include <node.h>
#include "bibutils.h"
#include "bibprogs.h"
using namespace v8;
void MyFunction(const FunctionCallbackInfo<Value>& args) {
Isolate* isolate = Isolate::GetCurrent();
HandleScope scope(isolate);
/****This is not production code just to check the execution***/
bibl b;
bibl_init( &b );
bibl_free( &b );
/**************************************************************/
args.GetReturnValue().Set(String::NewFromUtf8(isolate, "hello world"));
}
void CreateFunction(const FunctionCallbackInfo<Value>& args) {
Isolate* isolate = Isolate::GetCurrent();
HandleScope scope(isolate);
Local<FunctionTemplate> tpl = FunctionTemplate::New(isolate, MyFunction);
Local<Function> fn = tpl->GetFunction();
// omit this to make it anonymous
fn->SetName(String::NewFromUtf8(isolate, "theFunction"));
args.GetReturnValue().Set(fn);
}
Compilation Result
user1#ubuntu:~/node-addon-examples/5_function_factory/node_0.12$ npm install
> function_factory#0.0.0 install /home/user1/node-addon-examples/5_function_factory/node_0.12
> node-gyp rebuild
make: Entering directory `/home/user1/node-addon-examples/5_function_factory/node_0.12/build'
CXX(target) Release/obj.target/addon/addon.o
SOLINK_MODULE(target) Release/obj.target/addon.node
COPY Release/addon.node
make: Leaving directory `/home/user1/node-addon-examples/5_function_factory/node_0.12/build'
On Execution
user1#ubuntu:~/node-addon-examples/5_function_factory/node_0.12$ node addon.js
node: symbol lookup error: /home/user1/node-addon-examples/5_function_factory/node_0.12/build/Release/addon.node: undefined symbol: _Z9bibl_initP4bibl
Debug Info:
user1#ubuntu:~/node-addon-examples/5_function_factory/node_0.12$ nm -C build/Release/addon.node | grep bibl_init
U bibl_init(bibl*)
The problem was communication between C++ and C. In above case a C header file was included in C++ code. Compile was expecting the C++ code. So on compiling linker was got choked due to mismatch in compiled code.
So I used the extern "C" directive to tell the compiler about C header files by following code.
extern "C" {
#include "bibutils.h"
#include "bibprogs.h"
}

Why including an h file with external vars and funcs results in undefined references

What if I want these externals to be resolved in runtime with dlopen?
Im trying to understand why including an h file, with shared library external vars and funcs, to a C executable program results in undefined/unresolved. (when linking)
Why do I have to add "-lsomelib" flag to the gcc linkage if I only want these symbols to be resolved in runtime.
What does the link time linker need these deffinitions resolutions for. Why cant it wait for the resolution in runtime when using dlopen.
Can anyone help me understand this?
Here something that may help understanding:
there are 3 types of linking:
static linking (.a): the compiler includes the content of the library into your code at link time so that you can move the code to other computers with the same architecture and run it.
dynamic linking (.so): the compiler resolves the symbols at link time (during compilation); but the does not includes the code of the library in your executable. When the program is started, the library is loaded. And if the library is not found the program stop. You need the library on the computer that is running the program
dynamic loading: You are in charge of loading the library functions at runtime, using dlopen and etc. Specially used for plugins
see also: http://www.ibm.com/developerworks/library/l-dynamic-libraries/ and
Difference between shared objects (.so), static libraries (.a), and DLL's (.so)?
A header file (e.g. an *.h file referenced by some #include directive) is relevant to the C or C++ compiler. The linker does not know about source files (which are input to the compiler), but only about object files produced by the assembler (in executable and linkable format, i.e. ELF)
A library file (give by -lfoo) is relevant only at link time. The compiler does not know about libraries.
The dynamic linker needs to know which libraries should be linked. At runtime it does symbol resolution (against a fixed & known set of shared libraries). The dynamic linker won't try linking all the possible shared libraries present on your system (because it has too many shared objects, or because it may have several conflicting versions of a given library), it will link only a fixed set of libraries provided inside the executable. Use objdump(1) & readelf(1) & nm(1) to explore ELF object files and executables, and ldd(1) to understand shared libraries dependencies.
Notice that the g++ program is used both for compilation and for linking. (actually it is a driver program: it starts some cc1plus -the C++ compiler proper- to compile a C++ code to an assembly file, some as -the assembler- to assemble an assembly file into an object file, and some ld -the linker- to link object files and libraries).
Run g++ as g++ -v to understand what it is doing, i.e. what program[s] is it running.
If you don't link the required libraries, at link time, some references remain unresolved (because some object files contain an external reference and relocation).
(things are slightly more complex with link-time optimization, which we could ignore)
Read also Program Library HowTo, Levine's book linkers and loaders, and Drepper's paper: how to write shared libraries
If you use dynamic loading at runtime (by using dlopen(3) on some plugin), you need to know the type and signature of relevant functions (returned by dlsym(3)). A program loading plugins always have its specific plugin conventions. For examples look at the conventions used for geany plugins & GCC plugins (see also these slides about GCC plugins).
In practice, if you are developing your application accepting some plugins, you will define a set of names, their expected type, signature, and role. e.g.
typedef void plugin_start_function_t (const char*);
typedef int plugin_more_function_t (int, double);
then declare e.g. some variables (or fields in a data structure) to point to them with a naming convention
plugin_start_function_t* plustart; // app_plugin_start in plugins
#define NAME_plustart "app_plugin_start"
plugin_more_function_t* plumore; // app_plugin_more in plugins
#define NAME_plumore "app_plugin_more"
Then load the plugin and set these pointers, e.g.
void* plugdlh = dlopen(plugin_path, RTLD_NOW);
if (!plugdlh) {
fprintf(stderr, "failed to load %s: %s\n", plugin_path, dlerror());
exit(EXIT_FAILURE; }
then retrieve the symbols:
plustart = dlsym(plugdlh, NAME_plustart);
if (!plustart) {
fprintf(stderr, "failed to find %s in %s: %s\n",
NAME_plustart, plugin_path, dlerror();
exit(EXIT_FAILURE);
}
plumore = dlsym(plugdlh, NAME_plumore);
if (!plumore) {
fprintf(stderr, "failed to find %s in %s: %s\n",
NAME_plumore, plugin_path, dlerror();
exit(EXIT_FAILURE);
}
Then use appropriately the plustart and plumore function pointers.
In your plugin, you need to code
extern "C" void app_plugin_start(const char*);
extern "C" int app_plugin_more (int, double);
and give a definition to both of them. The plugin should be compiled as position independent code, e.g. with
g++ -Wall -fPIC -O -g pluginsrc1.c -o pluginsrc1.pic.o
g++ -Wall -fPIC -O -g pluginsrc2.c -o pluginsrc2.pic.o
and linked with
g++ -shared pluginsrc1.pic.o pluginsrc2.pic.o -o yourplugin.so
You may want to link extra shared libraries to your plugin.
You generally should link your main program (the one loading plugins) with the -rdynamic link flag (because you want some symbols of your main program to be visible to your plugins).
Read also the C++ dlopen mini howto

Link against a Windows .dll+.lib file combination with GCC under Cygwin?

I know how to link against libraries in Unix-ish contexts: If I'm working with .a or .so files, I specify the root search directory with -L/my/path/to/lib/ and for libMylib I add -lMyLib.
But what if I have
a .dll (e.g. in the Windows\System32 directory)?
a .dll (in Windows\System32) and a .lib (someplace else)?
These DLLs are by some other party; I don't have access to their sources - but do have access to the corresponding include files, against which I manage to compile.
If you can link against a .lib in Cygwin or MinGW, then you can (indirectly) link against a DLL.
In the MSVC world, it is not unusual to create an import library along with a DLL. It is a static library (.lib) that loads the DLL and wraps the interface of the DLL. You just call the wrapper functions in the (static) import library and let the import library do all the DLL-related things.
For the Windows API, there are import libraries in the WindowsSDK.
For your own MSVC DLLs, MSVC can automatically generate the import libraries when you build the DLL.
For a third party DLL, you can build a static wrapper library based on the corresponding header files.
Linking against the .lib file in Cygwin or MinGW is possible. Example:
g++ -o myprg myprg.o -lShlwapi
This links against Shlwapi.lib. (The library must be in the local directory or in the library path of the linker.)
Linking against import libraries of DLLs works the same way.
Note 1: Keep in mind the different ABIs and name mangeling. However, calling plain C functions in DLL or LIB files will work in most cases.
Note 2: Keep in mind that g++ requires the libraries to be specified in the correct order.
#einpoklum Converting my comment to an answer: #n.18e9 is correct in that you must use the full path name for the lib file without any -L or -l options.
g++ -o foo.exe foo.o c:\something\somethingelse\some.lib. You can also link directly to the Windows DLL file g++ -o foo.exe foo.o c:\something\somethingelse\some.dll.
Important - make sure you are linking to a lib file (and associated dll) generated for a 64-bit platform (on MSVC target X64, not Win32).
OK you wanted an example, well let's go.
Here are two examples using gcc/g++ to link to a Windows native DLL which exports plain C functions (using here x86_64-w64-mingw32/8.3.0 on Windows 10).
I'm using my own free xmlsq library as an example https://www.cryptosys.net/xmlsq.
You can download the core native DLL and all the source code quoted below. Make sure you use the 64-bit DLL.
The native Windows DLL diXmlsq.dll is written entirely in plain C code and exports simple C functions (extern "C").
In particular, for this example, it exports a XMLSQ_Gen_Version function that returns an integer value.
The DLL was compiled using MSVC 12.0 targetting the X64 platform. The associated library file generated by MSVC is diXmlsq.lib.
I should add that this DLL works exactly the same as a Windows "Win32 API" DLL, so the instructions here should work for the standard Windows libraries in Windows\System32 (again make sure you link against the 64-bit version).
Example 1. A plain C interface.
Both these commands compile without warning on my system:
> gcc -o test-ver test-ver.c "C:\fullpath\to\x64\diXmlsq.lib"
> gcc -o test-ver test-ver.c "C:\fullpath\to\x64\diXmlsq.dll"
diXmlsq.dll is compiled using the following definition file.
(You could alternatively use __declspec(dllexport))
Ref: https://learn.microsoft.com/en-us/cpp/build/exporting-from-a-dll?view=msvc-160
diXmlsq.def
LIBRARY "diXmlsq"
EXPORTS
XMLSQ_Gen_Version
diXmlsq.h - the C interface to diXmlsq.dll
#ifdef __cplusplus
extern "C" {
#endif
long __stdcall XMLSQ_Gen_Version(void);
#ifdef __cplusplus
}
#endif
To call the core function in a plain C program:
test-ver.c
#include <stdio.h>
#include "diXmlsq.h"
int main(void)
{
long n;
n = XMLSQ_Gen_Version();
printf("Version = %ld\n", n);
return 0;
}
Example 2. A C++ interface.
Both these commands compile without warning using g++ .
> g++ -o test-simple test-simple.cpp xmlsq.cpp "C:\fullpath\to\x64\diXmlsq.lib"
> g++ -o test-simple test-simple.cpp xmlsq.cpp "C:\fullpath\to\x64\diXmlsq.dll"
The idea of the C++ interface is to be an interface to the plain C library using the more convenient STL types like std::string and std::vector.
To keep things simple we'll just demonstrate the Gen::Version method.
Extracts of the C++ code follow:
test-simple.cpp - a test C++ program.
#include <iostream>
#include "xmlsq.hpp"
int main()
{
std::cout << "xmlsq::Gen::Version=" << xmlsq::Gen::Version() << std::endl;
}
xmlsq.hpp - the C++ interface
namespace xmlsq
{
class Gen {
private:
Gen() {} // Static methods only, so hide constructor.
public:
/** Get version number of core diXmlsq DLL. */
static int Version();
};
}
xmlsq.cpp - the C++ implementation.
#include "diXmlsq.h"
#include "xmlsq.hpp"
namespace xmlsq
{
int Gen::Version() {
int n = XMLSQ_Gen_Version();
return n;
}
}
Example 3. Attempting to link to the 32-bit library by mistake.
> gcc -o test-ver test-ver.c "C:\fullpath\to\Win32\diXmlsq.lib"
C:/Strawberry/c/bin/../lib/gcc/x86_64-w64-mingw32/8.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe:
C:\Users\user\AppData\Local\Temp\cce27Dhl.o:test-ver.c:(.text+0xe):
undefined reference to `XMLSQ_Gen_Version'
collect2.exe: error: ld returned 1 exit status

Resources