Recently I've got interested in learning the heap management of C (the malloc module). I want to break the malloc source files(e.g., malloc.c, arena.c) into smaller files so it's easier to read and study for me. I'm using the glibc 2.23 and have successfully built it locally (in a separate "build" folder) on Ubuntu 14.04 following the instructions on wiki.
As my initial attempt, I put the __malloc_assert into the files massert.h and massert.c but then realized I have no idea how to add them to the makefiles so they can get compiled and linked.
Since I moved __malloc_assert out of malloc.c, I got the link errors when running make again, which was expected:
/home/mvs/git/glibc/build/libc_pic.os: In function `detach_arena':
/home/mvs/git/glibc/malloc/arena.c:629: undefined reference to `__malloc_assert'
/home/mvs/git/glibc/build/libc_pic.os: In function `mremap_chunk':
/home/mvs/git/glibc/malloc/malloc.c:2832: undefined reference to `__malloc_assert'
/home/mvs/git/glibc/malloc/malloc.c:2813: undefined reference to `__malloc_assert'
/home/mvs/git/glibc/malloc/malloc.c:2812: undefined reference to `__malloc_assert'
/home/mvs/git/glibc/malloc/malloc.c:2830: undefined reference to `__malloc_assert'
/home/mvs/git/glibc/build/libc_pic.os:/home/mvs/git/glibc/malloc/malloc.c:2776: more undefined references to `__malloc_assert' follow
I thought I should look at how the malloc/malloc.c is used in the makefiles, but I couldn't find where it is used. I'm mainly looking at the following files:
glibc/Makeconfig
glibc/Makefile
glibc/Rules
glibc/malloc/Makefile
Alternatively, I've searched the makefile on libc-help mailing list and looked at all the results but didn't find one that matches what I want. Two of the threads, "glibc + add new function" and "Adding a function to glibc?", were talking about adding a new function to the library, which is not my case (I'm not adding new function but merely restructuring the code).
I'm kind of new to the makefile system and still reading the GNU makefile manual, but thought shooting an email here may get me out of the struggle more quickly.
Thanks!
You need to add massert (not massert.c) to the routines variable in malloc/Makefile.
There are several such variables: routines is for libc it self, but there is also libm-routines for libm, and so on.
By default, these source files are built for all variants: static (.o), shared (.os), profiling (.op, profiling builds are disabled by default though). Some special functions are only intended for static builds (.oS, they go into libc_nonshared.a) and are listed in static-only-routines as well. Specific build targets can be excluded using the elide-routines.os variable.
Related
Is there a way to produce a C static library from Go code, but without Go runtime function definitions?
Rationale:
Project A creates a C static library with go build -buildmode=c-archive, libA.a .
Works well: project B uses pure C and is able to easily create an executable, statically linking with libA.a, all is fine.
Problem 1: project C happens to also use Go, but would like to use libA.a as a regular C library. Now it has a link problem: the Go runtime functions such as e.g. _cgo_panic are now defined both in project C runtime (as it uses Go) and in libA.a.
Problem 2: project D uses pure C, same as B. Yet it wants to use two different libraries from project A, e.g. libA.a and some libA2.a. Sadly, it does not link either, because Go runtime functions are now defined in both libA.a and libA2.a.
The problems faced by project C and project D could be easily resolved if project A could produce its libraries without the Go runtime definitions inside. Project C could just link with libA.a. Project D would link with libA.a, libA2.a and some libGo.a that would contain definitions of all the Go runtime stuff.
What I tried:
Using linker flags at 'project C' level, such as -Wl,--allow-multiple-definition. Now its build fails with a cryptic message 'function symbol table not sorted by program counter'.
Manually removing go.o from 'libA.a' (as it's just an "ar" archive): didn't work as 'go.o' also contained implementations of my exported functions, so I've removed too much.
Using go build -buildmode=c-shared. As expected, it produces a dynamic library which uses another format, so I could not directly use it as a static library.
Any solution at the client side (such as finding a proper way to ignore duplicate definitions at the link stage for project C) would be also considered a valid answer.
I can also accept a negative answer (no solution) if it provides enough evidence.
Update: see a related question Is there a way to include multiple c-archive packages in a single binary
With the current implementation it's not going to work to use -buildmode=c-archive multiple times and put the results into multiple shared libraries, as you've discovered. The essential problem is that there has to be only one Go runtime, but you have multiple runtimes. When using -buildmode=c-archive there's no way to isolate the different runtimes.
The -buildmode=c-shared libraries differ from buildmode=c-archive in that they are built with -Bsymbolic which forces all their local references to be local. The effect is that we have multiple Go runtimes, but they don't refer to each other so there is no confusion.
You could try adding -Wl,-Bsymbolic to build each shared library that includes Go code in c-archive if your C code doesn't mind being linked with -Bsymbolic.
I wish you luck.
I'm getting errors in the lua plugin that I'm writing that are symptomatic of linking in two copies of the lua runtime, as per this message:
http://lua-users.org/lists/lua-l/2008-01/msg00671.html
Quote:
Which in turn means the equality test for dummynode is failing.
This is the usual symptom, if you've linked two copies of the Lua
core into your application (causing two instances of dummynode to
appear).
A common error is to link C extension modules (shared libraries)
with the static library. The linker command line for extension
modules must not ever contain -llua or anything similar!
The Lua core symbols (lua_insert() and so on) are only to be
exported from the executable which contains the Lua core itself.
All C extension modules loaded afterwards can then access these
symbols. Under ELF systems this is what -Wl,-E is for on the
linker line. MACH-O systems don't need this since all non-static
symbols are exported.
This is exactly the error I'm seeing... what I don't know is what I should be doing instead.
I've added the lua src directory to the include path of the DLL that is the c component of my lua plugin, but when I link it I get a pile of errors like:
Creating library file: libmo.dll.a
CMakeFiles/moshared.dir/objects.a(LTools.c.obj): In function `moLTools_dump':
d:/projects/mo-pong/deps/mo/src/mo/lua/LTools.c:38: undefined reference to `lua_gettop'
d:/projects/mo-pong/deps/mo/src/mo/lua/LTools.c:47: undefined reference to `lua_type'
d:/projects/mo-pong/deps/mo/src/mo/lua/LTools.c:48: undefined reference to `lua_typename'
d:/projects/mo-pong/deps/mo/src/mo/lua/LTools.c:49: undefined reference to `lua_tolstring'
So, in summary, I have this situation:
A parent binary that is statically linked to the lua runtime.
A lua library that loads a DLL with C code in it.
The C code in the DLL needs to invoke the lua c api (eg. lua_gettop())
How do I link that? Surely the dynamic library can't 'see' the symbols in the parent binary, because the parent binary isn't loading them from a DLL, they're statically linked.
...but if I link the symbols in as part of the plugin, I get the error above.
Help? This seems like a problem that should turn up a lot (dll depends on symbols in parent binary, how do you link it?) but I can't seem to see any useful threads about it.
(before you ask, no, I dont have control over the parent binary and I cant get it to load the lua symbols from the DLL)
It's probably best to use libtool for this to make your linking easier and more portable. The executable needs to be linked with -export-dynamic to export all the symbols in it, including the Lua symbols from the static library. The module needs to then be linked with -module -shared -avoid-version and, if on Windows, additionall -no-undefined; if on MacOS, additionally -no-undefined -flat_namespace -undefined suppress -bundle; Linux and FreeBSD need no other symbols. This will leave the module with undefined symbols that are satisfied in the parent. If there are any missing, the module will fail to be dlopened by the parent.
The semantics are slightly different for each environment, so it might take some fiddling. Sometimes order of the flags matters. Again, libtool is recommended since it hides much of the inconsistency.
Say I have 2 static libs
ex1.a
ex2.a
In both libs I will define 10 same functions
When Compiling a sample test code say "test.c" , I link with both static libs ex1.a and ex2.a
In "test.c" I will call only 3 functions, then I will get the
linker error "same symbols deifned in both ex1.a and ex2.a libraries" This is Ok.
My Question here is :
1. Why this error only display 3 functions as multiple defined.. Why not it list all 10 functions
In VC8 How can I list all multiple defined symbols without actualy calling that function in test code ...
Thanks,
Thats because, linker tries to resovle a symbol name, when it compiles and links a code which has the function call. Only when the code has some function calls, linker would try to resolve it in either the test code or the libraries linked along and thats when it would find multiple definitions. If no function called, then I guess no problem.
What you experience is the optimizing part of the linker: By default it won't include code that isn't referenced. The compiler will create multiple object files with most likely unresolved dependencies (calls that couldn't be satisfied by the code included). So the linker takes all object files passed and tries to find solutions for the unresolved dependencies. If it fails, it will check the available library files. If there are multiple options with the same exact name/signature it will start complaining cause it won't be able to decide which one to pick (for identical code this won't matter but imagine different implementations using different "behind the scenes" work on memory, such as debug and release stuff).
The only (and possibly easiest way) I could think of to detect all these multiple definitions would be creating another static library project including all source files used in both static libs. When creating a library the linker will include everything called or exported - you won't need specific code calling the stuff for the linker to see/include everything as long as it's exported.
However I still don't understand what you're actually trying to accomplish as a whole. Trying to find code shared between two libraries?
I have existing Linux shared object file (shared library) which has been stripped. I want to produce a new version of the library with some additional functions included. I had hoped that something like the following would work, but does not:
ld -o newlib.so newfuncs.o --whole-archive existinglib.so
I do not have the source to the existing library. I could get it but getting a full build environment with the necessary dependencies in place would be a lot of effort for what seems like a simple problem.
You might like to try coming at this from a slightly different angle by loading your object using preloading.
Set LD_PRELOAD to point to your new object
export LD_PRELOAD=/my/newfuncs/dir/newfuncs.o
and specify the existing library in the same way via your LD_LIBRARY_PATH.
This will then instruct the run time linker to search for needed symbols in your object before looking in objects located in your LD_LIBRARY_PATH.
BTW You can put calls in your object to then call the function that would've been called if you hadn't specified an LD_PRELOAD object or objects. This is why this is sometimes called interposing.
This is how many memory allocation analysis tools work. They interpose versions of malloc() and free() that records the calls to alloc() and free() before then calling the actual system alloc and free functions to perform the memory management.
There's plenty of tutorials on the interwebs on using LD_PRELOAD. One of the original and best is still "Building library interposers for fun and profit". Though written nine years ago and written for Solaris it is still an excellent resource.
HTH and good luck.
Completely untested idea:
# mv existinglib.so existinglib-real.so
# ld -o exlistinglib.so -shared newfuncs.o -lexistinglib-real
The dynamic linker, when loading a program that expects to load existinglib.so, will find your version, and also load existinglib-real.so on which it depends. It doesn't quite achieve the stated goal of your question, but should look as if it does to the program loading the library.
Shared libraries are not archives, they are really more like executables. You cannot therefore just stuff extra content into them in the same way you could for a .a static library.
Short answer: exactly what you asked for can't be done.
Longer answer: depending on why you want to do this, and exactly how existinglib.so has been linked, you may get close to desired behavior. In addition to LD_PRELOAD and renaming existinglib.so already mentioned, you could also use a linker script (cat /lib/libc.so to see what I mean).
If i have two object files both defining a symbol (function) "foobar".
Is it possible to tell the linker to obey the obj file order i give in the command line call and always take the symbol from first file and never the later one?
AFAIK the "weak" pragma works only on shared libraries but not on object files.
Please answer for all the C/C++ compiler/linker/operating system combinations you know cause i'm flexibel and use a lot of compiles (sun studio, intel, msvc, gcc, acc).
I believe that you will need to create a static library from the second object file, and then link the first object file and then the library. If a symbol is resolved by an object file, the linker will not search the libraries for it.
Alternatively place both object files in separate static libraries, and then the link order will be determined by their occurrence in the command line.
Creating a static library from an object file will vary depending on the tool chain. In GCC use the ar utility, and for MSVC lib.exe (or use the static library project wizard).
There is a danger here, the keyword here is called Interpositioning dependant code.
Let me give you an example here:
Supposing you have written a custom routine called malloc. And you link in the standard libraries, what will happen is this, functions that require the usage of malloc (the standard function) will use your custom version and the end result is the code may become unstable as the unintended side effect and something will appear 'broken'.
This is just something to bear in mind.
As in your case, you could 'overwrite' (I use quotes to emphasize) the other function but then how would you know which foobar is being used? This could lead to debugging grief when trying to figure out which foobar is called.
Hope this helps,
Best regards,
Tom.
You can make it as a .a file... Then the compiler gets the symbol and doesn't crib later