Hello Stack Overflow Community,
i am working on a c project to interleave multiple c programs into one binary, which can run the interleaved programs as treads or forks for benchmarking purposes.
Therefore i run make in each program folder of the desired programs and prelink all .o files with "ld -r" to one new .o file. After that i add a specific named function to each of these "big" .o files, which does nothing but run the main() of each program and providing the argc and argv. Then i use objcopy to localize every global Symbol except the unknown ones and the one of my specific function which shall run the main(). At last i link these manipulated .o files together with my program which runs the specific named functions as threads, or forks or after another.
Now to my Question/Problem:
I ran into a problem with static libs. I was using ffmpeg for testing, and it builds static libs such as libavcodc and libavutil and so on. Unfortunately, "ld -r" does not link .a files. So i tried to extract these libs with ar -x and then link the extracted .o files in the way mentioned above to the "big" new .o file. But i did not work because libavcodec and libavutil both include the file ff_inverse.o. That is obviously not a problem when i just build ffmpeg, which will link these static libraries. But still, both libraries include it, so there must be a machanism which makes the choice, which ff_inverse.o to use and to link. So my Question: How does this work? Where is the difference?
The way ld does it with normal linking is to prioritize the libraries. Libraries listed first in the command line are linked in first, and only if symbols still are unresolved does it move on to the next library. When linking static libraries, it ignores the name of each .o file, because the name is unnecessary, only the exported symbols are necessary. You may want to emulate that behavior, by extracting libraries in a sorted order.
Related
I work for a group in which our test bucket has hundreds of .c source programs. The .c programs are fairly small and they all include the same 10 .h header files. These .h files are fairly large.
Each time we get a new library file to link our test programs to test, we run a script to recompile and run our test bucket against. The problem is that the compiling takes fairly long, especially if the environment is virtual.
Is there a way to compile the .h header files once, put in a separate object file and have those many .c source files link to said object file? I think this will speed up compiling time. I am willing to change/remove all the #include in the .c source programs.
Any suggestions to speeding up compile time is greatly appreciated.
Also, I should say that a script executes a makefile PER .c source test program! The makefile is not told to compile all programs in the current directory. Each test program is compiled into its own executable.
You could use precompiled header feature. See http://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html
You've asked further suggestions to speed up your compilation.
One way can be using ccache. Basically, ccache keeps a cache of the object files compiled so far and returns them (instead of re-compiling again over and over) when it recognises that the same source file is being compiled again.
Using it should be as simple as
Install ccache
Prefix your gcc/cc/g++ command with ccache
Rewrite your headers. Strip off all definition and leave in header. Strip off all implementation and put in new .c. Compile as library. Link with solution. Distribute library on runtime system.
If I understand correctly, the way libraries typically work is by using precompiled code in object files ( .so on Linux systems? ), while providing header files ( .h ) for use in projects.
What happens is when you compile, the #include <library.h> directive finds that header and pastes its contents in the source file being compiled. Then, once the source file is compiled, it is linked to the precompiled object file. That way, the library can be included in a huge number of projects without it needing to be compiled from source each time. The only part that must be recompiled when linking to a library is the ( relatively ) small amount of code in the headers, which essentially makes library functions and variables accessible to the source code.
All this means is that to drastically speed up compilation, your best bet is to take all of the functions out of the 10 .h files, and instead leave only the function prototypes in the headers. Once you have all of the functions in separate .c source files, you can compile them into an object file ( typically -c flag ). Then, whenever you need to compile a new program against the 10 headers you typically use, you can instead include your stripped down version of the headers, and link to the precompiled object. Since only the new code in the .c file has to be compiled, instead of all of the code in the headers, the process should be much faster.
I'm trying to write a simple syntax checker for C code using the frontend available in libclang. Due to deployment concerns, I need to be able to statically link all the libraries in libclang, and not pass around the .so file that has all the libraries.
I'm building clang/llvm from source, and in llvm/Release+Asserts/lib I have a bunch of .a files that I think I should be able to use, but it never seems to work (the linker spews out thousands of errors about missing symbols). However, when I compile it using the libclang.so also present in that directory as follows:
clang main.c -o bin/dlc -I../llvm/tools/clang/include -L../llvm/Release+Asserts/lib/ -lclang
Everything seems to work well.
What is the minimum set of .a files I need to include to make this work? I've tried including absolutely all of the .a files in the build output directory, with them provided to clang/gcc in different orders, without any success. I only need the functions mentioned in libclang's Index.h, but there don't seem to be any resources or documentation on what the various libclang*.a files are for. It would be very helpful to know which files libclang.so pulls in.
The following is supposed to work, as long the whole project has all static libraries (I counted 116 in my Release/lib directory).
clang main.c -o bin/dlc -I../llvm/tools/clang/include ../llvm/Release/lib/*.a
[edit: clang main.c -o bin/dlc -I../llvm/tools/clang/include ../llvm/Release/lib/libclang.a ../llvm/Release/lib/*.a]
Note that the output binary is not static, so you don't need any -static flag for gcc or ld, if you're using this syntax.
If that doesn't work you might need to list the libraries in order: if some library requires a function available in another library, then it may be necessary to list it first in the command line. See comments about link order at:
http://gcc.gnu.org/onlinedocs/gcc-4.7.2/gcc/Link-Options.html#Link-Options
This question already has an answer here:
Closed 10 years ago.
Possible Duplicate:
How to pack multiple library archives (.a) into one archive file?
I have a situation where I must provide only a single static library (.a file) to an executable file to build it.
However, I split this lib in 2 parts because one part is common to other executable files and the other is needed only by one.
So now I have lib1 (for exe1) and lib2 (for all exes)
The problem is that I can't provide two libs, so I must merge for exe1, lib2 into lib1
I tried my compiling the lib1.o with -llib2 but even if it works, it looks like if nothing happened
Are there any other way? I'm can only think about using raw object files but I don't like this idea
There's no need for two static libraries; when a static library is used, only the functions (or variables) that are needed are copied to the executable - unlike a shared library where everything in the library is accessible to the executable.
Mechanically, the other question referenced describes what you need to do:
Extract all the object files from one library
Add them to the other library
Or:
files=$(ar t lib1.a)
ar x lib1.a
ar r lib2.a $files
rm -f $files lib1.a
You can even compile each source file, produce all .o and create two different libs by using ar.
The whole library will be produced using all .o (the ones you put in lib1.a and lib2.a together), the smaller one will use just a reduced set of .o files.
Than... a single Makefile, .o files produced once, two libraryes coming out from this job: the complete one (libaplus2.a) and the reduced one (lib1.a).
If one builds static libraries in one's build scripts and one wants to use those static libraries in linking the final executable, the order one mentions the .a files is important:
g++ main.o hw.a gui.a -o executable
If gui.a uses something defined in hw.a the link will fail, because at the time hw.a is processed, the linker doesn't yet know that the definition is needed later, and doesn't include it in the being.generated executable. Manually fiddling around with the linker line is not practical, so a solution is to use --start-group and --end-group which makes the linker run twice through the libraries until no undefined symbols are found anymore.
g++ main.o -Wl,--start-group hw.a gui.a -Wl,--end-group -o executable
However the GNU ld manual says
Using this option has a significant performance cost. It is best to use it only when there are unavoidable circular references between two or more archives.
So I thought that it may be better to take all .a files and put them together into one .a file with an index (-s option of GNU ar) which says in what order the files need to be linked. Then one gives only that one .a file to g++.
But I wonder whether that's faster or slower than using the group commands. And are there any problems with that approach? I also wonder, is there better way to solve these interdependency problems?
EDIT: I've written a program that takes a list of .a files and generates a merged .a file. Works with the GNU common ar format. Packing together all static libs of LLVM works like this
$ ./arcat -o combined.a ~/usr/llvm/lib/libLLVM*.a
I compared the speed against unpacking all .a files manually and then putting them into a new .a file using ar, recomputing the index. Using my arcat tool, I get consistent runtimes around 500ms. Using the manual way, time varies greatly, and takes around 2s. So I think it's worth it.
Code is here. I put it into the public domain :)
You can determine the order using the lorder and tsort utilities, for example
libs='/usr/lib/libncurses.a /usr/lib/libedit.a'
libs_ordered=$(lorder $libs | tsort)
resulting in /usr/lib/libedit.a /usr/lib/libncurses.a because libedit depends on libncurses.
This is probably only a benefit above --start-group if you do not run lorder and tsort again for each link command. Also, it does not allow mutual/cyclic dependencies like --start-group does.
Is there a third option where you just build a single library to begin with? I had a similar problem and I eventually decided to go with the third option.
In my experience, group is slower than just unifying the .a files. You can extract all files from the archive, then create a new .a file from from the smaller files
However, you have to be careful about a circumstance where both files happen to contain the same definition (you can explicitly check for this by using nm to see what definitions are contained in each library)
got a nasm project and i'm calling a c function from it
I put the name of the function in "extern"
and when linking i put all the links together but i can an error of "undefined reference to"
here is my compile/link command
gcc -o Project4 Project4.o array1c.c readdouble.o writedouble.o readarray.o printarray.o addarray.o invertarray.o invertarray2.o invertarray3.o averagearray.o quicksort.c
I would first compile all of your .c files using the "gcc -c" command into object files, then link those resulting .o files (such as "array1c.o" and "quicksort.o") together with your other pre-existing object files and see if that still gives you an undefined reference. That may be an unnecessary step, but I've never combined raw .c files and .o files in a single call to gcc.
You may also have to add an underscore to the beginning of any c-functions called ... I know this an be a platform dependent thing (i.e., Linux typically doesn't need underscores on c-functions whereas OSX and some other UNIX platforms do).
Lastly you could try, using ld, to just link all the object files together at once rather than linking some of the object files together into Project4.o, and then linking that to what you had assembled using nasm (at least that's what I'm assuming you're doing, i.e., you're making a Project4.o, and then calling functions from that in your assembly code).
Hope this helps,
Jason