I have an open source project that relies on another open source project (let's call that other project X). Both are written in C. I've had to hack pieces of X to get multi-threading to work. This causes some issues when trying to package up the code for distribution. To make things easier, I've just included the entirety of X within mine along with the few little hacks I've made.
I'd like to do something more sophisticated now in order to keep the improved functionality of X (it has frequent releases and mine does not) without having to repackage the whole project (with my hacks) within my project again each time that X has a release.
There are only 3 or 4 functions in that need to override. I can follow what is going on in this IBM Tutorial, but how can I modify my Makefile.am to generate the Makefile changes suggested in that article? To summarize, the article suggests writing my own functions with the same signatures as the ones I want to override (in a file called libfuncs.c) and then add the following 'libs' target to the makefile:
all: libs setresgid-tester
libs: libfuncs.c
gcc -shared -Wl,-soname,libfuncs.so.1 -o libfuncs.so.1.0 libfuncs.c
ln -s libfuncs.so.1.0 libfuncs.so.1
ln -s libfuncs.so.1 libfuncs.so
setresgid-tester: setresgid-tester.c
gcc -o setresgid-tester setresgid-tester.c
All of that makes sense to me. What I need to do, however, is to have this 'libs' target created for me with the autotools. I have a Makefile.am works well right now. How do I modify it to produce the desired results above? It was difficult to find in the autotools documentation, but I may just not have known exactly what to look for.
I'm happy to provide more details if helpful. Thanks in advance.
Related
I am trying to compile some C extensions, on platform with Ubuntu 14.04, for Postgres 9.5.
In my case, I want to write my C code and compile it first to a standalone executable (as you can see from my Makefile below). This is because I am also using the NumPy API and writing functions that convert Postgres ArrayType arrays into NumPy PyArray objects, and then use some NumPy array functions. It's very tricky to get the details right, deallocate NpyIter objects correctly, etc., so I definitely need to compile, run, observe errors, and test all before finalizing the details of how the library is built for the final part where I say CREATE EXTENSION in Postgres.
When compiling, I get several undefined reference issues, such as:
tmp.c:(.text+0x2d6): undefined reference to `get_typlenbyvalalign'
tmp.c:(.text+0x346): undefined reference to `deconstruct_array'
tmp.c:(.text+0x41f): undefined reference to `DatumGetFloat8'
tmp.c:(.text+0x4ae): undefined reference to `pfree'
tmp.c:(.text+0x4ba): undefined reference to `pfree'
These are server-side functions from the Postgres C API, but with lots of Googling and lots of strongarming of pgxs I cannot figure out how to obtain the name or path of the backend Postgres library that I'm failing to link.
Almost all searches just mention libpq, but these functions are not defined in that client-side API library, so I'm looking for something else.
For reference, here's the Makefile I'm currently using. Including the library directory from pg_config --libdir must also be incorrect, as it does not cause any changes in the undefined reference errors.
INCLUDEDIRS := -I.
INCLUDEDIRS += -I/usr/include/python2.7
INCLUDEDIRS += -I/home/username/anaconda/lib/python2.7/site-packages/numpy/core/include
INCLUDEDIRS += -I$(shell pg_config --includedir-server)
INCLUDEDIRS += -I$(shell pg_config --includedir)
LIBS := -L$(shell pg_config --libdir)
LIBS += -lpython2.7
tmp: tmp.c Makefile
gcc tmp.c -o tmp $(INCLUDEDIRS) $(LIBS)
Output of pg_config --libdir is:
user#computer:~/programming$ pg_config --libdir
/usr/lib/x86_64-linux-gnu
In that library directory I also found libpgcommon and when I add it to the Makefile, some of the undefined references disappear, but not all. These still remain:
tmp.c:(.text+0x2d6): undefined reference to `get_typlenbyvalalign'
tmp.c:(.text+0x346): undefined reference to `deconstruct_array'
tmp.c:(.text+0x41f): undefined reference to `DatumGetFloat8'
So pfree was found by linking with libpgcommon, but nothing else.
Digging even further, inside of postgres.h I can see where the DatumGetFloat8 macro is defined (line 662):
#ifdef USE_FLOAT8_BYVAL
extern float8 DatumGetFloat8(Datum X);
#else
#define DatumGetFloat8(X) (* ((float8 *) DatumGetPointer(X)))
#endif
So, it must be the case that Postgres was installed in some way that made use of the USE_FLOAT8_BYVAL flag. Is that standard? Would you expect that from a plain Postgres install using package repos for a popular OS like Ubuntu?
Given this, what is the other source code or library from which DatumGetFloat8 is extern'd from? For example, searching Google for "postgres DatumGetFloat8" sheds pretty much no light on this. The best I can find is a message thread from 2011 stating (not sure if correctly):
A missing reference to DatumGetFloat8 implies that the server was built
with float8 pass by value and pljava was built with float8 pass by
reference.
(The pljava bit is irrelevant for me).
A PostgreSQL extension is a shared library and not a standalone executable, so you got to use -shared -fpic -o tmp.so in your Makefile. Don't link with -lpq or -lpgcommon.
Then gcc won't complain about undefined references – these will be resolved with the executable postgres when the shared library is loaded into PostgreSQL, either with the SQL command LOAD or when a PostgreSQL C function defined with that library is called.
Of course you'll have to use the same -D defines as the PostgreSQL server when it was built, so that for example USE_FLOAT8_BYVAL is set or unset in both. Otherwise your extension may fail to load or could crash in interesting ways.
(Since you were asking: there is nothing bad about passing float8 by value; it is actually more efficient that way if your architecture supports it. DatumGetFloat8 is defined and exported in postgres if USE_FLOAT8_BYVAL is defined.)
To make all that easy, it is best to use the PostgreSQL extension building infrastructure PGXS. That will take care of all these problems for you.
All you have to do is create an appropriate Makefile, in your case it might be as simple as
MODULES = tmp
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)
Then run make and make install, and you are ready to go!
Since your problem is to test and debug your code that uses the NumPy API, you should modularize your PostgreSQL extension so that you can do that more easily. You could for example write numpy.c which contains all the code that accesses that API and pgxs.c which contains the PostgreSQL extension code and calls functions in numpy.c.
Then you can write special testing code and link it with numpy.o to test your NumPy functions.
If you are writing a Postgres extension (the kind where you say CREATE EXTENSION foo), you don't need to worry too much about all the INCLUDEDIR etc. stuff. It is easier than you think! I've written a couple very simple Postgres extensions, despite doing C only a few times a year, and it's actually pretty easy. Your Makefile can be literally 6 lines long, like this:
MODULES = myextensionname
EXTENSION = myextensionname
DATA = myextensionname--1.0.sql
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)
You should also have a myextensionname.control file that is something like this:
comment = 'Something about your extension'
default_version = '1.0'
module_pathname = '$libdir/myextensionname'
relocatable = true
And you need a myextensionname.c like so:
#include <postgres.h>
#include <fmgr.h>
#include <catalog/pg_type.h>
#include <utils/builtins.h>
// whatever other things you need to include here
PG_MODULE_MAGIC;
// your code here
Then you write a myextensionname--1.0.sql file that is run when someone says CREATE EXTENSION, and that's where you create types/functions/whatever else you need (in SQL).
If you want a couple really easy examples to copy from, here are my two projects:
https://github.com/pjungwir/aggs_for_arrays
https://github.com/pjungwir/inetrange
If you want more advanced extension examples, look in the contrib dir of a local Postgres checkout.
Regarding your question about this specifically:
So, it must be the case that Postgres was installed in some way that made use of the USE_FLOAT8_BYVAL flag.
If you use a Makefile like the one above, you don't need to worry about that, because your extension will be compiled with the same flags, so it will treat DatumGetFloat8 the same way as your installed Postgres (whether as a function or a macro). I think you will find that USE_FLOAT_BYVAL is just the tip of the iceberg, so rather than setting individual flags, you should use include $(PGXS) to get everything to match.
EDIT: I just saw your comment that you want to write a standalone binary for testing purposes. I think I would do that after making a working extension, so you can get more familiar with the process. Also, you can look in my own projects and the contrib extensions for examples of how other people handle testing.
I would like to compile the following C file on an embedded platform:
https://github.com/openwsn-berkeley/openwsn-fw/blob/develop/firmware/openos/bsp/chips/at86rf231/radio.c
However, as you can see, on lines 20-26 of radio.c it references "radiotimer_capture_cbt":
typedef struct {
radiotimer_capture_cbt startFrame_cb;
radiotimer_capture_cbt endFrame_cb;
radio_state_t state;
} radio_vars_t;
radio_vars_t radio_vars;
So now I need to hunt down where it is defined and make sure I include the right header.
I have cloned the entire GIT repository here: https://github.com/openwsn-berkeley/openwsn-fw, and I'm looking for a way to compile this easily.
Is there a better way to get this compiled other than going through the brutal dependency nightmare?
My ultimate goal is only to get radio.c compiled and anything it needs. I do not see any makefiles in this project so I'm expecting they want us to use an IDE.
The project seems to use scons as a build system. So the simplest way is to dive into the scons files.
There's a small scons file in the directory containing the linked file and two main script in the top directory.
But if you want to play, first remove headers include, try to compile (using -c) to know which one are really needed. Once you get an object file (.o) you can use nm to identify missing symbols (marked with U.) Good luck …
Does the gcc output of the object file (C language) vary between compilations? There is no time-specific information, no change in compilation options or the source code. No change in linked libraries, environmental variables either. This is a VxWorks MIPS64 cross compiler, if that helps. I personally think it shouldn't change. But I observe that sometimes randomly, the instructions generated changes. I don't know what's the reason. Can anyone throw some light on this?
How is this built? For example, if I built the very same Linux kernel, it includes a counter that is incremented each build. GCC has options to use profiler information to guide code generation, if the profiling information changes, so will the code.
What did you analyze? The generated assembly, an objdump of object files or the executable? How did you compare the different versions? Are you sure you looked at executable code, not compiler/assembler/linker timestamps?
Did anything change in the environment? New libraries (and header files/declarations/macro definitions!)? New compiler, linker? New kernel (yes, some header files originate with the kernel source and are shipped with it)?
Any changes in environment variables (another user doing the compiling, different machine, different hookup to the net gives a different IP address that makes it's way into the build)?
I'd try tracing the build process in detail (run a build and capture the output in a file, and do so again; compare those).
Completely mystified...
I had a similar problem with g++. Pre 4.3 versions produced exactly the same object files each time. With 4.3 (and later?) some of the mangled symbol names are different for each run - even without -g or other recordings. Perhaps the use a time stamp or random number (I hope not). Obviously some of those symbols make it into the .o symbol table and you get a difference.
Stripping the object file(s) makes them equal again (wrt. binary comparison).
g++ -c file.C ; strip file.o; cmp file.o origfile.o
Why should it vary? It is the same result always. Try this:
for i in `seq 1000`; do gcc 1.c; md5sum a.out; done | sort | uniq | wc -l
The answer is always 1. Replace 1.c and a.out to suit your needs.
The above counts how many different executables are generated by gcc when compiling the same source for 1000 times.
I've found that in at least some environments, the same source may yield a different executable if the source tree for the subsequent build is located in a different directory. Example:
Checkout a pristine copy of your project to dir1. Do a full rebuild from scratch.
Then, with the same user on the same machine, checkout the same exact copy of your source code to dir2 (dir1 != dir2). Do another full rebuild from scratch.
These builds are minutes apart, with no change in the toolchain or any 3rd party libs or code. Binary comparison of source code is the same. However, the executable in dir1 has different md5sum than the executable in dir2.
If I compare the different executables in BeyondCompare's hex editor, the difference is not just some tiny section that could plausibly be a timestamp.
I do get the same executable if I build in dir1, then rebuild again in dir1. Same if I keep building the same source over and over from dir2.
My only guess is that some sort of absolute paths of the include hierarchy are embedded in the executable.
My gcc sometimes produces different code for exactly the same Input. The output object files differ in exactly one byte.
Sometimes this causes linker Errors, because one possible object file is invalid. Recompiling another version usually fixes the linker error.
The gcc Version is 4.3.4 on Suse Linux Enterprise.
The gcc Parameters are:
cc -std=c++0x -Wall -fno-builtin -march=native -g -I<path1> -I<path2> -I<path3> -o obj/file.o -c file.cpp
If someone experiences the same effect, then please let me know.
I'm trying to write a simple syntax checker for C code using the frontend available in libclang. Due to deployment concerns, I need to be able to statically link all the libraries in libclang, and not pass around the .so file that has all the libraries.
I'm building clang/llvm from source, and in llvm/Release+Asserts/lib I have a bunch of .a files that I think I should be able to use, but it never seems to work (the linker spews out thousands of errors about missing symbols). However, when I compile it using the libclang.so also present in that directory as follows:
clang main.c -o bin/dlc -I../llvm/tools/clang/include -L../llvm/Release+Asserts/lib/ -lclang
Everything seems to work well.
What is the minimum set of .a files I need to include to make this work? I've tried including absolutely all of the .a files in the build output directory, with them provided to clang/gcc in different orders, without any success. I only need the functions mentioned in libclang's Index.h, but there don't seem to be any resources or documentation on what the various libclang*.a files are for. It would be very helpful to know which files libclang.so pulls in.
The following is supposed to work, as long the whole project has all static libraries (I counted 116 in my Release/lib directory).
clang main.c -o bin/dlc -I../llvm/tools/clang/include ../llvm/Release/lib/*.a
[edit: clang main.c -o bin/dlc -I../llvm/tools/clang/include ../llvm/Release/lib/libclang.a ../llvm/Release/lib/*.a]
Note that the output binary is not static, so you don't need any -static flag for gcc or ld, if you're using this syntax.
If that doesn't work you might need to list the libraries in order: if some library requires a function available in another library, then it may be necessary to list it first in the command line. See comments about link order at:
http://gcc.gnu.org/onlinedocs/gcc-4.7.2/gcc/Link-Options.html#Link-Options
I've got several directories with subdirectories containing c or asm files and I want them all compiled/assembled and then linked. I'm not especially picky where the object files go (e.g. a special bin folder or in the src folder) as long as a make clean removes them all.
The structure would look something like this:
/src
/dir1
/dir1_1
+file1_1.s
+file1_2.s
+file1.s
/dir2
+file2.c
I'm sure there's some easy way to create a makefile that compiles all files without me having to specify where it should look (compiling all files in one directory is doable with wildcards, but what then?).
Do a Google search for 'recursive make considered harmful'. You'll find the original article which postulates that the recursive make procedure is a bad way of doing business, and you'll find some links to other places which debate the validity of the proposition.
Basically, there are two ways to do builds in a directory hierarchy (with make).
Recursive make: each directory contains a makefile which builds in sub-directories and then builds in the current directory.
Non-recursive make: the makefile includes all the dependent makefiles, and builds up the complete dependency structure for the entire project and only builds the requisite software.
I work routinely on a product where the main build sequence is driven by a hybrid system that uses a shell script plus one makefile for each directory. One section of the product is managed by a 'RMCH' makefile; most of it is not. The build script deals with phases of the build, and sequences the directories, and runs make in each directory when it is time to do so. (The source code is in 20k+ files spread over a multitude of directories - it is a big project/product.)
I've also converted a medium-small project (about 20 directories of relevance, and about 400 source files) to work with RMCH (from a script + makefile-per-directory system). It was a bit mind-blowing at first, but works pretty neatly now it is done. Whether I did it correctly is open for debate; it was primarily a learning exercise, though I also did some work modifying the code to work with a modern curses library instead of the archaic BSD library that was used as a part of the code (archaic, as in 1982-vintage - the code was last seriously developed in about 1986) and generally upgrading to modern (standard C) standards. It was also a chance to work with git - so, all in all, quite an extensive learning experience.
If you can wrap your brain around RMCH, it is a good system. If done correctly, with complete and accurate dependency tracking, it removes the guess-work from the build sequence, and it does run fast. However, migrating even a medium size project to it is fairly hard work - it would be a daunting task to do it on the main product I work on, though the system might well benefit from it.
An alternative is to look at other alternatives to make, such as cmake, rake, scons, bras, imake, or ant or whatever else takes your fancy. Most of those are easily discoverable via a Google search; the hard one is bras, which is based on Tcl (as in Tcl/Tk), but is probably largely dead now. And imake is mentioned more for completeness than as a serious suggestion. You might also look at the GNU Autotools. Those do not abandon make; they build atop make.
If your project is small enough, you might get away with using a single hand-crafted makefile instead of a more sophisticated build system: check out the manual page on transformation functions to see what's possible.
Your example project could be compiled with the following non-recursive makefile:
targets = $(patsubst %$(1),%$(2),$(foreach dir,$(3),$(wildcard $(dir)/*$(1))))
asmdirs := src/dir1 src/dir1/dir1_1
cdirs := src/dir2
asmobjects := $(call targets,.s,.o,$(asmdirs))
cobjects := $(call targets,.c,.o,$(cdirs))
.PHONY : all clean
all : $(asmobjects) $(cobjects)
clean :
rm -f $(asmobjects) $(cobjects)
$(cobjects) : %.o : %.c
gcc -o $# -c $<
$(asmobjects) : %.o : %.s
gcc -o $# -c $<
However, because make can access the shell, you could also use standard unix tools like find instead of the somewhat limited builtin functions, eg
asmsources := $(shell find src -name '*.s')
csources := $(shell find src -name '*.c')
asmobjects := $(asmsources:.s=.o)
cobjects := $(csources:.c=.o)