I just learned about the 4k demo scene contest. It consists in creating a 4KB executable which renders a nice 3D scene. The cited demo was build for Windows, so I was wondering, how one could create 4KB OpenGL scenes on Linux.
A bare "hello world" already consumes 8KB:
$ cat ex.c
#include <stdio.h>
int main()
{
printf("Hello world\n");
}
$ gcc -Os ex.c -o ex
$ ls -l ex
-rwxrwxr-x 1 cklein cklein 8374 2012-05-11 13:56 ex
The main reason why with the standard settings you can't make a small tool is that a lot of symbols and references to standard libraries are pulled into your binary. You must be explicit to to remove even that basic stuff.
Here's how I did it:
http://phresnel.org/gpl/4k/ntropy2k7/
Relevant Options:
Mostly self-explaining:
gcc main.c -o fourk0001 -Os -mfpmath=387 \
-mfancy-math-387 -fmerge-all-constants -fsingle-precision-constant \
-fno-math-errno -Wall -ldl -ffast-math -nostartfiles -nostdlib \
-fno-unroll-loops -fshort-double
Massage:
strip helps you get rid of unneeded symbols embedded in your binary:
strip -R .note -R .comment -R .eh_frame -R .eh_frame_hdr -s fourk0001
Code:
You may have to tweak and trial and error a lot. Sometimes, a loop gives smaller code, sometimes a call, sometimes a force inlined function. In my code, e.g., instead of having a clean linked list that contains all flame transforms in fancy polymorphic style, I have a fixed array where each element is a big entity containing all parameters, used or unused, as a union of all flames as per Scott Draves flame paper.
Your tricks won't be portable, other versions of g++ might give suboptimal results.
Note that with above parameters, you do not write a main() function, but rather a _start() function.
Also note that using libraries is a bit different. Instead of linking SDL and standard library functions the classy, convenient way, you must do it manually. E.g.
void *libSDL = dlopen( "libSDL.so", RTLD_LAZY );
void *libC = dlopen( "libc.so", RTLD_LAZY );
#if 1
SDL_SetVideoMode_t sym_SDL_SetVideoMode = dlsym(libSDL, "SDL_SetVideoMode");
g_sdlbuff = sym_SDL_SetVideoMode(WIDTH,HEIGHT,32,SDL_HWSURFACE|SDL_DOUBLEBUF);
#else
((SDL_SetVideoMode_t)dlsym(libSDL, "SDL_SetVideoMode"))(WIDTH,HEIGHT,32,SDL_HWSURFACE|SDL_DOUBLEBUF);
#endif
//> need malloc, probably kinda craft (we only use it once :| )
//> load some sdl cruft (cruft!)
malloc_t sym_malloc = dlsym( libC, "malloc" );
sym_rand = dlsym( libC, "rand" );
sym_srand = dlsym( libC, "srand" );
sym_SDL_Flip = dlsym(libSDL, "SDL_Flip");
sym_SDL_LockSurface = dlsym(libSDL, "SDL_LockSurface");
sym_SDL_UnlockSurface = dlsym(libSDL, "SDL_UnlockSurface");
sym_SDL_MapRGB = dlsym(libSDL, "SDL_MapRGB");
And even though no assembler has to be harmed, your code might yield UB.
edit:
Oops, I lied about assembly.
void _start() {
...
asm( "int $0x80" :: "a"(1), "b"(42) );
}
this will make your program return 42.
A Whirlwind Tutorial on Creating Really Teensy ELF Executables for Linux is an interesting article that goes through a step-by-step process to create an ELF executable as small as possible.
I don't want to spoil the ending, but the author gets it down to a lot smaller than 4K ;)
Take a look at this article in KSplice blog from a while back. It talks about linking without the standard libraries.
https://blogs.oracle.com/ksplice/entry/hello_from_a_libc_free
Related
I'm writing a program using libraries foo1.a and foo2.a.
Inside foo2.a, it uses foo3.a, which implements a function funcfoo. But foo1.a contains the same function that implements its own funcfoo. My main program wants to use funcfoo from foo1.a, and some other functions from foo2.a, while also makes sure that foo2.a only uses funcfoo from foo3.a.
Is there anyway I can enforce this to happen?
Short answer : you can't
Long answer : you still can't because name (like function name as well as global variable or enum name ... any name) have to be unique across your binary[1]
but
there is maybe action you can do.
If you own the source code of any library involved in this mess, make a complete new library with the same stuff but renamed.
Basically, i say "make a new major version of you library, since you will change the function name present in the library, thus breaking the retro-compatibility".
If you library is "foo1.a" containing the function "min", then make a "foo2.a" with "foo2_min" function.
It's usually a good pratice to prefix/suffix your internal function (be it in a library or directly in your binary) with something. Like if your company name is "My Little Pony", "MLP_" sound a good prefix (do a google search, just in case).
That's what user694733 said in the comment.
If you're using an IDE, then it should be easy and quick to do so with the "rename refactoring" feature.
I advise to do the most renaming possible in order to avoid further scenario like the one your stuck with now.
You don't have the source code, but the library licence allow you to modify it.
If it's format is open (like a good old so), you can use objcopy.
I read it can do that, but I never do it myself so .... good luck
If it's a close format, either you have the documentation about it and you have to do a lot of work, or you're completly stuck.
A third party own the code source, and/or the library licence don't allow you to modify it
If it's a third party library, maybe your company has subscribed to his support, so you better contact them directly.
.
As far as I know, there is no possibility to "encapsulate" a library into a "spacename" nativly in C.
I hope this answer an help, and I hope it's accurate and complete.
[1] Well, this is not completly rigth as you can create a local variable with a global variable's name, and this will compile and run. In the local context, it's the local variable that will be used. gcc can warn about this kind of scenario with -Wshadow option.
But unless you're doing some shady hack, this situation is usually something that you want to avoid.
Most linkers obey the order of the libraries as you provide them on the command line. If you link your program first with "foo1.a", it will resolve the references of funcfoo in your program with the implemention of "foo1.a". Place "foo2.a" second, which will leave an open reference to funcfoo. With "foo3.a" placed third, this will be resolved with the second implementation.
EDIT 1:
I'm afraid that I was wrong. A quick check (just before going to sleep) revealed that when "foo2.a" is linked, the reference to funcfoo is resolved with the implementation of the already loaded "foo1.a". :-(
I will do some more research, but please don't hold your breath.
EDIT 2:
OK, it took some time, but with the help of "objcopy" is works as proposed. You can use its option --redefine-sym old=new to "rename" symbols even in a library.
I have prepared a small example to follow the steps as a prove of concept. In the real project, the libraries are already built, so just the last commands of the shown list are needed.
Let's start with the main source:
#include "foo1.h"
#include "foo2.h"
int main(void) {
funcfoo();
funcbar();
return 0;
}
It includes the following header files, first "foo1.h" and second "foo2.h":
#ifndef FOO1_H
#define FOO1_H
void funcfoo(void);
#endif
#ifndef FOO2_H
#define FOO2_H
void funcbar(void);
#endif
Of course there are implementations of both, again first "foo1.c" and second "foo2.c":
#include <stdio.h>
#include "foo1.h"
void funcfoo(void) {
puts("funcfoo() in foo1");
}
#include "foo2.h"
#include "foo3.h"
void funcbar(void) {
funcfoo();
}
The third library "foo3" implements the same function as "foo1", first the header file and then the implementation file:
#ifndef FOO3_H
#define FOO3_H
void funcfoo(void);
#endif
#include <stdio.h>
#include "foo3.h"
void funcfoo(void) {
puts("funcfoo() in foo3");
}
These are the commands to build the application:
gcc -c -Wall -Wextra foo1.c -o foo1.o
ar cr libfoo1.a foo1.o
gcc -c -Wall -Wextra foo2.c -o foo2.o
ar cr libfoo2.a foo2.o
gcc -c -Wall -Wextra foo3.c -o foo3.o
ar cr libfoo3.a foo3.o
# In the real prject, only the following steps are needed:
gcc -c -Wall -Wextra main.c -o main.o
objcopy --redefine-sym funcfoo=funcfoo2 libfoo2.a libfoo2n.a
objcopy --redefine-sym funcfoo=funcfoo2 libfoo3.a libfoo3n.a
gcc main.o -L. -lfoo1 -lfoo2n -lfoo3n -o app
As you can see, the tool "objcopy" of the common "binutils" is used to redefine the name of the conflicting function from funcfoo to funcfoo2. I also let it create a new output library to save the original.
And when run, the application prints:
funcfoo() in foo1
funcfoo() in foo3
I'm trying to set up a SDL2 and C development environment on Windows 10 with MinGW-w64.
When trying to run the basic c app with SDL initialization, it compiles without warnings but fails to run afterwards, again without any warnings. Executable just exits.
Here's the source:
#include<SDL2/SDL.h>
int main(int argc, char* argv[]) {
puts("\nmain...\n");
if (SDL_Init(SDL_INIT_VIDEO) < 0) {
printf("\nInit error: %s\n", SDL_GetError());
}
else {
puts("\nSDL init success...");
}
}
... and the makefile:
OBJS = sdl_init.c
EXE_NAME = sdl_init_test
CFLAGS_W = -w -Wl,-subsystem,windows
LFLAGS_W = -lmingw32 -lSDL2main -lSDL2
INCS_W = -IC:\MinGW\devlibs\SDL2-2.0.12\x86_64-w64-mingw32\include
LIBS_W = -LC:\MinGW\devlibs\SDL2-2.0.12\x86_64-w64-mingw32\lib
windows_debug:
gcc $(OBJS) $(INCS_W) $(LIBS_W) $(CFLAGS_W) $(LFLAGS_W) -g -o $(EXE_NAME).exe
... and the weird output from gdb:
Reading symbols from .\sdl_init_test.exe...
(gdb) list main
12 ../../src/mingw-w64-crt/crt/crt0_c.c: No such file or directory.
(gdb) b main
Breakpoint 1 at 0x402e70: file ../../src/mingw-w64-crt/crt/crt0_c.c, line 17.
I'm assuming I'm doing something wrong in the linking phase, but can't pinpoint it exactly.
On Linux, everything compiles, runs and debugs as expected.
Here's a corrected makefile, as answered which will compile and work fine in Windows console:
SRC = sdl_init.c
EXE_NAME = sdl_init_test
CFLAGS_W = -Wall -Wl,-subsystem,console
LFLAGS_W = -lmingw32 -lSDL2main -lSDL2
INCS_W = -IC:\MinGW\devlibs\SDL2-2.0.12\x86_64-w64-mingw32\include
LIBS_W = -LC:\MinGW\devlibs\SDL2-2.0.12\x86_64-w64-mingw32\lib
windows_debug:
gcc $(SRC) $(INCS_W) $(LIBS_W) $(CFLAGS_W) $(LFLAGS_W) -g -o $(EXE_NAME).exe
Aside from startup issue with missing dynamic library, you seem to be mislead (arguably by SDL actually being misleading in that aspect) that your b main in gdb sets breakpoint in your main function. That's not the case as SDL redefines main to SDL_main, so if you have #include "SDL2.h" or something similar and SDL have main wrapper implemented for your operating system - your function gets renamed. Internally main (or wmain, or WinMain, or whatever target system uses as user-defined code entry point) is implemented in SDL2main library that you link with, and it calls SDL_main (your code).
TL;DR use b SDL_main in gdb instead.
Second point is why you don't see output text. That's once again windows specific, basically because you've build "GUI" app, which is different from "console" app, and don't really have its stdout associated with console output. Output is still there but you can't see it - but it can be redirected to other program or file, e.g. your_program.exe | more or your_program.exe > stdout.txt. There are ways to reconnect stdout to console (some freopen with CON magic, as I recall), or you can just build console program instead with -Wl,-subsystem,console.
As a side note, -w compiler flag (that could be loosely read as "don't ever warn me about any potential problems with my code as I'm 100% sure it is absolutely perfect and all your warnings are unjustified complaints about my perfect code" (sorry)) is a really really bad idea, with some very rare exceptions. Compilers, especially gcc and clang, are very good at giving warnings in places where it really matter, allowing you to spot mistakes early. You want more warnings (e.g. -Wall -Wextra, probably more), not no warnings at all. And while we're at it, OBJS in makefile logically should mean object files, not sources (of course you technically can call your variables anything you like, it is just misleading).
Is it possible to compile a C++ (or the like) program without generating the executable file but writing it and executing it directly from memory?
For example with GCC and clang, something that has a similar effect to:
c++ hello.cpp -o hello.x && ./hello.x $# && rm -f hello.x
In the command line.
But without the burden of writing an executable to disk to immediately load/rerun it.
(If possible, the procedure may not use disk space or at least not space in the current directory which might be read-only).
Possible? Not the way you seem to wish. The task has two parts:
1) How to get the binary into memory
When we specify /dev/stdout as output file in Linux we can then pipe into our program x0 that reads
an executable from stdin and executes it:
gcc -pipe YourFiles1.cpp YourFile2.cpp -o/dev/stdout -Wall | ./x0
In x0 we can just read from stdin until reaching the end of the file:
int main(int argc, const char ** argv)
{
const int stdin = 0;
size_t ntotal = 0;
char * buf = 0;
while(true)
{
/* increasing buffer size dynamically since we do not know how many bytes to read */
buf = (char*)realloc(buf, ntotal+4096*sizeof(char));
int nread = read(stdin, buf+ntotal, 4096);
if (nread<0) break;
ntotal += nread;
}
memexec(buf, ntotal, argv);
}
It would also be possible for x0 directly execute the compiler and read the output. This question has been answered here: Redirecting exec output to a buffer or file
Caveat: I just figured out that for some strange reason this does not work when I use pipe | but works when I use the x0 < foo.
Note: If you are willing to modify your compiler or you do JIT like LLVM, clang and other frameworks you could directly generate executable code. However for the rest of this discussion I assume you want to use an existing compiler.
Note: Execution via temporary file
Other programs such as UPX achieve a similar behavior by executing a temporary file, this is easier and more portable than the approach outlined below. On systems where /tmp is mapped to a RAM disk for example typical servers, the temporary file will be memory based anyway.
#include<cstring> // size_t
#include <fcntl.h>
#include <stdio.h> // perror
#include <stdlib.h> // mkostemp
#include <sys/stat.h> // O_WRONLY
#include <unistd.h> // read
int memexec(void * exe, size_t exe_size, const char * argv)
{
/* random temporary file name in /tmp */
char name[15] = "/tmp/fooXXXXXX";
/* creates temporary file, returns writeable file descriptor */
int fd_wr = mkostemp(name, O_WRONLY);
/* makes file executable and readonly */
chmod(name, S_IRUSR | S_IXUSR);
/* creates read-only file descriptor before deleting the file */
int fd_ro = open(name, O_RDONLY);
/* removes file from file system, kernel buffers content in memory until all fd closed */
unlink(name);
/* writes executable to file */
write(fd_wr, exe, exe_size);
/* fexecve will not work as long as there in a open writeable file descriptor */
close(fd_wr);
char *const newenviron[] = { NULL };
/* -fpermissive */
fexecve(fd_ro, argv, newenviron);
perror("failed");
}
Caveat: Error handling is left out for clarities sake. Includes for sake of brevity.
Note: By combining step main() and memexec() into a single function and using splice(2) for copying directly between stdin and fd_wr the program could be significantly optimized.
2) Execution directly from memory
One does not simply load and execute an ELF binary from memory. Some preparation, mostly related to dynamic linking, has to happen. There is a lot of material explaining the various steps of the ELF linking process and studying it makes me believe that theoretically possible. See for example this closely related question on SO however there seems not to exist a working solution.
Update UserModeExec seems to come very close.
Writing a working implementation would be very time consuming, and surely raise some interesting questions in its own right. I like to believe this is by design: for most applications it is strongly undesirable to (accidentially) execute its input data because it allows code injection.
What happens exactly when an ELF is executed? Normally the kernel receives a file name and then creates a process, loads and maps the different sections of the executable into memory, performs a lot of sanity checks and marks it as executable before passing control and a file name back to the run-time linker ld-linux.so (part of libc). The takes care of relocating functions, handling additional libraries, setting up global objects and jumping to the executables entry point. AIU this heavy lifting is done by dl_main() (implemented in libc/elf/rtld.c).
Even fexecve is implemented using a file in /proc and it is this need for a file name that leads us to reimplement parts of this linking process.
Libraries
UserModeExec
libelf -- read, modify, create ELF files
eresi -- play with elfes
OSKit (seems like a dead project though)
Reading
http://www.linuxjournal.com/article/1060?page=0,0 -- introduction
http://wiki.osdev.org/ELF -- good overview
http://s.eresi-project.org/inc/articles/elf-rtld.txt -- more detailed Linux-specific explanation
http://www.codeproject.com/Articles/33340/Code-Injection-into-Running-Linux-Application -- how to get to hello world
http://www.acsu.buffalo.edu/~charngda/elf.html -- nice reference of ELF structure
Loaders and Linkers by John Levine -- deeoer explanation of linking
Related Questions at SO
Linux user-space ELF loader
ELF Dynamic loader symbol lookup ordering
load-time ELF relocation
How do global variables get initialized by the elf loader
So it seems possible, you decide whether is also practical.
Yes, though doing it properly requires designing significant parts of the compiler with this in mind. The LLVM guys have done this, first with a kinda-separate JIT, and later with the MC subproject. I don't think there's a ready-made tool doing it. But in principle, it's just a matter of linking to clang and llvm, passing the source to clang, and passing the IR it creates to MCJIT. Maybe a demo does this (I vaguely recall a basic C interpreter that worked like this, though I think it was based on the legacy JIT).
Edit: Found the demo I recalled. Also, there's cling, which seems to do basically what I described, but better.
Linux can create virtual file systems in RAM using tempfs. For example, I have my tmp directory set up in my file system table like so:
tmpfs /tmp tmpfs nodev,nosuid 0 0
Using this, any files I put in /tmp are stored in my RAM.
Windows doesn't seem to have any "official" way of doing this, but has many third-party options.
Without this "RAM disk" concept, you would likely have to heavily modify a compiler and linker to operate completely in memory.
If you are not specifically tied to C++, you may also consider other JIT based solutions:
in Common Lisp SBCL is able to generate machine code on the fly
you could use TinyCC and its libtcc.a which emits quickly poor (i.e. unoptimized) machine code from C code in memory.
consider also any JITing library, e.g. libjit, GNU Lightning, LLVM, GCCJIT, asmjit
of course emitting C++ code on some tmpfs and compiling it...
But if you want good machine code, you'll need it to be optimized, and that is not fast (so the time to write to a filesystem is negligible).
If you are tied to C++ generated code, you need a good C++ optimizing compiler (e.g. g++ or clang++); they take significant time to compile C++ code to optimized binary, so you should generate to some file foo.cc (perhaps in a RAM file system like some tmpfs, but that would give a minor gain, since most of the time is spent inside g++ or clang++ optimization passes, not reading from disk), then compile that foo.cc to foo.so (using perhaps make, or at least forking g++ -Wall -shared -O2 foo.cc -o foo.so, perhaps with additional libraries). At last have your main program dlopen that generated foo.so. FWIW, MELT was doing exactly that, and on Linux workstation the manydl.c program shows that a process can generate then dlopen(3) many hundred thousands of temporary plugins, each one being obtained by generating a temporary C file and compiling it. For C++ read the C++ dlopen mini HOWTO.
Alternatively, generate a self-contained source program foobar.cc, compile it to an executable foobarbin e.g. with g++ -O2 foobar.cc -o foobarbin and execute with execve that foobarbin executable binary
When generating C++ code, you may want to avoid generating tiny C++ source files (e.g. a dozen lines only; if possible, generate C++ files of a few hundred lines at least; unless lots of template expansion happens thru extensive use of existing C++ containers, where generating a small C++ function combining them makes sense). For instance, try if possible to put several generated C++ functions in the same generated C++ file (but avoid having very big generated C++ functions, e.g. 10KLOC in a single function; they take a lot of time to be compiled by GCC). You could consider, if relevant, to have only one single #include in that generated C++ file, and pre-compile that commonly included header.
Jacques Pitrat's book Artificial Beings, the conscience of a conscious machine (ISBN 9781848211018) explains in details why generating code at runtime is useful (in symbolic artificial intelligence systems like his CAIA system). The RefPerSys project is trying to follow that idea and generate some C++ code (and hopefully, more and more of it) at runtime. Partial evaluation is a relevant concept.
Your software is likely to spend more CPU time in generating C++ code than GCC in compiling it.
tcc compiler "-run" option allows for exactly this, compile into memory, run there and finally discard the compiled stuff. No filesystem space needed. "tcc -run" can be used in shebang to allow for C script, from tcc man page:
#!/usr/local/bin/tcc -run
#include <stdio.h>
int main()
{
printf("Hello World\n");
return 0;
}
C scripts allow for mixed bash/C scripts, with "tcc -run" not needing any temporary space:
#!/bin/bash
echo "foo"
sed -n "/^\/\*\*$/,\$p" $0 | tcc -run -
exit
/**
*/
#include <stdio.h>
int main()
{
printf("bar\n");
return 0;
}
Execution output:
$ ./shtcc2
foo
bar
$
C scripts with gcc are possible as well, but need temporary space like others mentioned to store executable. This script produces same output as the previous one:
#!/bin/bash
exc=/tmp/`basename $0`
if [ $0 -nt $exc ]; then sed -n "/^\/\*\*$/,\$p" $0 | gcc -x c - -o $exc; fi
echo "foo"
$exc
exit
/**
*/
#include <stdio.h>
int main()
{
printf("bar\n");
return 0;
}
C scripts with suffix ".c" are nice, headtail.c was my first ".c" file that needed to be executable:
$ echo -e "1\n2\n3\n4\n5\n6\n7" | ./headtail.c
1
2
3
6
7
$
I like C scripts, because you just have one file, you can easily move around, and changes in bash or C part require no further action, they just work on next execution.
P.S:
The above shown "tcc -run" C script has a problem, C script stdin is not available for executed C code. Reason was that I passed extracted C code via pipe to "tcc -run". New gist run_from_memory_stdin.c does it correctly:
...
echo "foo"
tcc -run <(sed -n "/^\/\*\*$/,\$p" $0) 42
...
"foo" is printed by bash part, "bar 42" from C part (42 is passed argv[1]), and piped script input gets printed from C code then:
$ route -n | ./run_from_memory_stdin.c
foo
bar 42
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.29.58.98 0.0.0.0 UG 306 0 0 wlan1
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 wlan0
169.254.0.0 0.0.0.0 255.255.0.0 U 303 0 0 wlan0
172.29.58.96 0.0.0.0 255.255.255.252 U 306 0 0 wlan1
$
One can easily modify the compiler itself. It sounds hard first but thinking about it, it seams obvious. So modifying the compiler sources directly expose a library and make it a shared library should not take that much of afford (depending on the actual implementation).
Just replace every file access with a solution of a memory mapped file.
It is something I am about to do with compiling something transparently in the background to op codes and execute those from within Java.
-
But thinking about your original question it seams you want to speed up compilation and your edit and run cycle. First of all get a SSD-Disk you get almost memory speed (use a PCI version) and lets say its C we are talking about. C does this linking step resulting in very complex operations that are likely to take more time than reading and writing from / to disk. So just put everything on SSD and live with the lag.
Finally the answer to OP question is yes!
I found memrun repo from guitmz, that demoed running (x86_64) ELF from memory, with golang and assembler. I forked that, and provided C version of memrun, that runs ELF binaries (verified on x86_64 and armv7l), either from standard input, or via first argument process substitution. The repo contains demos and documentation (memrun.c is 47 lines of code only):
https://github.com/Hermann-SW/memrun/tree/master/C#memrun
Here is simplest example, with "-o /dev/fd/1" gcc compiled ELF gets sent to stdout, and piped to memrun, which executes it:
pi#raspberrypi400:~/memrun/C $ gcc info.c -o /dev/fd/1 | ./memrun
My process ID : 20043
argv[0] : ./memrun
no argv[1]
evecve --> /usr/bin/ls -l /proc/20043/fd
total 0
lr-x------ 1 pi pi 64 Sep 18 22:27 0 -> 'pipe:[1601148]'
lrwx------ 1 pi pi 64 Sep 18 22:27 1 -> /dev/pts/4
lrwx------ 1 pi pi 64 Sep 18 22:27 2 -> /dev/pts/4
lr-x------ 1 pi pi 64 Sep 18 22:27 3 -> /proc/20043/fd
pi#raspberrypi400:~/memrun/C $
The reason I was interested in this topic was usage in "C script"s. run_from_memory_stdin.c demonstrates all together:
pi#raspberrypi400:~/memrun/C $ wc memrun.c | ./run_from_memory_stdin.c
foo
bar 42
47 141 1005 memrun.c
pi#raspberrypi400:~/memrun/C $
The C script producing shown output is so small ...
#!/bin/bash
echo "foo"
./memrun <(gcc -o /dev/fd/1 -x c <(sed -n "/^\/\*\*$/,\$p" $0)) 42
exit
/**
*/
#include <stdio.h>
int main(int argc, char *argv[])
{
printf("bar %s\n", argc>1 ? argv[1] : "(undef)");
for(int c=getchar(); EOF!=c; c=getchar()) { putchar(c); }
return 0;
}
P.S:
I added tcc's "-run" option to gcc and g++, for details see:
https://github.com/Hermann-SW/memrun/tree/master/C#adding-tcc--run-option-to-gcc-and-g
Just nice, and nothing gets stored in filesystem:
pi#raspberrypi400:~/memrun/C $ uname -a | g++ -O3 -Wall -run demo.cpp 42
bar 42
Linux raspberrypi400 5.10.60-v7l+ #1449 SMP Wed Aug 25 15:00:44 BST 2021 armv7l GNU/Linux
pi#raspberrypi400:~/memrun/C $
I am currently working trying to develop software for a SAM7X256 microcontroller in C. The device is running contiki OS and I am using the yagarto toolchain.
While studying the map file (to try to figure out why the .text region had grown so much) I discovered that several kb of the .text region where assigned to unwind support (see below)
.text 0x00116824 0xee4 c:/toolchains/yagarto/bin/../lib/gcc/arm-none-eabi/4.6.2\libgcc.a(unwind-arm.o)
0x00116c4c _Unwind_VRS_Get
......
0x0011763c __gnu_Unwind_Backtrace
.text 0x00117708 0x1b0 c:/toolchains/yagarto/bin/../lib/gcc/arm-none-eabi/4.6.2\libgcc.a(libunwind.o)
0x00117708 __restore_core_regs
0x00117708 restore_core_regs
....
0x00117894 _Unwind_Backtrace
.text 0x001178b8 0x558 c:/toolchains/yagarto/bin/../lib/gcc/arm-none-eabi/4.6.2\libgcc.a(pr-support.o)
0x00117958 __gnu_unwind_execute
...
0x00117e08 _Unwind_GetTextRelBase
I have tried finding looking for some information on unwinding and found 1 and 2. However the following is still unclear to me:
When/why do I need unwinding support?
What part of my code is causing pr-support.o, unwind-arm.o and libunwind.o to be linked?
If applicable, how do I avoid linking the items below.
In case it is necessary I am including a link to the complete map file
Thanks in advance for your help
Edit 1:
Adding Linker commands
CC = arm-none-eabi-gcc
CFLAGSNO = -I. -I$(CONTIKI)/core -I$(CONTIKI_CPU) -I$(CONTIKI_CPU)/loader \
-I$(CONTIKI_CPU)/dbg-io \
-I$(CONTIKI)/platform/$(TARGET) \
${addprefix -I,$(APPDIRS)} \
-DWITH_UIP -DWITH_ASCII -DMCK=$(MCK) \
-Wall $(ARCH_FLAGS) -g -D SUBTARGET=$(SUBTARGET)
CFLAGS += $(CFLAGSNO) -O -DRUN_AS_SYSTEM -DROM_RUN -ffunction-sections
LDFLAGS += -L $(CONTIKI_CPU) --verbose -T $(LINKERSCRIPT) -nostartfiles -Wl,-Map,$(TARGET).map
$(CC) $(LDFLAGS) $(CFLAGS) -nostartfiles -o project.elf -lc Project.a
Several parts to this answer:
the unwinding library functions are pulled in from exception "personality routines" (__aeabi_unwind_cpp_pr0 etc.) that are mentioned in exception tables in some of the GCC library function modules.
your map file shows that bpapi.o (a module which contains integer division functions) pulls in this exception code. I don't see this in the latest YAGARTO, but I do it in _divdi3.o which is another integer division helper module. I can reproduce the effect of the unwinding code being pulled in by writing a trivial main() that does a 64-bit division.
the general reason for C code having (non-trivial) exception tables is so that C++ exceptions can be thrown "through" the C code when you arbitrarily mix C and C++ code in your application.
functions which can't throw or call throwing functions, should, if they have exception tables at all, only need trivial ones marked as CANTUNWIND, so that the unwinding library isn't pulled in. You'd expect division helpers to be in this category and in fact in CodeSourcery's distribution, _divdi3.o is marked CANTUNWIND.
so the root cause is that YAGARTO's GCC library (libgcc.a) is built inappropriately. Not quite incorrectly, as it should still work, but it's code bloat that you wouldn't expect in an embedded toolchain.
Can you do anything about this? There seems to be no simple way to get the GNU linker to ignore ARM exception sections, even with a /DISCARD/ script - the link to the text section overrides that. But what you can do is add a stub definition for the exception personality routine:
void __aeabi_unwind_cpp_pr0(void) {}
int main(void) { return *(unsigned long long *)0x1000 / 3; }
compiles to 4K using YAGARTO, compared to 14K without the stub. But you might want to investigate alternative GNU tools distributions too.
GCC has an option that eliminates exception handling.
-fno-exceptions
While I'm not familiar with yagarto to say for sure, it may have a similar option. On GCC, this option eliminates this overhead at the expense of support for standard exceptions.
One of the most important rules and best practices when writing a library, is putting all symbols of the
library into a library specific namespace. C++ makes this easy, due to the namespace keyword. In
C the usual approach is to prefix the identifiers with some library specific prefix.
Rules of the C standard put some constraints on those (for safe compilation): A C compiler may look at only the first
8 characters of an identifier, so foobar2k_eggs and foobar2k_spam may be interpreted as the same
identifiers validly – however every modern compiler allows for arbitrary long identifiers, so in our times
(the 21st century) we should not have to bother about this.
But what if you're facing some libraries of which you cannot change the symbol names / idenfiers? Maybe you got
only a static binary and the headers or don't want to, or are not allowed to adjust and recompile yourself.
At least in the case of static libraries you can work around it quite conveniently.
Consider those headers of libraries foo and bar. For the sake of this tutorial I'll also give you the source files
examples/ex01/foo.h
int spam(void);
double eggs(void);
examples/ex01/foo.c (this may be opaque/not available)
int the_spams;
double the_eggs;
int spam()
{
return the_spams++;
}
double eggs()
{
return the_eggs--;
}
example/ex01/bar.h
int spam(int new_spams);
double eggs(double new_eggs);
examples/ex01/bar.c (this may be opaque/not available)
int the_spams;
double the_eggs;
int spam(int new_spams)
{
int old_spams = the_spams;
the_spams = new_spams;
return old_spams;
}
double eggs(double new_eggs)
{
double old_eggs = the_eggs;
the_eggs = new_eggs;
return old_eggs;
}
We want to use those in a program foobar
example/ex01/foobar.c
#include <stdio.h>
#include "foo.h"
#include "bar.h"
int main()
{
const int new_bar_spam = 3;
const double new_bar_eggs = 5.0f;
printf("foo: spam = %d, eggs = %f\n", spam(), eggs() );
printf("bar: old spam = %d, new spam = %d ; old eggs = %f, new eggs = %f\n",
spam(new_bar_spam), new_bar_spam,
eggs(new_bar_eggs), new_bar_eggs );
return 0;
}
One problem becomes apparent immediately: C doesn't know overloading. So we have two times two functions with
identical name but of different signature. So we need some way to distinguish those. Anyway, lets see what a
compiler has to say about this:
example/ex01/ $ make
cc -c -o foobar.o foobar.c
In file included from foobar.c:4:
bar.h:1: error: conflicting types for ‘spam’
foo.h:1: note: previous declaration of ‘spam’ was here
bar.h:2: error: conflicting types for ‘eggs’
foo.h:2: note: previous declaration of ‘eggs’ was here
foobar.c: In function ‘main’:
foobar.c:11: error: too few arguments to function ‘spam’
foobar.c:11: error: too few arguments to function ‘eggs’
make: *** [foobar.o] Error 1
Okay, this was no surprise, it just told us, what we already knew, or at least suspected.
So can we somehow resolve that identifer collision without modifying the original libraries'
source code or headers? In fact we can.
First lets resolve the compile time issues. For this we surround the header includes with a
bunch of preprocessor #define directives that prefix all the symbols exported by the library.
Later we do this with some nice cozy wrapper-header, but just for the sake of demonstrating
what's going on were doing it verbatim in the foobar.c source file:
example/ex02/foobar.c
#include <stdio.h>
#define spam foo_spam
#define eggs foo_eggs
# include "foo.h"
#undef spam
#undef eggs
#define spam bar_spam
#define eggs bar_eggs
# include "bar.h"
#undef spam
#undef eggs
int main()
{
const int new_bar_spam = 3;
const double new_bar_eggs = 5.0f;
printf("foo: spam = %d, eggs = %f\n", foo_spam(), foo_eggs() );
printf("bar: old spam = %d, new spam = %d ; old eggs = %f, new eggs = %f\n",
bar_spam(new_bar_spam), new_bar_spam,
bar_eggs(new_bar_eggs), new_bar_eggs );
return 0;
}
Now if we compile this...
example/ex02/ $ make
cc -c -o foobar.o foobar.c
cc foobar.o foo.o bar.o -o foobar
bar.o: In function `spam':
bar.c:(.text+0x0): multiple definition of `spam'
foo.o:foo.c:(.text+0x0): first defined here
bar.o: In function `eggs':
bar.c:(.text+0x1e): multiple definition of `eggs'
foo.o:foo.c:(.text+0x19): first defined here
foobar.o: In function `main':
foobar.c:(.text+0x1e): undefined reference to `foo_eggs'
foobar.c:(.text+0x28): undefined reference to `foo_spam'
foobar.c:(.text+0x4d): undefined reference to `bar_eggs'
foobar.c:(.text+0x5c): undefined reference to `bar_spam'
collect2: ld returned 1 exit status
make: *** [foobar] Error 1
... it first looks like things got worse. But look closely: Actually the compilation stage
went just fine. It's just the linker which is now complaining that there are symbols colliding
and it tells us the location (source file and line) where this happens. And as we can see
those symbols are unprefixed.
Let's take a look at the symbol tables with the nm utility:
example/ex02/ $ nm foo.o
0000000000000019 T eggs
0000000000000000 T spam
0000000000000008 C the_eggs
0000000000000004 C the_spams
example/ex02/ $ nm bar.o
0000000000000019 T eggs
0000000000000000 T spam
0000000000000008 C the_eggs
0000000000000004 C the_spams
So now we're challenged with the exercise to prefix those symbols in some opaque binary. Yes, I know
in the course of this example we have the sources and could change this there. But for now, just assume
you have only those .o files, or a .a (which actually is just a bunch of .o).
objcopy to the rescue
There is one tool particularily interesting for us: objcopy
objcopy works on temporary files, so we can use it as if it were operating in-place. There is one
option/operation called --prefix-symbols and you have 3 guesses what it does.
So let's throw this fella onto our stubborn libraries:
example/ex03/ $ objcopy --prefix-symbols=foo_ foo.o
example/ex03/ $ objcopy --prefix-symbols=bar_ bar.o
nm shows us that this seemed to work:
example/ex03/ $ nm foo.o
0000000000000019 T foo_eggs
0000000000000000 T foo_spam
0000000000000008 C foo_the_eggs
0000000000000004 C foo_the_spams
example/ex03/ $ nm bar.o
000000000000001e T bar_eggs
0000000000000000 T bar_spam
0000000000000008 C bar_the_eggs
0000000000000004 C bar_the_spams
Lets try linking this whole thing:
example/ex03/ $ make
cc foobar.o foo.o bar.o -o foobar
And indeed, it worked:
example/ex03/ $ ./foobar
foo: spam = 0, eggs = 0.000000
bar: old spam = 0, new spam = 3 ; old eggs = 0.000000, new eggs = 5.000000
Now I leave it as an exercise to the reader to implement a tool/script that automatically extracts the
symbols of a library using nm, writes a wrapper header file of the structure
/* wrapper header wrapper_foo.h for foo.h */
#define spam foo_spam
#define eggs foo_eggs
/* ... */
#include <foo.h>
#undef spam
#undef eggs
/* ... */
and applies the symbol prefix to the static library's object files using objcopy.
What about shared libraries?
In principle the same could be done with shared libraries. However shared libraries, the name tells it,
are shared among multiple programs, so messing with a shared library in this way is not such a good idea.
You will not get around writing a trampoline wrapper. Even worse you cannot link against the shared library
on the object file level, but are forced to do dynamic loading. But this deserves its very own article.
Stay tuned, and happy coding.
Rules of the C standard put some constraints on those (for safe compilation): A C compiler may look at only the first 8 characters of an identifier, so foobar2k_eggs and foobar2k_spam may be interpreted as the same identifiers validly – however every modern compiler allows for arbitrary long identifiers, so in our times (the 21st century) we should not have to bother about this.
This is not just an extension of modern compilers; the current C standard also requires the compiler to support reasonably long external names. I forget the exact length but it's something like 31 characters now if I remember right.
But what if you're facing some libraries of which you cannot change the symbol names / idenfiers? Maybe you got only a static binary and the headers or don't want to, or are not allowed to adjust and recompile yourself.
Then you're stuck. Complain to the author of the library. I once encountered such a bug where users of my application were unable to build it on Debian due to Debian's libSDL linking libsoundfile, which (at least at the time) polluted the global namespace horribly with variables like dsp (I kid you not!). I complained to Debian, and they fixed their packages and sent the fix upstream, where I assume it was applied, since I never heard of the problem again.
I really think this is the best approach, because it solves the problem for everyone. Any local hack you do will leave the problem in the library for the next unfortunate user to encounter and fight with again.
If you really do need a quick fix, and you have source, you could add a bunch of -Dfoo=crappylib_foo -Dbar=crappylib_bar etc. to the makefile to fix it. If not, use the objcopy solution you found.
If you're using GCC, the --allow-multiple-definition linker switch is a handy debugging tool. This hogties the linker into using the first definition (and not whining about it). More about it here.
This has helped me during development when I have the source to a vendor-supplied library available and need to trace into a library function for some reason or other. The switch allows you to compile and link in a local copy of a source file and still link to the unmodified static vendor library. Don't forget to yank the switch back out of the make symbols once the voyage of discovery is complete. Shipping release code with intentional name space collisions is prone to pitfalls including unintentional name space collisions.