I see this term a lot, 'symbol', and after searching on google I still can't find a definition that makes sense to me.
For example in the manual of ar command on Linux, it's said :
ar creates an index to the symbols defined in relocatable object modules in the archive when you specify the modifier s.
Are function declarations / variable declarations / defines / structure declarations etc, symbols ? Or is a symbol a term for .o files ?
In this context, what is a symbol exactly ? Act like I'm a complete beginner who knows nothing when you form your answer please !
A symbol is the name of things (functions, variables etc). Here is an example file:
#include <stdlib.h>
struct foo {
char *a;
};
struct foo global_foo;
static struct foo static_foo;
struct foo *foo_create() {
return malloc(sizeof(struct foo));
}
which I then compile and make an archive and then use nm to list its symbols:
$ gcc -c 1.c && ar -rc 1.a 1.o && nm -s 1.a
Archive index:
global_foo in 1.o
foo_create in 1.o
1.o:
0000000000000000 T foo_create
0000000000000000 B global_foo
U _GLOBAL_OFFSET_TABLE_
U malloc
0000000000000008 b static_foo
It depends on how you want to define symbol. If you define symbol as "something that stands for or suggests something else by reason of relationship, association, convention, or accidental resemblance" (definition from Merriam-Webster dictionary), then yes, you can say that variable names, function names, struct names, etc. are symbols because they symbolize variables, functions, struct, etc.
If you define symbol as a data type, then that is something a little different. A symbol in this case would be something you create that you can guarantee it to be unique. This is very useful for cases where you want to store a bunch of names, but you want to make sure that no names conflict with each other by making them unique.
The particular example that you gave us about the ar command is talking about symbols as a data type. The ar command stores references to archived files that are internally mapped in a data structure called a symbol table, which essentially means that ar guarantees that each and every names of file that are archived are unique (the way they do this is by mapping the names of each file to an index, so even if they were files with the same name, it would be unique to a specific number that it is mapped to).
Related
Why are some relocation entries in an ELF file symbol name + addend while others are section + addend? I am looking to clear up some confusion and gain a deeper understanding of ELFs. Below is my investigation.
I have a very simple C file, test.c:
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
static void func1(void)
{
fprintf(stdout, "Inside func1\n");
}
// ... a couple other simple *static* functions
int main (void)
{
func1();
// ... call some other functions
exit(EXIT_SUCCESS);
}
I then compile this into an object file with:
clang -O0 -Wall -g -c test.c -o test.o
If look at the relocations with readelf -r test.o I see the entries that refer to my static functions as follows (this one is picked from the .rela.debug_info section):
Offset Info Type Symbol's Value Symbol's Name + Addend
...
000000000000006f 0000000400000001 R_X86_64_64 0000000000000000 .text + b0
...
Why are these functions referred to as section + addend rather than symbol name + addend? I see entries for the functions in the .symtab using readelf -s test.o:
Num: Value Size Type Bind Vis Ndx Name
...
2: 00000000000000b0 31 FUNC LOCAL DEFAULT 2 func1
...
Additionally, when I disassemble the object file (via objdump -d), I see that the functions are there and weren't optimized into main or anything.
If I don't make the functions static and then look at the relocations, I see the same as before when the type is R_X86_64_64, but I also see entries that use the symbol name plus an addend with type R_X86_64_PC32. So for example in .rela.text:
Offset Info Type Symbol's Value Symbol's Name + Addend
...
00000000000000fe 0000001200000002 R_X86_64_PC32 0000000000000000 func1 + 1c
...
Please let me know if more examples/readelf output would be helpful. Thank you for taking the time to read this.
Why are these functions referred to as section + addend rather than symbol name + addend?
The function names for static functions are not guaranteed to be present at link time. You could remove them with e.g. objcopy --strip-unneeded or objcopy --strip-symbol, and the result will still link.
I see entries for the functions in the .symtab using readelf -s test.o
I believe the only reason they are kept is to help debugging, and they are not used by the linker at all. But I have not verified this by looking at linker source, and so did not answer this related question.
Eli Bendersky's blog also mentions this in his blog post. From the section titled "Extra credit: Why was the call relocation needed?":
In short, however, when ml_util_func is global, it may be overridden in the executable or another shared library, so when linking our shared library, the linker can't just assume the offset is known and hard-code it [12]. It makes all references to global symbols relocatable in order to allow the dynamic loader to decide how to resolve them. This is why declaring the function static makes a difference - since it's no longer global or exported, the linker can hard-code its offset in the code.
The full post should be read to get complete context, but I thought I would share it here as it presents better examples than in my question and reinforces the solution that Employed Russian gave.
I have two files
main.c
void swap();
int buf[2] = {1, 2};
int main()
{
swap();
return 0;
}
swap.c
extern int buf[];
int* bufp0 = &buf[0]; /* .data */
int* bufp1; /* .bss */
void swap()
{
int temp;
bufp1 = &buf[1];
temp = *bufp0;
*bufp0 = *bufp1;
*bufp1 = temp;
}
Here are 2 excerpts from a book
During this scan, the linker maintains a set E of relocatable object files that
will be merged to form the executable, a set U of unresolved symbols
(i.e., symbols referred to, but not yet defined), and a set D of symbols that
have been defined in previous input files.
Initially, E, U , and D are empty.
For each input file f on the command line, the linker determines if f is an
object file or an archive. If f is an object file, the linker adds f to E, updates
U and D to reflect the symbol definitions and references in f , and proceeds
to the next input file.
If f is an archive, the linker attempts to match the unresolved symbols in U
against the symbols defined by the members of the archive. If some archive
member, m, defines a symbol that resolves a reference in U , then m is added
to E, and the linker updates U and D to reflect the symbol definitions and
references in m. This process iterates over the member object files in the
archive until a fixed point is reached where U and D no longer change. At
this point, any member object files not contained in E are simply discarded
and the linker proceeds to the next input file.
If U is nonempty when the linker finishes scanning the input files on the
command line, it prints an error and terminates. Otherwise, it merges and
relocates the object files in E to build the output executable file.
The general rule for libraries is to place them at the end of the command
line. If the members of the different libraries are independent, in that no member
references a symbol defined by another member, then the libraries can be placed
at the end of the command line in any order.
If, on the other hand, the libraries are not independent, then they must be
ordered so that for each symbol s that is referenced externally by a member of an
archive, at least one definition of s follows a reference to s on the command line.
For example, suppose foo.c calls functions in libx.a and libz.a that call func-
tions in liby.a. Then libx.a and libz.a must precede liby.a on the command
line:
unix> gcc foo.c libx.a libz.a liby.a
I ran the following command to statically link the two object files ( without creating any archive file )
gcc -static -o main.o main.c swap.c
I expected the above command to fail because both main.c and swap.c have references that are defined in each other. But contrary to my expectations, it was successful. I expect it to be successful only if I pass main.c again at the end of the command line.
How did the linker resolve the references in both the files in this case? Does the working of a linker differ when it tries to statically link multiple object files instead of archive files? My guess is that the linker circled back to main.c to resolve the reference buf in swap.c.
Generally, the default behavior of linkers is to include everything from each object module file given to it and to take from a library only the object modules that define references the linker is aware of when processing the library.
So, when the linker processes main.o, it prepares everything in it to go into the output file it is building. That includes remembering (whether in memory or with auxiliary files the linker maintains temporarily) all the symbols defined by main.o and all the symbols that main.o has unresolved references to. When the linker processes swap.o, it adds everything from swap.o into the output file it is building. Further, for any references in main.o that are satisfied by definitions in swap.o, it resolves those references. And, for any references in swap.o that are satisfied by definitions in main.o, it resolves those references.
As the text you quote says, for an object module file:
“(...) the linker adds f to E, updates U and D to reflect the symbol definitions and references in f, and proceeds to the next input file.”
That step is actually the same for each object module the linker adds to the executable, whether the object module comes from an object module file or comes from a library file. The difference is that if the object module is in a file, then the linker adds it to the executable unconditionally, but, if the object module is in a library, the linker adds it to the executable only if it defines a symbol the linker is currently seeking.
The gcc commands as in your case (without -c) produces an executable image. The command compiles every '.c' file on the command line into a '.o' presentation. Then it calls a linker (ld) pointing to all .o files from the command line. The linker resolves references and generates an executable named ... main.o in your case (-o names the executable). You can run it.
A static archive library is just a collection of .o files whcih were compiled separateley. The linker checks all of them in the archive to resolve symbols. You can pre-compile your .c files with '-c' qualifier, generate .o files and then use them on the command line, or create an acrhive of them and use the archive instead.
My C textbook creates an archive using the -s command line option which adds an index to the following archive file.
ar -rcs libfile.a file1.o file2.o
However, I don't understand what an index is or it's purpose?
Converting comments into an answer.
There's a long, complicated history tied up in part with the ranlib program. Basically, for the linker to be able to scan a library efficiently, some program — either ar itself or ranlib — adds a special 'index' file which identifies the symbols defined in the library and the object file within the archive that defines each of those symbols. Many versions of ar add that automatically if any of the saved files is an object file. Some, it seems, require prodding with the -s option. Others don't add the index file at all and rely on ranlib to do the job.
The ar on macOS documents:
-s — Write an object-file index into the archive, or update an existing one, even if no other change is made to the archive. You may use this modifier flag either with any operation, or alone. Running ar s on an archive is equivalent to running ranlib on it.
I've not needed to use this option explicitly on macOS for a long time (nor have I run ranlib) — I think things changed sometime in the middle of the first decade of the millennium.
Don't object files already come with symbol tables? Wouldn't it seem redundant to add another symbol table to the start of the archive file?
Each object file contains information about what's in that one object file (and information about referenced objects as well as defined ones); the archive index contains much simpler information about which of the many object files in the archive defines each symbol, so that the linker doesn't have to scan each object file in the archive separately.
So, would it be correct to say that the index at the start of the archive is just a giant symbol table which replaces the individual symbol tables in each object file so the linker has an easier job?
Not replaces — augments. It allows the linker to identify which object file(s) to pull into the linked executable. The linker then processes the object files just as it does any other object file, checking for definitions and references, marking newly defined references as satisfied and identifying previously unused references that are not yet defined. But the linker doesn't have to read every file in the archive to work out which symbols are defined by the file — it knows from the index file which ones are defined.
To clarify the index allows the linker to find the specific object file which defines a symbol rather than having to scan every object file to resolve a symbol?
Yes, that’s right.
During compiling this C code
extern void Default_Handler(void);
void NMI_Handler(void) __attribute__ ((weak, alias ("Default_Handler")));
I've recive this
error: 'NMI_Handler' aliased to undefined symbol 'Default_Handler'
How I can make alias on external defined function?
Compiler:
gcc version 7.2.1 20170904 (release) [ARM/embedded-7-branch revision 255204] (GNU Tools for Arm Embedded Processors 7-2017-q4-major)
To alias an external function, you can use objcopy. Here's an example:
Say I have the definition of a function and I alias it, like in the following program (called myimp.c):
// myimp.c
int
myadd(int x, int y)
{
return x+y;
}
int
coolguy (int x, int y) __attribute__((alias("myadd")));
If we compile this (but don't link) we get an object file and we can check out its symbols:
# compile
$ cc -c myimp.c
# look at the symbols
$ nm myimp.o
0000000000000000 T coolguy
0000000000000000 T myadd
So we can see that the __attribute__((alias("myadd")) is simply adding a symbol coolguy with the same value, 0000000000000000, as myadd. If you look in the man page for nm, it will say that T means it is a global symbol in the .text section. This makes sense because the function is not static and comprises instructions (as opposed to data).
So if we have the object file, but not the source, and we want to add a function alias we can do so using objcopy --add-symbol.
Say we have the object file resulting from compiling this source code:
// myimp2.c
int
myadd(int x, int y)
{
return x+y;
}
Compiling as above, we'll get myimp2.o, whose symbol table tools like this:
# look at the symbols
$ nm myimp2.o
0000000000000000 T myadd
So we want to add the coolguy symbol, which we do as follows
# objcopy --add-symbol coolguy=.text:0000000000000000,global myimp2.o myimp2_augmented.o
Look at the documentation on --add-symbol in the objcopy man page. Basically .text specifies the section, after the colon the value, and then global I found by passing a bad type and objcopy failed and told me a list of the valid types, from which I chose global.
Try running nm on myimp2_augmented.o, you'll see that we've added the symbol.
Now you should be able to link with myimp2_augmented.o instead of myimp.o and your program can call coolguy without linking errors.
As suggested here, add the following to your linker command file:
PROVIDE(NMI_Handler = Default_Handler);
(And remove the weak alias declaration from the C code.)
here NMI_Handler has a weak definition attribute means that if NMI_Handler is not defined then definition of Default_Handler will be used for NMI_Handler. generally there are strong definition available for interrupts like NMI_Handler or Default_Handler.you might need to add those file in your compilation to remove error.if you wants to handle NMI_Handler in your own way then you can define it somewhere and that definition will be included instead of weak one.
It's popular to employee weak and alias attribute to implement override mechanism for native C project especially in cases of firmware projects.
Both of declaration and definition can be noted by __attribute__((weak)) and its linking priority rules is related to if any declaration exists in a head file.
It would be good tactic that just to code and compile it. Then see what would happen when linking is look for its definition. I usually check symbols by -Wl,-trace,-trace-symbol=ANY_FUNCTION_NAME_YOU_WANT and nm OBJECT_FILE_NAME.
Usually I just create a weak attribute function definition and let other objects override it.
An example shown as below:
In C file:
//non-exported weak attribute definition
__attribute__((weak)) void startup_handler_dummy(void) {
printf("[%s][%s]Entry (NO OVERRIDE HANLDER COMES HERE)\n", __FILE__, __func__);
}
//weak and alias attribute declaration
void startup_handler_dummy_impl(void) __attribute__((weak, alias("startup_handler_dummy")));
//exported weak attribute definition
__attribute__((weak)) void startup_handler(void) {
startup_handler_dummy_impl();
}
In header file:
#ifndef _BOARD_H_
#define _BOARD_H_
#define STARTUP_HANDLER startup_handler //to export symbol
#endif
Hope my code can help :)
And you can see more details and program screen-shots on MyGitHub.
One of the most important rules and best practices when writing a library, is putting all symbols of the
library into a library specific namespace. C++ makes this easy, due to the namespace keyword. In
C the usual approach is to prefix the identifiers with some library specific prefix.
Rules of the C standard put some constraints on those (for safe compilation): A C compiler may look at only the first
8 characters of an identifier, so foobar2k_eggs and foobar2k_spam may be interpreted as the same
identifiers validly – however every modern compiler allows for arbitrary long identifiers, so in our times
(the 21st century) we should not have to bother about this.
But what if you're facing some libraries of which you cannot change the symbol names / idenfiers? Maybe you got
only a static binary and the headers or don't want to, or are not allowed to adjust and recompile yourself.
At least in the case of static libraries you can work around it quite conveniently.
Consider those headers of libraries foo and bar. For the sake of this tutorial I'll also give you the source files
examples/ex01/foo.h
int spam(void);
double eggs(void);
examples/ex01/foo.c (this may be opaque/not available)
int the_spams;
double the_eggs;
int spam()
{
return the_spams++;
}
double eggs()
{
return the_eggs--;
}
example/ex01/bar.h
int spam(int new_spams);
double eggs(double new_eggs);
examples/ex01/bar.c (this may be opaque/not available)
int the_spams;
double the_eggs;
int spam(int new_spams)
{
int old_spams = the_spams;
the_spams = new_spams;
return old_spams;
}
double eggs(double new_eggs)
{
double old_eggs = the_eggs;
the_eggs = new_eggs;
return old_eggs;
}
We want to use those in a program foobar
example/ex01/foobar.c
#include <stdio.h>
#include "foo.h"
#include "bar.h"
int main()
{
const int new_bar_spam = 3;
const double new_bar_eggs = 5.0f;
printf("foo: spam = %d, eggs = %f\n", spam(), eggs() );
printf("bar: old spam = %d, new spam = %d ; old eggs = %f, new eggs = %f\n",
spam(new_bar_spam), new_bar_spam,
eggs(new_bar_eggs), new_bar_eggs );
return 0;
}
One problem becomes apparent immediately: C doesn't know overloading. So we have two times two functions with
identical name but of different signature. So we need some way to distinguish those. Anyway, lets see what a
compiler has to say about this:
example/ex01/ $ make
cc -c -o foobar.o foobar.c
In file included from foobar.c:4:
bar.h:1: error: conflicting types for ‘spam’
foo.h:1: note: previous declaration of ‘spam’ was here
bar.h:2: error: conflicting types for ‘eggs’
foo.h:2: note: previous declaration of ‘eggs’ was here
foobar.c: In function ‘main’:
foobar.c:11: error: too few arguments to function ‘spam’
foobar.c:11: error: too few arguments to function ‘eggs’
make: *** [foobar.o] Error 1
Okay, this was no surprise, it just told us, what we already knew, or at least suspected.
So can we somehow resolve that identifer collision without modifying the original libraries'
source code or headers? In fact we can.
First lets resolve the compile time issues. For this we surround the header includes with a
bunch of preprocessor #define directives that prefix all the symbols exported by the library.
Later we do this with some nice cozy wrapper-header, but just for the sake of demonstrating
what's going on were doing it verbatim in the foobar.c source file:
example/ex02/foobar.c
#include <stdio.h>
#define spam foo_spam
#define eggs foo_eggs
# include "foo.h"
#undef spam
#undef eggs
#define spam bar_spam
#define eggs bar_eggs
# include "bar.h"
#undef spam
#undef eggs
int main()
{
const int new_bar_spam = 3;
const double new_bar_eggs = 5.0f;
printf("foo: spam = %d, eggs = %f\n", foo_spam(), foo_eggs() );
printf("bar: old spam = %d, new spam = %d ; old eggs = %f, new eggs = %f\n",
bar_spam(new_bar_spam), new_bar_spam,
bar_eggs(new_bar_eggs), new_bar_eggs );
return 0;
}
Now if we compile this...
example/ex02/ $ make
cc -c -o foobar.o foobar.c
cc foobar.o foo.o bar.o -o foobar
bar.o: In function `spam':
bar.c:(.text+0x0): multiple definition of `spam'
foo.o:foo.c:(.text+0x0): first defined here
bar.o: In function `eggs':
bar.c:(.text+0x1e): multiple definition of `eggs'
foo.o:foo.c:(.text+0x19): first defined here
foobar.o: In function `main':
foobar.c:(.text+0x1e): undefined reference to `foo_eggs'
foobar.c:(.text+0x28): undefined reference to `foo_spam'
foobar.c:(.text+0x4d): undefined reference to `bar_eggs'
foobar.c:(.text+0x5c): undefined reference to `bar_spam'
collect2: ld returned 1 exit status
make: *** [foobar] Error 1
... it first looks like things got worse. But look closely: Actually the compilation stage
went just fine. It's just the linker which is now complaining that there are symbols colliding
and it tells us the location (source file and line) where this happens. And as we can see
those symbols are unprefixed.
Let's take a look at the symbol tables with the nm utility:
example/ex02/ $ nm foo.o
0000000000000019 T eggs
0000000000000000 T spam
0000000000000008 C the_eggs
0000000000000004 C the_spams
example/ex02/ $ nm bar.o
0000000000000019 T eggs
0000000000000000 T spam
0000000000000008 C the_eggs
0000000000000004 C the_spams
So now we're challenged with the exercise to prefix those symbols in some opaque binary. Yes, I know
in the course of this example we have the sources and could change this there. But for now, just assume
you have only those .o files, or a .a (which actually is just a bunch of .o).
objcopy to the rescue
There is one tool particularily interesting for us: objcopy
objcopy works on temporary files, so we can use it as if it were operating in-place. There is one
option/operation called --prefix-symbols and you have 3 guesses what it does.
So let's throw this fella onto our stubborn libraries:
example/ex03/ $ objcopy --prefix-symbols=foo_ foo.o
example/ex03/ $ objcopy --prefix-symbols=bar_ bar.o
nm shows us that this seemed to work:
example/ex03/ $ nm foo.o
0000000000000019 T foo_eggs
0000000000000000 T foo_spam
0000000000000008 C foo_the_eggs
0000000000000004 C foo_the_spams
example/ex03/ $ nm bar.o
000000000000001e T bar_eggs
0000000000000000 T bar_spam
0000000000000008 C bar_the_eggs
0000000000000004 C bar_the_spams
Lets try linking this whole thing:
example/ex03/ $ make
cc foobar.o foo.o bar.o -o foobar
And indeed, it worked:
example/ex03/ $ ./foobar
foo: spam = 0, eggs = 0.000000
bar: old spam = 0, new spam = 3 ; old eggs = 0.000000, new eggs = 5.000000
Now I leave it as an exercise to the reader to implement a tool/script that automatically extracts the
symbols of a library using nm, writes a wrapper header file of the structure
/* wrapper header wrapper_foo.h for foo.h */
#define spam foo_spam
#define eggs foo_eggs
/* ... */
#include <foo.h>
#undef spam
#undef eggs
/* ... */
and applies the symbol prefix to the static library's object files using objcopy.
What about shared libraries?
In principle the same could be done with shared libraries. However shared libraries, the name tells it,
are shared among multiple programs, so messing with a shared library in this way is not such a good idea.
You will not get around writing a trampoline wrapper. Even worse you cannot link against the shared library
on the object file level, but are forced to do dynamic loading. But this deserves its very own article.
Stay tuned, and happy coding.
Rules of the C standard put some constraints on those (for safe compilation): A C compiler may look at only the first 8 characters of an identifier, so foobar2k_eggs and foobar2k_spam may be interpreted as the same identifiers validly – however every modern compiler allows for arbitrary long identifiers, so in our times (the 21st century) we should not have to bother about this.
This is not just an extension of modern compilers; the current C standard also requires the compiler to support reasonably long external names. I forget the exact length but it's something like 31 characters now if I remember right.
But what if you're facing some libraries of which you cannot change the symbol names / idenfiers? Maybe you got only a static binary and the headers or don't want to, or are not allowed to adjust and recompile yourself.
Then you're stuck. Complain to the author of the library. I once encountered such a bug where users of my application were unable to build it on Debian due to Debian's libSDL linking libsoundfile, which (at least at the time) polluted the global namespace horribly with variables like dsp (I kid you not!). I complained to Debian, and they fixed their packages and sent the fix upstream, where I assume it was applied, since I never heard of the problem again.
I really think this is the best approach, because it solves the problem for everyone. Any local hack you do will leave the problem in the library for the next unfortunate user to encounter and fight with again.
If you really do need a quick fix, and you have source, you could add a bunch of -Dfoo=crappylib_foo -Dbar=crappylib_bar etc. to the makefile to fix it. If not, use the objcopy solution you found.
If you're using GCC, the --allow-multiple-definition linker switch is a handy debugging tool. This hogties the linker into using the first definition (and not whining about it). More about it here.
This has helped me during development when I have the source to a vendor-supplied library available and need to trace into a library function for some reason or other. The switch allows you to compile and link in a local copy of a source file and still link to the unmodified static vendor library. Don't forget to yank the switch back out of the make symbols once the voyage of discovery is complete. Shipping release code with intentional name space collisions is prone to pitfalls including unintentional name space collisions.