I noticed something peculiar while inspecting a map file produced with the -Map=mapfile option of the GNU linker. It listed a couple of symbols as belonging to the .text section, whereas the binary's symbol table lists them as part of the .rodata section. I suspect that this is some sort of cheeky optimization, as the compiler probably determined that those symbols are only ever read, but what surprises me is that the map file doesn't reflect that. My understanding is the linking is pretty much the final stage of the compilation process and all optimization happens before it. Is that correct? Why were these symbols optimized afterwards?
As you've probably inferred, the toolchain is GCC. The source code is written in C.
Related
I would like to know if someone is aware of a trick to retrieve the list of files that had been (or ideally will be) used by linker to produce an executable.
Some kind of solution must exist. A a static source analyzer, or a hack, such as compiling with some weird flags, and analyzing produced executable with another tool, or force the linker to output this information.
The goal is to provide a tool that strip useless source files from a list of source files.
The end goal is to ease the build process, by allowing him to give a list of usable source files. Then my tool would only compile the ones actually used by linker instead of everything.
This would allow for some unit_test to still be runnable even if some others are broken and can't compile, while not asking the user to manually list every test dependencies manually in the cmake.
I am targetting linux for now, but will be intersted in the futur to do the same trick on others OS. So I would like a cross-platform solution, eventhought I doubt I will have it :)
Thanks for your help
Edit because I see that it is confusing, what I mean by
allowing him to give a list of usable source file
is that, in cmake, for exemple. If you use add_executable(name, sources), then sources is considered as the sources to compile and link on.
I want to wrap add_executable, so sources is viewed as a set of usable if necessary sources files.
I'm afraid the idea of detecting never linked source files is not a fruitful one.
To build a program, CMake will not compile a source file if it not going to link the resulting object
file into the program. I can understand how you might think that this happens, but it doesn't happen.
CMake already does what you would like it to do and the same is true of every other build automation system going back to
their invention in the 1970s. The fundamental purpose of all
such systems is to ensure that the building of a program
compiles a source file name.(c|cc|f|m|...) if and only if
the object file name.o is going to be linked into the program
and is out of date or does not exist. You can always defeat this purpose by
egregiously bad coding of the project's build spec (CMakeLists.txt, Makefile, SConstruct, etc.),
but with CMake you would need to be really trying to do it, and
trying quite expertly.
If you do not want name.c to be compiled and the object file name.o
linked into a target program, then you do not tell the build system
that name.o or name.c is a prerequisite of the program. Don't tell
it what you know is not true. It is elementary competence not to specify redundant prerequisites of
a build system target.
The linker will link all its input object files into an output
program without question. It does not ask whether or not they are "needed"
by the program because it cannot answer that question. Neither the
linker nor any possible static analysis tool can know what program
you intend to produce when you input some object files for linkage.
It can only be assumed that you intend to produce the program that
results from the linkage of those object files, assuming the
linkage is successful.
If those object files cannot be linked into a program at all, the linker will tell you
that, and why. Otherwise, if you have linked object files that you didn't
intend to link, you can only discover that for yourself, by noticing
the mistake in the build log, or failing that by testing the program and/or inspecting its contents and comparing
your observations with your expectations.
Given your choice of object files for linkage, you can instruct the linker
to detect any code sections or data sections it extracts those object files in
which no symbols are defined that can be referenced by the program, and to
throw away all such unreferenced input sections instead of linking them
into the program. This is called linktime "garbage collection". You tell the
linker to do it by passing the option -Wl,-gc-sections in the
gcc linkage command. See this question
to learn how to maximise the collectible garbage. This is what you
can do to remove redundant object code from the linkage.
But you can only collect any garbage from a program in this way if the program
is dynamically opaque, i.e not linked with the option -rdynamic
: then the global symbols defined in the program's static image are not visible
to the OS loader and cannot be referenced from outside its static image by dynamic
libraries in the same process. In this case the linker can determine by static
analysis that a symbol whose definition is not referenced in the program's static
image cannot be referenced at all, since it cannot be referenced dynamically,
and if all symbols defined in an input section are statically unreferenced then
it can garbage-collect the section.
If the program has been linked -rdynamic then -Wl,-gc-sections will
collect no garbage, and this is quite right, because if the program is
not dynamically opaque then it is impossible for static analysis to determine that anything
defined in its linkage cannot be referenced.
It's noteworthy that although -rdynamic is not a default linkage
option for GCC, it is a default linkage option for CMake projects using
the GCC toolchain. So to use linktime garbage collection in CMake projects
you would always have to override the -rdynamic default. And obviously it would only be
valid to do this if you have determined that it is alright for the program to
be dynamically opaque.
I am learning about linking right now (self-taught) and I am having some trouble understanding some concepts.
After the preprocessing, compilation, and assembly of a source code file, you got a relocatable object file with an ELF format (WLOG). In this ____.o file, there is a .text section that contains the machine code of the individual source code.
Does this machine code correspond to the run-time addresses of the code that is in the input file? Like if the machine code where to run (assuming no unresolved external references) would the runtime profile of the code match the machine code here?
If this is true, is it safe to say that symbol references in this code are pointing to the runtime address of their corresponding symbols?
I need to know this so that I can better understand the linking process which happens directly after this process.
Does this machine code correspond to the run-time addresses of the code that is in the input file?
No.
It can't, because the code in a single .o file doesn't know what other object files will be linked into the main executable. Imagine foo.o saying "I want to be at address 0x123000", and bar.o saying "I want to be at address 0x123004". They clearly can't be both at the same address.
The "final" runtime addresses are determined by the linker, which collects all the different .o files, resolves references between them, and lays out the final executable in memory. (Even this isn't a complete story, as shared libraries and position-independent executables complicate the answer more.)
As far as I know compiler convert source code to machine code. But this code do not have any OS-related sections and linker add them to file.
But is it's possible to make some executable without linker?
Answering your question very literally - yes, it is possible to make an executive file without a linker: you don't need a compiler or linker to generate machine code. Binaries are a series of opcodes and relevant information (offsets, addresses etc). If you open a binary editor then type out some opcodes and make a program. Save and run it.
Of course the binary will be processor specific, just as if you had compiled a binary (native) executive. Here's a reference to the Intel x86 opcodes.
http://ref.x86asm.net/coder32.html.
If you're however asking, "Can I compile a source file directly into an executive file without a linker?" then speaking purely: no - unless the compiler has aspects of a linker integrated within it. The compiler generates intermediate objects that are passed on to the linker to "link" them into a binary such as a library or executive. Without the link step the pipeline is not complete.
Let's first make a statement that is to be considered true, compilers do not generate machine code that can be immediately executed (JIT's do, but lets ignore that).
Instead they generate files (object, static, dynamic, executable) which describe what they contains as well as groups of symbols. Symbols can be global variables or functions.
But symbols just like the file itself contain metadata. This metadata is very important. See the machine code stored in a symbol is the raw instructions for the target architecture but it does not know where memory is stored.
While modern CPU's give each process its own address space, a symbol may not land and probably won't land in the same address twice. In very recent times this is a security measure, but in past its so that dynamic linking works correctly.
So when the OS loads up an executable or shared library it can place it wherever it wants and by doing so make it not repeatable. Otherwise we'd all have to start caring and saying "this file contains 100% of the code I intend to execute". Usually on load the raw binary in the symbol table get transformed by patching it with the symbol locations in RAM. Making everything just work.
In summary the compiler emits files that allow for dynamic patching of assembly
prior to execution. If it didn't, we would be living in a very restrictive and problematic world.
Linkers even have scripts to change how they operate. They are a very complex and delicate piece of software required to make our programs work.
Have a read of the PE-COFF and ELF standards if you want to get an idea of just how complex those formats really are.
Recently I realized that the size of my executables is quite large. I am developing software for Cortex-M microcontrollers, using Eclipse and GCC.
To check this out, I used an example project I found on the internet, that simply blinks an LED by manipulating the registers directly and that is makefile based.
I created a very similar project using my libraries, startup code, linker scripts etc that uses Eclipse's managed makefiles.
The first project compiled successfully, and produced a binary file of app. 6kB. The second project produced a binary file of app.48kB! That is obviously quite a large difference for essentially the same result, and the later is definitely a huge file, for just blinking an LED. In both cases optimizations were off.
In my own libraries, there are some volatile buffers, that may be the excuse for the large BSS, or data sections, so I decided to begin by concentrating on the text section (which is still 5 times larger 5kB to 27kB).
I took a look at the map file to see what is really linked to the binary file. Same or similar functions had also similar size.
There is one thing that seems very-very odd to me. There are functions wich are defined only once in the whole project, but appear to have been linked multiple times, each time from a different object file, and each time occupying space in the text section. Take a look for example to function .text.port_lock.
Is this normal? How can I reduce the final file size, and how can I tell the toolchain to only link once each function?
The map file
Edit:
As stated in the comments the two programs are not different, it is the same thing, with minor modifications (e.g. startup code, and function to access the GPIO register). I am not testing GCC's ability to optimize code, thus I used -O0. I try to understand the map file, and why I see the some functions multiple times.
You are misreading the map file. None of the occurrences of .text.port_lock,
for example, represents a definition of the ChibiOS function void port_lock(void).
All the occurrences of .text.port_lock refer to input linker sections.
The first 4 occurrences, lying within the section of the map file titled
Discarded input sections, refer to input linker sections that the linker
discarded. For example:
.text.port_lock
0x00000000 0x1c /home/fotis/Documents/Resources/Chibios/Chibios/Debug/libChibios.a(chmempools.o)
means that the linker found a section .text.port_lock of size 28 bytes
in input file /home/fotis/Documents/Resources/Chibios/Chibios/Debug/libChibios.a(chmempools.o)
and threw it away.
The next 6 occurrences, lying within the the section of the map file titled
Linker script and memory map all refer to input linker sections that were
mapped into the output .text section. For example the first one:
.text.port_lock
0x000012a8 0x1c /tmp/ccaossic.ltrans0.ltrans.o
means that the linker found a section .text.port_lock of size 28 bytes
in input file /tmp/ccaossic.ltrans0.ltrans.o
and mapped it at address 0x000012a8 in the output .text section. Likewise the
second occurrence:
.text.port_lock
0x00001f70 0x1c /home/fotis/Documents/Resources/Chibios/Chibios/Debug/libChibios.a(chsys.o)
means that an input section of the same name and size also was found in input
file /home/fotis/Documents/Resources/Chibios/Chibios/Debug/libChibios.a(chsys.o)
and was mapped at address 0x00001f70 in the output .text section.
Altogether there are .text.port_lock input sections, all of them 28 bytes,
mapped in your output .text section from these input files:
/tmp/ccaossic.ltrans0.ltrans.o
/home/fotis/Documents/Resources/Chibios/Chibios/Debug/libChibios.a(chsys.o)
/home/fotis/Documents/Resources/Chibios/Chibios/Debug/libChibios.a(chthreads.o)
/home/fotis/Documents/Resources/Chibios/Chibios/Debug/libChibios.a(chcore_v7m.o)
/home/fotis/Documents/Resources/Chibios/Chibios/Debug/libChibios.a(chmemcore.o)
/home/fotis/Documents/Resources/Chibios/Chibios/Debug/libChibios.a(chschd.o)
In all 6 of these cases, the input section contains no symbols, and in particular
no functions. For contrast, here is an example of an input section that does contain symbols:
.text 0x000002f0 0x28 /home/fotis/Documents/Resources/Chibios/Chibios/Debug/libChibios.a(chcoreasm_v7m.o)
0x000002f0 _port_switch
0x00000300 _port_thread_start
0x00000310 _port_switch_from_isr
0x00000314 _port_exit_from_isr
This is the input .text section from /home/fotis/Documents/Resources/Chibios/Chibios/Debug/libChibios.a(chcoreasm_v7m.o).
The map file contains no indication that the port_lock function is linked multiple times. It contains no
indication that this function is linked at all. If it were linked multiple times then there
would have been a multiple-definition linkage error (except in the event that it had
been annotated as a weak symbol).
Why these six 28-byte input sections containing no symbols are all linked, or if they
need to be, is a matter about which I have no adequate evidence or ChibiOS expertise. I notice
that all but one of the object files from which these input sections come are
archive members of libChibios. In that light, it is worth remembering that if your linkage
requires an archive member for any reason then by default you will link the whole
archive member, even it contains more stuff than you need. On the other hand, the fact
that some port_lock input sections are discarded and some are kept suggests that there
is a need to keep the ones that are kept. If for my own cunning reasons I write a source file
essentially like:
static int __attribute__((section(".text.foo"))) __attribute__((used))
boo(int i)
{
return i * 2;
}
int bar(int i)
{
return boo(i);
}
then in my map file you will see an empty input section called .text.foo. This
doesn't tell you anything about the symbols I'm linking.
How can I tell the toolchain to only link once each function?
The linker will not link any symbol definition more than once, except in the special
case of weak symbols. Your map file contains no evidence of any function being linked more than once.
How can I reduce the final file size?
Compile with -Os for your release, of course. And to minimize linkage redundancy,
see this question.
Reading a linker map file is usually a clunky way of investigating the symbols and sections
in your binaries. Prefer objdump, readelf
and nm
I'm playing around with ASM, byte code and executable memory.
I'm using echo -e "<asm instruction here>" | as -o /dev/null -64 -al -msyntax=att to "compile" my code. Anyways, I noticed that movq 2, %rax will result in 0000 488B0425 02000000, but both call printk and call printf will result in 0000 E8000000 00, which I believe is because of the fact that as doesn't know anything about printk or printf.
My question is how can I get the address of those functions so I can use them in my byte code?
The assembler does not know the values of external symbols. It writes a file with multiple sets of data in it. One set of data is the bytes to put into a program’s text section. Another set of data is information about external symbols used and what adjustments to make to the bytes in the text section when those external symbols are resolved. The file produced by the assembler does not contain any information about the values of symbols that come from external libraries.
The linker resolves external symbols. When producing a statically linked executable, it will fully resolve the external symbols. When producing dynamically linked executable, resolution of external symbols is completed during program execution.
After you statically link an executable, you can use a tool such as nm to see the values of symbols in it (if they have not been stripped). Then you could use those values in assembly. However, hardcoding those values into assembly will fail if the program is ever linked in a different way, that puts the routines at different locations.
If you dynamically link an executable, you would have to inspect the executing program to determine where its external symbols were placed. Again, you could in theory use those values in assembly, but this would be even more prone to failure than doing so with static linking, especially on systems that deliberately change the addresses used in executables to thwart malicious code.
Hard-coding symbol values into assembly is not the normal way of writing assembly language or other code and should not be done except for learning or special-purpose debugging or system inspection.
Use the proper tool to inspect your object files. Assuming any Unix, use
nm -AP file.o
to see the addresses of all symbols (functions and variables). Since printf is likely in a library, run nm on that library for the same effect.