As I understand it the GCC compiler performs four steps when I compile a C program.
Preprocessing - C code (*.c) with macros to C code without macros (*.c)
Compiling - C code (*.c) to Assembly language (*.s)
Assembling - Assembly language (*.s) to Object code (*.o)
Linking - Object code (*.o) to executable (*)
The first three steps make perfect sense to me, but I am still confused as to what linking actually does.
After step three why can't I run the *.o file? At that point my C code is now in object/machine/byte code and can be interpreted by the CPU directly. Yet when I make my *.o file executable and try to run it I get this error:
bash: ./helloworld.o: cannot execute binary file: Exec format error
Why do I get this error? If I have a tiny C program (for example a hello world program) with only one C file it would appear to me that linking has no purpose because there's nothing to link. So what does linking in the compilation process actually do?
Thanks in advance for any replies.
If I have a tiny C program (for example a hello world program)
Even your helloworld program does use #inlude<stdio.h>, doesn't it? That means you're using a library, and the linking step is there to combine the necessary object code (here the library code) to create a binary for you.
For a detailed descriptions of what the linking step does (and compare with compiling) - see this question
Linking in rough explanation is:
Find all the matching segments from each object file, and concat them together. This way we end up with one large .code, one .data, one .bss etc.
Resolve all symbols that are used. Many symbols are local, so that they can be resolved immediately. Unresolved symbols will be searched for in the libraries requested to link with. When this is done, the result will be a symbol table / link map.
Make an file that is actually executable. On Linux, it usually just happens that both executable, libraries and object files all are in the ELF format. This is not true for all platforms.
The simple answer is that .o executables serve different purposes and have a different format.
If you want the complete answer you will need to read the necessary documentation for your platforms binary format.
On linux this will be here. This document will describe the difference between the intermediate format and the final executable format.
Just as an aside the linux kernel module loader does use .o (or rather .ko) files directly.
Related
I would like to know if someone is aware of a trick to retrieve the list of files that had been (or ideally will be) used by linker to produce an executable.
Some kind of solution must exist. A a static source analyzer, or a hack, such as compiling with some weird flags, and analyzing produced executable with another tool, or force the linker to output this information.
The goal is to provide a tool that strip useless source files from a list of source files.
The end goal is to ease the build process, by allowing him to give a list of usable source files. Then my tool would only compile the ones actually used by linker instead of everything.
This would allow for some unit_test to still be runnable even if some others are broken and can't compile, while not asking the user to manually list every test dependencies manually in the cmake.
I am targetting linux for now, but will be intersted in the futur to do the same trick on others OS. So I would like a cross-platform solution, eventhought I doubt I will have it :)
Thanks for your help
Edit because I see that it is confusing, what I mean by
allowing him to give a list of usable source file
is that, in cmake, for exemple. If you use add_executable(name, sources), then sources is considered as the sources to compile and link on.
I want to wrap add_executable, so sources is viewed as a set of usable if necessary sources files.
I'm afraid the idea of detecting never linked source files is not a fruitful one.
To build a program, CMake will not compile a source file if it not going to link the resulting object
file into the program. I can understand how you might think that this happens, but it doesn't happen.
CMake already does what you would like it to do and the same is true of every other build automation system going back to
their invention in the 1970s. The fundamental purpose of all
such systems is to ensure that the building of a program
compiles a source file name.(c|cc|f|m|...) if and only if
the object file name.o is going to be linked into the program
and is out of date or does not exist. You can always defeat this purpose by
egregiously bad coding of the project's build spec (CMakeLists.txt, Makefile, SConstruct, etc.),
but with CMake you would need to be really trying to do it, and
trying quite expertly.
If you do not want name.c to be compiled and the object file name.o
linked into a target program, then you do not tell the build system
that name.o or name.c is a prerequisite of the program. Don't tell
it what you know is not true. It is elementary competence not to specify redundant prerequisites of
a build system target.
The linker will link all its input object files into an output
program without question. It does not ask whether or not they are "needed"
by the program because it cannot answer that question. Neither the
linker nor any possible static analysis tool can know what program
you intend to produce when you input some object files for linkage.
It can only be assumed that you intend to produce the program that
results from the linkage of those object files, assuming the
linkage is successful.
If those object files cannot be linked into a program at all, the linker will tell you
that, and why. Otherwise, if you have linked object files that you didn't
intend to link, you can only discover that for yourself, by noticing
the mistake in the build log, or failing that by testing the program and/or inspecting its contents and comparing
your observations with your expectations.
Given your choice of object files for linkage, you can instruct the linker
to detect any code sections or data sections it extracts those object files in
which no symbols are defined that can be referenced by the program, and to
throw away all such unreferenced input sections instead of linking them
into the program. This is called linktime "garbage collection". You tell the
linker to do it by passing the option -Wl,-gc-sections in the
gcc linkage command. See this question
to learn how to maximise the collectible garbage. This is what you
can do to remove redundant object code from the linkage.
But you can only collect any garbage from a program in this way if the program
is dynamically opaque, i.e not linked with the option -rdynamic
: then the global symbols defined in the program's static image are not visible
to the OS loader and cannot be referenced from outside its static image by dynamic
libraries in the same process. In this case the linker can determine by static
analysis that a symbol whose definition is not referenced in the program's static
image cannot be referenced at all, since it cannot be referenced dynamically,
and if all symbols defined in an input section are statically unreferenced then
it can garbage-collect the section.
If the program has been linked -rdynamic then -Wl,-gc-sections will
collect no garbage, and this is quite right, because if the program is
not dynamically opaque then it is impossible for static analysis to determine that anything
defined in its linkage cannot be referenced.
It's noteworthy that although -rdynamic is not a default linkage
option for GCC, it is a default linkage option for CMake projects using
the GCC toolchain. So to use linktime garbage collection in CMake projects
you would always have to override the -rdynamic default. And obviously it would only be
valid to do this if you have determined that it is alright for the program to
be dynamically opaque.
So I have been studying this Modular programming that mainly compiles each file of the program at a time. Say we have FILE.c and OTHER.c that both are in the same program. To compile it, we do this in the prompt
$gcc FILE.c OTHER.c -c
Using the -c flag to compile it into .o files (FILE.o and OTHER.o) and only when that happens do we translate it (compile) to executable using
$gcc FILE.o OTHER.o -o
I know I can just do it and skip the middle part but as it shows everywhere, they do it first and then they compile it into executable, which I can't understand at all.
May I know why?
If you are working on a project with several modules, you don't want to recompile all modules if only some of them have been modified. The final linking command is however always needed. Build tools such as make is used to keep track of which modules need to be compiled or recompiled.
Doing it in two steps allows to separate more clearly the compiling and linking phases.
The output of the compiling step is object (.o) files that are machine code but missing the external references of each module (i.e. each c file); for instance file.c might use a function defined in other.c, but the compiler doesn't care about that dependency in that step;
The input of the linking step is the object files, and its output is the executable. The linking step bind together the object files by filling the blanks (i.e. resolving dependencies between objets files). That's also where you add the libraries to your executable.
This part of another answer responds to your question:
You might ask why there are separate compilation and linking steps.
First, it's probably easier to implement things that way. The compiler
does its thing, and the linker does its thing -- by keeping the
functions separate, the complexity of the program is reduced. Another
(more obvious) advantage is that this allows the creation of large
programs without having to redo the compilation step every time a file
is changed. Instead, using so called "conditional compilation", it is
necessary to compile only those source files that have changed; for
the rest, the object files are sufficient input for the linker.
Finally, this makes it simple to implement libraries of pre-compiled
code: just create object files and link them just like any other
object file. (The fact that each file is compiled separately from
information contained in other files, incidentally, is called the
"separate compilation model".)
It was too long to put in a comment, please give credit to the original answer.
Don't get me wrong by looking at the question title - I know what they are (format for portable executable files). But my interest scope is slightly different
MY CONFUSION
I am involved in re-hosting/retargeting applications that are originally from third parties. The problem is that sometimes the formats for object codes are also in .elf, .COFF formats and still says, "Executable and linkable".
I am primarily a Windows user and know that when you compile and assemble your C/C++ code, you get something similar to .o or .obj. that are not executable (well, I never tried to execute them). But when you complete linking static and dynamic libraries and finish building, the executable appears. My understanding is that you can then go about and link that executable or "bash" test it with some form of script if necessary.
However, in Linux (or UNIX-like systems) there are .o files after you compile and assemble the C/C++ code. And once the linking is done, the executable is in a.out format (at least in Ubuntu distribution of Linux). It may very well be .elf in some other distrib. In my quick web search none of the sources mentioned anything about .o files as executables.
QUESTIONS
Therefore my question turns into the followings:
What is the true definitions for portable executables and object code?
How is it that Windows and UNIX platform covers both executables annd object code under the same file format (.COFF, .elf).
Am I misinterpreting "Linkable"? My interpretation of "Linkable" is something that is compiled object code and can then be "linked" to other static/dynamic link libraries. Is this a stupid thought?
Based on question 1. (and perhaps 2) do I need to use symbol tables (e.g. .LUM or .MAP files) with object code then? Symbols as in debug symbols and using them when re-hosting the executables/object files on a different machine.
Thanks in advance for the right nudges. Meanwhile, I will keep digging and update the question if necessary.
UPDATE
I have managed to dig this out from somewhere :( Seems like a lot to swallow to me.
I am primarily a Windows user and know that when you compile your C/C++ code, you get something similar to .o or .obj. that are not executable
Well, last time I compiled stuff on Windows, the result of the compilation was an .obj file, which is exactly what its name suggests: it's an object file. You're right in that it's not an executable in itself. It contains machine code which doesn't (yet) contain enough information to be directly run on the CPU.
However, in Linux (or UNIX-like systems) there are .o files after you compile the C/C++ code. And once the linking is done, the executable is in a.out format (at least in Ubuntu distribution of Linux). It may very well be .elf in some other distrib.
Living in the 90's, that is :P No modern compilers I am aware of target the a.out format as their default output format for object code. Maybe it's a misleading default of GCC to put the object code into a file called a.out when no explicit output file name is specified, but if you run the file command on a.out, you'll find out that it's an ELF file. The a.out format is ancient and it's kind of "de facto obsolete".
What is the true definitions for portable executables and object code?
You've already got the Wikipedia link to object files, here's the one to "Portable Executable".
How is it that Windows and UNIX platform covers both executables annd object code under the same file format (.COFF, .elf).
Because the ELF format (and apparently COFF too) has been designed like so. And why not? It's just the very same machine code after all, it seems quite logical to use one file format during all the compilation steps. Just like we don't like when dynamic libraries and stand-alone executables have a different format. (That's why ELF is called ELF - it's an "Executable and Linkable Format".)
Am I misinterpreting "Linkable"?
I don't know. From your question it's not clear to me what you think "linkable" is. In general, it means that it's a file that can be linked against, i. e. a library.
Based on question 1. (and perhaps 2) do I need to use symbol tables (e.g. .LUM or .MAP files) with object code then? Symbols as in debug symbols and using them when re-hosting the object files on a different machine.
I think this one is not related to the executable format used. If you want to debug, you have to generate debugging information no matter what. But if you don't need to debug, then you're free to omit them of course.
I am reading about libraries in C but I have not yet found an explanation on what an object file is. What's the real difference between any other compiled file and an object file?
I would be glad if someone could explain in human language.
An object file is the real output from the compilation phase. It's mostly machine code, but has info that allows a linker to see what symbols are in it as well as symbols it requires in order to work. (For reference, "symbols" are basically names of global objects, functions, etc.)
A linker takes all these object files and combines them to form one executable (assuming that it can, i.e.: that there aren't any duplicate or undefined symbols). A lot of compilers will do this for you (read: they run the linker on their own) if you don't tell them to "just compile" using command-line options. (-c is a common "just compile; don't link" option.)
An Object file is the compiled file itself. There is no difference between the two.
An executable file is formed by linking the Object files.
Object file contains low level instructions which can be understood by the CPU. That is why it is also called machine code.
This low level machine code is the binary representation of the instructions which you can also write directly using assembly language and then process the assembly language code (represented in English) into machine language (represented in Hex) using an assembler.
Here's a typical high level flow for this process for code in High Level Language such as C
--> goes through pre-processor
--> to give optimized code, still in C
--> goes through compiler
--> to give assembly code
--> goes through an assembler
--> to give code in machine language which is stored in OBJECT FILES
--> goes through Linker
--> to get an executable file.
This flow can have some variations for example most compilers can directly generate the machine language code, without going through an assembler. Similarly, they can do the pre-processing for you. Still, it is nice to break up the constituents for a better understanding.
There are 3 kind of object files.
1. Relocatable object files:
Contain machine code in a form that can be combined with other relocatable object files at link time, in order to form an executable object file.
If you have an a.c source file, to create its object file with GCC you should run:
gcc a.c -c
The full process would be:
preprocessor (cpp) would run over a.c
Its output (still source; cpp) will feed into the compiler (cc1).
Its output (assembly) will feed into the assembler (as)
assembler (as) will produce the relocatable object file.
That relocatable object file contains:
object code, and metadata for linking, and debugging (if -g was used)
it is not directly executable.
2. Shared object files:
Special type of relocatable object file that can be loaded dynamically, either at load time, or at run time.
Shared libraries are an example of these kinds of objects.
3. Executable object files:
Contain machine code that can be directly loaded into memory (by the loader, e.g execve) and subsequently executed.
The result of running the linker over multiple relocatable object files is an executable object file. The linker merges all the input object files from the command line, from left-to-right, by merging all the same-type input sections (e.g. .data) to the same-type output section. It uses symbol resolution and relocation.
Bonus: Static vs Dynamic Libraries
When linking against a static library the functions that are referenced in the input objects are copied to the final executable.
With dynamic libraries a symbol table is created instead that will enable a dynamic linking with the library's functions/globals. Thus, the result is a partially executable object file, as it depends on the library. If the library doesn't exist, the file can no longer execute.
The linking process can be done as follows:
ld a.o -o myexecutable
The command: gcc a.c -o myexecutable will invoke all the commands mentioned at point 1 and at point 3 (cpp -> cc1 -> as -> ld1)
1: actually is collect2, which is a wrapper over ld.
An object file is just what you get when you compile one (or several) source file(s).
It can be either a fully completed executable or library, or intermediate files.
The object files typically contain native code, linker information, debugging symbols and so forth.
Object files are codes that are dependent on functions, symbols, and text to run the program. Just like old telex machines, which required teletyping to send signals to other telex machine.
In the same way processor's require binary code to run, object files are like binary code but not linked. Linking creates additional files so that the user does not have to have compile the C language themselves. Users can directly open the exe file once the object file is linked with some compiler like c language , or vb etc.
What is the difference between a compiler and a linker in C?
The compiler converts code written in a human-readable programming language into a machine code representation which is understood by your processor. This step creates object files.
Once this step is done by the compiler, another step is needed to create a working executable that can be invoked and run, that is, associate the function calls (for example) that your compiled code needs to invoke in order to work. For example, your code could call sprintf, which is a routine in the C standard library. Your code has nothing that does the actual service provided by sprintf, it just reports that it must be called, but the actual code resides somewhere in the common C library. To perform this (and many others) linkages, the linker must be invoked. After linking, you obtain the actual executable that can run.
A compiler generates object code files (machine language) from source code.
A linker combines these object code files into an executable.
Many IDEs invoke them in succession, so you never actually see the linker at work. Some languages/compilers do not have a distinct linker and linking is done by the compiler as part of its work.
In Simple words -> Linker comes into act whenever a '.obj' file needs to be linked with its library functions as compiler doesn't understand what is (scanf or printf..etc) , compiler just converts '.c' file to '.obj' file if there's no error without understanding library functions we used. So To make 'obj' file to 'exe'(executable file) we need linker because it makes compiler understand of library functions.