what's difference between --start-group and --whole-archive in ld - linker

Question

To be honest, I think this question should be easy by staring at man ld. However, by reading manpage and read the code done by others, I find people use them interchangeably or at the same time when they think there might be an issue with the order of the libraries passed to the linker.
I'm wondering what's the difference between these two options and what's the best practice when using them.
Thanks!
Related links:
What are the --start-group and --end-group command line options?

Answer 1

At the time of writing, the Stackoverflow tag wiki on static libraries
tells us:
A static library is an archive of object files. Used as a linker input, the linker extracts the object files it needs to carry on the linkage.
The needed object files are those that provide the linker with definitions for symbols that it finds are used, without definition, in other input files.
The needed object files, and no others, are extracted from the archive and input to the linkage exactly as if they were individual input files in the
linkage command and the static library was not mentioned at all.
...
A linker will normally support an option (GNU ld: --whole-archive, MS link: /WHOLEARCHIVE) to override the
default processing of static libraries and instead link all the contained object files, whether they are needed or not.
A static library contributes nothing to a linkage except the object files that are extracted from it, which may be vary in different linkages.
It is to be contrasted with a shared library, another kind of file altogether with a very different role in linkage.
That should make it clear what --whole-archive does. The scope of --whole-archive continues until the end of
the linker commandline or until --no-whole-archive appears1.
By default, the linker will examine a static library only at each point where that library occurs in the commandline sequence of linker inputs.
It will not go backwards to rexamine a static library for the sake of resolving symbol references it discovers in later inputs.
The --start-group ... --end-group pair of options changes that default behaviour. It directs the linker to examine the static libraries
mentioned in ... repeatedly, in that order, for as long doing so yields any new resolutions of new symbol references. --start-group ... --end-group has no effect on
the linker's default selection of object files from the static libraries in .... It will extract and link only the object files that it needs, unless --whole-archive is also
in effect.
Summing that up:-
--start-group lib0.a ... libN.a --end-group tells the linker: Keep searching through lib0.a ... libN.a for object files you need until you don't find any more.
--whole-archive lib0.a ... libN.a --no-whole-archive tells the linker: Forget about what you need. Just link all the object files in all of lib0.a ... libN.a.
You can see then that any linkage you can make succeed with --start-group lib0.a ... libN.a --end-group will also succeed with --whole-archive lib0.a ... libN.a --no-whole-archive,
because the latter will link all of the necessary object files and all the unnecessary ones, without bothering to tell the difference.
But the converse isn't true. Here is a trivial example:
x.c
#include <stdio.h>
void x(void)
{
puts(__func__);
}
y.c
#include <stdio.h>
void y(void)
{
puts(__func__);
}
main.c
extern void x(void);
int main(void)
{
x();
return 0;
}
Compile all source files:
$ gcc -Wall -c x.c y.c main.c
Make a static library, archiving x.o and y.o:
ar rcs libxy.a x.o y.o
Attempt to link a program with main.o and libxy.a in the wrong order:
$ gcc -o prog libxy.a main.o
main.o: In function `main':
main.c:(.text+0x5): undefined reference to `x'
collect2: error: ld returned 1 exit status
That failed because the reference to x in main.o was only discovered
by the linker too late to find the definition of x in libxy.a(x.o). It reached libxy.a
first and found no object files that it needed. At that point, it hadn't yet linked
any object files at all into the program, so there were 0 symbol references it needed to resolve. Having
considered libxy.a and found no use for it, it does not consider it again.
The correct linkage, of course, is:
$ gcc -o prog main.o libxy.a
but if you don't realise that you simply have the linkage order back-to-front, you can
get the linkage to succeed with --whole-archive:
$ gcc -o prog -Wl,--whole-archive libxy.a -Wl,--no-whole-archive main.o
$ ./prog
x
Obviously, you can't get it to succeed with
$ gcc -o prog -Wl,--start-group libxy.a -Wl,--end-group main.o
main.o: In function `main':
main.c:(.text+0x5): undefined reference to `x'
collect2: error: ld returned 1 exit status
because that is no different from:
$ gcc -o prog libxy.a main.o
Now here's an example of a linkage that fails with the default behaviour, but can be
made to succeed with --start-group ... --end-group.
a.c
#include <stdio.h>
void a(void)
{
puts(__func__);
}
b.c
#include <stdio.h>
void b(void)
{
puts(__func__);
}
ab.c
extern void b(void);
void ab(void)
{
b();
}
ba.c
extern void a(void);
void ba(void)
{
a();
}
abba.c
extern void ab(void);
extern void ba(void);
void abba(void)
{
ab();
ba();
}
main2.c
extern void abba(void);
int main(void)
{
abba();
return 0;
}
Compile all sources:-
$ gcc -Wall a.c b.c ab.c ba.c abba.c main2.c
Then make the following static libaries:
$ ar rcs libbab.a ba.o b.o x.o
$ ar rcs libaba.a ab.o a.o y.o
$ ar rcs libabba.a abba.o
(Notice that those old object files x.o and y.o are archived here again).
Here, libabba.a is dependent on both libbab.a and libaba.a. Specifically,
libabba.a(abba.o) makes a reference to ab, which is defined in libaba.a(ab.o);
and it also makes a reference to ba, which is defined in libbab.a(ba.o).
So in the linkage order, libabba.a must occur before either libbab.a and libaba.a
And libbab.a is dependent on libaba.a. Specifically, libbab.a(ba.o) makes a
reference to a, which is defined in libaba(a.o).
But libaba.a is also dependent on libbab.a. libaba(ab.o) makes a reference
to b, which is defined in libbab(b.o). There is a circular dependency between
libbab.a and libaba.a. So whichever one of them we place first in a default linkage, it
will fail with undefined reference errors. Either this way:
$ gcc -o prog2 main2.o libabba.a libaba.a libbab.a
libbab.a(ba.o): In function `ba':
ba.c:(.text+0x5): undefined reference to `a'
collect2: error: ld returned 1 exit status
Or that way:
$ gcc -o prog2 main2.o libabba.a libbab.a libaba.a
libaba.a(ab.o): In function `ab':
ab.c:(.text+0x5): undefined reference to `b'
collect2: error: ld returned 1 exit status
A circular dependency is a problem to which --start-group ... --end-group is the solution:
$ gcc -o prog2 main2.o libabba.a -Wl,--start-group libbab.a libaba.a -Wl,--end-group
$ ./prog2
b
a
Therefore --whole-archive ... --no-whole-archive is also a solution:
$ gcc -o prog2 main2.o libabba.a -Wl,--whole-archive libbab.a libaba.a -Wl,--no-whole-archive
$ ./prog2
b
a
But it is a bad solution. Let's trace which of our object files are actually linked
into the program in each case.
With --start-group ... --end-group:
$ gcc -o prog2 main2.o libabba.a -Wl,--start-group libbab.a libaba.a -Wl,--end-group -Wl,--trace
/usr/bin/x86_64-linux-gnu-ld: mode elf_x86_64
/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/Scrt1.o
/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/crti.o
/usr/lib/gcc/x86_64-linux-gnu/7/crtbeginS.o
main2.o
(libabba.a)abba.o
(libbab.a)ba.o
(libaba.a)ab.o
(libaba.a)a.o
(libbab.a)b.o
libgcc_s.so.1 (/usr/lib/gcc/x86_64-linux-gnu/7/libgcc_s.so.1)
/lib/x86_64-linux-gnu/libc.so.6
(/usr/lib/x86_64-linux-gnu/libc_nonshared.a)elf-init.oS
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
libgcc_s.so.1 (/usr/lib/gcc/x86_64-linux-gnu/7/libgcc_s.so.1)
/usr/lib/gcc/x86_64-linux-gnu/7/crtendS.o
/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/crtn.o
they are:
main2.o
(libabba.a)abba.o
(libbab.a)ba.o
(libaba.a)ab.o
(libaba.a)a.o
(libbab.a)b.o
which are exactly the ones that are needed in the program.
With --whole-archive ... --no-whole-archive:
$ gcc -o prog2 main2.o libabba.a -Wl,--whole-archive libbab.a libaba.a -Wl,--no-whole-archive -Wl,-trace
/usr/bin/x86_64-linux-gnu-ld: mode elf_x86_64
/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/Scrt1.o
/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/crti.o
/usr/lib/gcc/x86_64-linux-gnu/7/crtbeginS.o
main2.o
(libabba.a)abba.o
(libbab.a)ba.o
(libbab.a)b.o
(libbab.a)x.o
(libaba.a)ab.o
(libaba.a)a.o
(libaba.a)y.o
libgcc_s.so.1 (/usr/lib/gcc/x86_64-linux-gnu/7/libgcc_s.so.1)
/lib/x86_64-linux-gnu/libc.so.6
(/usr/lib/x86_64-linux-gnu/libc_nonshared.a)elf-init.oS
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
libgcc_s.so.1 (/usr/lib/gcc/x86_64-linux-gnu/7/libgcc_s.so.1)
/usr/lib/gcc/x86_64-linux-gnu/7/crtendS.o
/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/crtn.o
they are:
main2.o
(libabba.a)abba.o
(libbab.a)ba.o
(libbab.a)b.o
(libbab.a)x.o
(libaba.a)ab.o
(libaba.a)a.o
(libaba.a)y.o
The same as before, plus:
(libbab.a)x.o
(libaba.a)y.o
which are dead code (and there could have been more of it, without limit). The redundant functions x() and y() are defined in the image:
$ nm prog2 | egrep 'T (x|y)'
000000000000067a T x
00000000000006ac T y
Takeaway
Use default linkage, putting your inputs in dependency order, if you can.
Use --start-group ... --end-group to surmount circular dependencies between
libraries, when you can't fix the libraries. Be aware that linkage speed will take a hit.
Use --whole-archive ... --no-whole-archive only if you actually need to link
all the object files in all the static libraries in .... Otherwise, do one of
the previous two things.
[1] And be aware that the linker's commandline, when the linker is invoked
by GCC, is actually much longer than the linkage options you explicitly pass in
the GCC commandline, with boiler-plate options silently appended. Therefore
always close --whole-archive with --no-whole-archive, and close --start-group
with --end-group.