I've recently compiled a simple hello world C program under Debian Linux using gcc:
gcc -mtune=native -march=native -m32 -s -Wunused -O2 -o hello hello.c
The file size was 2980 bytes. I opened it in a hex editor and i saw the following lines:
GCC: (Debian 4.4.5-8) 4.4.5 GCC: (Debian 4.4.5-10) 4.4.5 .shstrtab .interp .note.ABI-tag .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rel.dyn .rel.plt .init .text .fini .rodata .eh_frame .ctors .dtors .jcr .dynamic .got .got.plt data.data .bss .comment
Are they really needed? No way to reduce executable size?
use -Qn to avoid that.
aa$ touch hello.c
aa$ gcc -c hello.c
aa$ objdump -s hello.o
hello.o: file format elf32-i386
Contents of section .comment:
0000 00474343 3a202844 65626961 6e20342e .GCC: (Debian 4.
0010 372e322d 35292034 2e372e32 00 7.2-5) 4.7.2.
aa$ gcc -Qn -c hello.c
aa$ objdump -s hello.o
hello.o: file format elf32-i386
aa$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/i486-linux-gnu/4.7/lto-wrapper
Target: i486-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.7.2-5' --with-bugurl=file:///usr/share/doc/gcc-4.7/README.Bugs --enable-languages=c,c++,go,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.7 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.7 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --enable-targets=all --with-arch-32=i586 --with-tune=generic --enable-checking=release --build=i486-linux-gnu --host=i486-linux-gnu --target=i486-linux-gnu
Thread model: posix
gcc version 4.7.2 (Debian 4.7.2-5)
aa$
That's in a comment section in the ELF binary. You can strip it out:
$ gcc -m32 -O2 -s -o t t.c
$ ls -l t
-rwxr-xr-x 1 me users 5488 Jun 7 11:58 t
$ readelf -p .comment t
String dump of section '.comment':
[ 0] GCC: (Gentoo 4.5.1-r1 p1.4, pie-0.4.5) 4.5.1
[ 2d] GCC: (Gentoo 4.5.2 p1.1, pie-0.4.5) 4.5.2
$ strip -R .comment t
$ readelf -p .comment t
readelf: Warning: Section '.comment' was not dumped because it does not exist!
$ ls -l t
-rwxr-xr-x 1 me users 5352 Jun 7 11:58 t
The gains are tiny though, not sure it's worth it.
This is in a comment section which isn't loaded in memory (and note that ELF files usually use padding so that memory mapping them will keep a correct alignment). If you want to get rid of such unneeded sections, see the various objcopy options and find out:
objcopy --remove-section .comment a.o b.o
I had the same issue myself, but using MinGW's GCC implementation - stripping the executable and passing the -Qn option did nothing, and I couldn't remove the ".comment" section as there wasn't one.
In order to stop the compiler including this information, regardless of which sections are in your executable, you can pass the -fno-ident parameter to the compiler and linker:
Without the parameter (strings -a [filename]):
!This program cannot be run in DOS mode.
.text
0`.rdata
0#.idata
GCC: (tdm64-2) 4.8.1
With the parameter:
!This program cannot be run in DOS mode.
.text
0`.idata
It appears that you'd be able to 'just' strip that if you don't want it; See this page for a nice run-down.
http://timelessname.com/elfbin/
Note that the page (of course) also resorts to using assembly, which you may not want to do, but the general gist applies
You can inform the loader which sections to include in your output with a linker script. You can see what sections are included in the file using the objdump command. As you've noticed there's a good bit of 'junk' in an elf - junk that is until you wish you had it.
Note though, that the size of an elf executable file is not indicative of the memory foot print of the image as realized in memory. A lot of the 'junk' isn't in the memory image and the image can call sbreak and or mmap to acquire more memory, the elf file takes no account of stack usage - essentially all of your automatic variables are unaccounted for. These are only three examples others abound.
Related
Can anyone explain why gcc is adding the .comment and .note.gnu.property sections in the object code, and how can I tell to gcc not to add them, is it possible?
Thanks.
why gcc is adding the .comment and .note.gnu.property sections
You can examine the contents of the .comment section with e.g. readelf -x.comment:
echo "int foo() { return 42; }" | gcc -xc - -c -o foo.o
readelf -x.comment foo.o
Hex dump of section '.comment':
0x00000000 00474343 3a202844 65626961 6e203131 .GCC: (Debian 11
0x00000010 2e322e30 2d313029 2031312e 322e3000 .2.0-10) 11.2.0.
Obviously the "why" is to make it easier to understand what compiler produced the object file.
I don't believe there is a GCC flag to suppress this, but objcopy --remove-section .comment foo.o bar.o will get rid of it.
The .note.gnu.property can be removed in similar fashion.
Here is a discussion of what it may contain.
tks makred it, objcopy -O binary -R .note -R .comment -R .note.gnu.property foo , elf format to binary, size from 129M to 4K
If I have the following program written in C (compiled with GCC on Debian 8.7), I am able to call atexit() as you would expect:
#include <stdlib.h>
void exit_handler(void) {
return;
}
int main () {
atexit(exit_handler);
return 0;
}
And when I compile and run it:
$ gcc test.c
$ ./a.out
Outputs nothing, just as you would expect. In fact, when I run ldd, I get:
$ ldd a.out
linux-vdso.so.1 (0x00007fffbe592000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe07d3a8000)
/lib64/ld-linux-x86-64.so.2 (0x00007fe07d753000)
However, libc does not seem to have any symbols for atexit, amd only has__cxa_atexit and __cxa_threaded_atexit_impl:
$ nm --dynamic /lib/x86_64-linux-gnu/libc.so.6 | grep 'atexit'
0000000000037d90 T __cxa_atexit
0000000000037fa0 T __cxa_thread_atexit_impl
As you would then expect, if I try to link to libc dynamically, I cannot actually call atexit(), such as in the following Racket program which links to libc and tries to find atexit:
#lang racket
(require ffi/unsafe)
(get-ffi-obj 'atexit (ffi-lib "libc" '("6")) (_fun (_fun -> _void) -> _int))
Giving the output:
$ racket findatexit.rkt
ffi-obj: couldn't get "atexit" from "libc.so.6" (/lib/x86_64-linux-gnu/libc.so.6: undefined symbol: atexit)
What I want to know here is:
If libc does not have any symbol for atexit on Linux, why can I still call it from a C program?
Is there any way I can call atexit or a similar function dynamically on Linux?
(I should note that atexit does appear to be a symbol on OS X, so its just Linux that seems unusual here.)
Edit:
At the suggestion of #Jonathan, I also ran:
$ gcc -c test.c
$ nm test.o
U atexit
0000000000000000 T exit_handler
0000000000000007 T main
Which seems to indicate the atexit symbol is there somewhere, but it does not appear in any of the libraries ldd is showing.
I did some poking around on a Centos 7 virtual machine, and I think I found it — but it was anything but obvious!
Found it!
In /usr/lib64/libc_nonshared.a:
$ nm /usr/lib64/libc_nonshared.a | grep -i atexit
atexit.oS:
0000000000000000 T atexit
U __cxa_atexit
$
Why look in that library? Good question — and a long story. Are you sitting comfortably? Then I'll begin…
Steps taken to get there
Use the test.c code from the question.
Compile it with gcc -v test.c:
$ gcc -v test.c
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC)
COLLECT_GCC_OPTIONS='-v' '-mtune=generic' '-march=x86-64'
/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/cc1 -quiet -v test.c -quiet -dumpbase test.c -mtune=generic -march=x86-64 -auxbase test -version -o /tmp/ccPHTer7.s
GNU C (GCC) version 4.8.5 20150623 (Red Hat 4.8.5-11) (x86_64-redhat-linux)
compiled by GNU C version 4.8.5 20150623 (Red Hat 4.8.5-11), GMP version 6.0.0, MPFR version 3.1.1, MPC version 1.0.1
GGC heuristics: --param ggc-min-expand=96 --param ggc-min-heapsize=124992
ignoring nonexistent directory "/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include-fixed"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../x86_64-redhat-linux/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/include
/usr/local/include
/usr/include
End of search list.
GNU C (GCC) version 4.8.5 20150623 (Red Hat 4.8.5-11) (x86_64-redhat-linux)
compiled by GNU C version 4.8.5 20150623 (Red Hat 4.8.5-11), GMP version 6.0.0, MPFR version 3.1.1, MPC version 1.0.1
GGC heuristics: --param ggc-min-expand=96 --param ggc-min-heapsize=124992
Compiler executable checksum: 356f86e67978d665416e07d560c8ba0d
COLLECT_GCC_OPTIONS='-v' '-mtune=generic' '-march=x86-64'
as -v --64 -o /tmp/cc5WHEA4.o /tmp/ccPHTer7.s
GNU assembler version 2.25.1 (x86_64-redhat-linux) using BFD version version 2.25.1-22.base.el7
COMPILER_PATH=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/:/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/:/usr/libexec/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/4.8.5/:/usr/lib/gcc/x86_64-redhat-linux/
LIBRARY_PATH=/usr/lib/gcc/x86_64-redhat-linux/4.8.5/:/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-mtune=generic' '-march=x86-64'
/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/collect2 --build-id --no-add-needed --eh-frame-hdr --hash-style=gnu -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/crt1.o /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/crti.o /usr/lib/gcc/x86_64-redhat-linux/4.8.5/crtbegin.o -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../.. /tmp/cc5WHEA4.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-redhat-linux/4.8.5/crtend.o /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/crtn.o
$
The interesting part is the collect2 command line at the end. Written with one argument per line, that is:
/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/collect2
--build-id
--no-add-needed
--eh-frame-hdr
--hash-style=gnu
-m
elf_x86_64
-dynamic-linker
/lib64/ld-linux-x86-64.so.2
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/crt1.o
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/crti.o
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/crtbegin.o
-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5
-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64
-L/lib/../lib64
-L/usr/lib/../lib64
-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../..
/tmp/cc5WHEA4.o
-lgcc
--as-needed
-lgcc_s
--no-as-needed
-lc
-lgcc
--as-needed
-lgcc_s
--no-as-needed
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/crtend.o
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/crtn.o
So, there are a bunch of cr*.o files, plus three libraries: -lc, -lgcc and -lgcc_s to look for, and a bunch of directories to look in:
-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5, -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64, -L/lib/../lib64, -L/usr/lib/../lib64, -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../... The /tmp/cc5WHEA4.o is the object file created from test.c.
Applying some clean-up code to the path names, and then using ls to help find the libraries yields a list of files to examine further:
/lib64/ld-linux-x86-64.so.2
/usr/lib64/crt1.o
/usr/lib64/crti.o
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/crtbegin.o
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/crtend.o
/usr/lib64/crtn.o
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/libgcc.a
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/libgcc_s.so
/usr/lib64/libgcc_s.so.1
/lib64/libgcc_s.so.1
/usr/lib64/libgcc_s.so.1
/usr/lib64/libc.so
/usr/lib64/libc.so.6
/lib64/libc.so
/lib64/libc.so.6
/usr/lib64/libc.so
/usr/lib64/libc.so.6
That list of files was saved in a file yy (unimaginative name), and then used in:
$ nm -o $(<yy) | tee nm.log | grep -i atexit
nm: _trampoline.o: no symbols
nm: __main.o: no symbols
nm: _ctors.o: no symbols
nm: /usr/lib/gcc/x86_64-redhat-linux/4.8.5/libgcc_s.so: no symbols
nm: /usr/lib64/libgcc_s.so.1: no symbols
nm: /lib64/libgcc_s.so.1: no symbols
nm: /usr/lib64/libgcc_s.so.1: no symbols
nm: /usr/lib64/libc.so: File format not recognized
/usr/lib64/libc.so.6:00000000003bcc00 b added_atexit_handler.9157
/usr/lib64/libc.so.6:0000000000038c90 T __cxa_atexit
/usr/lib64/libc.so.6:0000000000038c90 t __cxa_atexit_internal
/usr/lib64/libc.so.6:00000000003b6838 d __elf_set___libc_atexit_element__IO_cleanup__
/usr/lib64/libc.so.6:0000000000038c40 t __internal_atexit
/usr/lib64/libc.so.6:00000000003b6838 d __start___libc_atexit
/usr/lib64/libc.so.6:00000000003b6840 d __stop___libc_atexit
nm: /lib64/libc.so: File format not recognized
/lib64/libc.so.6:00000000003bcc00 b added_atexit_handler.9157
/lib64/libc.so.6:0000000000038c90 T __cxa_atexit
/lib64/libc.so.6:0000000000038c90 t __cxa_atexit_internal
/lib64/libc.so.6:00000000003b6838 d __elf_set___libc_atexit_element__IO_cleanup__
/lib64/libc.so.6:0000000000038c40 t __internal_atexit
nm: /usr/lib64/libc.so: File format not recognized
/lib64/libc.so.6:00000000003b6838 d __start___libc_atexit
/lib64/libc.so.6:00000000003b6840 d __stop___libc_atexit
/usr/lib64/libc.so.6:00000000003bcc00 b added_atexit_handler.9157
/usr/lib64/libc.so.6:0000000000038c90 T __cxa_atexit
/usr/lib64/libc.so.6:0000000000038c90 t __cxa_atexit_internal
/usr/lib64/libc.so.6:00000000003b6838 d __elf_set___libc_atexit_element__IO_cleanup__
/usr/lib64/libc.so.6:0000000000038c40 t __internal_atexit
/usr/lib64/libc.so.6:00000000003b6838 d __start___libc_atexit
/usr/lib64/libc.so.6:00000000003b6840 d __stop___libc_atexit
$
There's no evidence of a plain atexit function there. Where's it hiding, and what's with those 'File format not recognized' messages?
$ file /usr/lib64/libc.so
/usr/lib64/libc.so: ASCII text
$
ASCII text? What?
$ cat /usr/lib64/libc.so
/* GNU ld script
Use the shared library, but some functions are only in
the static library, so try that secondarily. */
OUTPUT_FORMAT(elf64-x86-64)
GROUP ( /lib64/libc.so.6 /usr/lib64/libc_nonshared.a AS_NEEDED ( /lib64/ld-linux-x86-64.so.2 ) )
$
OK; what's in /usr/lib64/libc_nonshared.a?
$ nm /usr/lib64/libc_nonshared.a | grep -i atexit
atexit.oS:
0000000000000000 T atexit
U __cxa_atexit
$
Bingo! Found it!
So, it seems that the collect2 linker used by GCC is able to load files not listed on the command line, and that one of those files is /usr/lib64/libc_nonshared.a, and that this library has atexit() in it. Consequently, you should be able to invoke atexit() because it is statically linked into the executable … unless there's some more black magic hidden away here that I've not sussed out.
I'm trying to install LISTSERV by lsoft.com on redhat. The package comes with a compiler test, and this test is failing.
The OS is:
Red Hat Enterprise Linux Server release 6.8 (Santiago)
The Arch is:
x86_64
The compiler is:
gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-17)
Here's the full "gcc -v":
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-libgcj-multifile --enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC)
compilertest.c:
main()
{}
The compiler test command:
gcc -O compilertest.c
The error message:
/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../crt1.o: could not read symbols: File in wrong format
collect2: ld returned 1 exit status
The file exists as a real file (i.e. not a symbolic link):
-rw-r--r-- 1 root root 1240 Jan 28 2016 /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../crt1.o
So, what am I missing? Was something compiled using 32b instead of 64b? Am I missing a library or have the wrong version of one?
$ readelf -h /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../crt1.o
ELF Header:
Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: REL (Relocatable file)
Machine: Intel 80386
Version: 0x1
Entry point address: 0x0
Start of program headers: 0 (bytes into file)
Start of section headers: 288 (bytes into file)
Flags: 0x0
Size of this header: 52 (bytes)
Size of program headers: 0 (bytes)
Number of program headers: 0
Size of section headers: 40 (bytes)
Number of section headers: 13
Section header string table index: 10
$ env -i PATH=/usr/bin gcc -O compilertest.c
/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../crt1.o: could not read symbols: File in wrong format
collect2: ld returned 1 exit status
$ gcc -v compilertest.c
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-libgcj-multifile --enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib --with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC)
COLLECT_GCC_OPTIONS='-v' '-mtune=generic'
/usr/libexec/gcc/x86_64-redhat-linux/4.4.7/cc1 -quiet -v compilertest.c -quiet -dumpbase compilertest.c -mtune=generic -auxbase compilertest -version -o /tmp/ccGTjhJB.s
ignoring nonexistent directory "/usr/lib/gcc/x86_64-redhat-linux/4.4.7/include-fixed"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../x86_64-redhat-linux/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/local/include
/usr/lib/gcc/x86_64-redhat-linux/4.4.7/include
/usr/include
End of search list.
GNU C (GCC) version 4.4.7 20120313 (Red Hat 4.4.7-17) (x86_64-redhat-linux)
compiled by GNU C version 4.4.7 20120313 (Red Hat 4.4.7-17), GMP version 4.3.1, MPFR version 2.4.1.
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: 2e79a9448b1040fd1e442d3c6bfaea69
COLLECT_GCC_OPTIONS='-v' '-mtune=generic'
as -V -Qy -o /tmp/cc4clAj5.o /tmp/ccGTjhJB.s
GNU assembler version 2.20.51.0.2 (x86_64-redhat-linux) using BFD version version 2.20.51.0.2-5.44.el6 20100205
COMPILER_PATH=/usr/libexec/gcc/x86_64-redhat-linux/4.4.7/:/usr/libexec/gcc/x86_64-redhat-linux/4.4.7/:/usr/libexec/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/4.4.7/:/usr/lib/gcc/x86_64-redhat-linux/:/usr/libexec/gcc/x86_64-redhat-linux/4.4.7/:/usr/libexec/gcc/x86_64-redhat-linux/:/usr/lib/gcc/x86_64-redhat-linux/4.4.7/:/usr/lib/gcc/x86_64-redhat-linux/
LIBRARY_PATH=/usr/lib/gcc/x86_64-redhat-linux/4.4.7/:/usr/lib/gcc/x86_64-redhat-linux/4.4.7/:/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-mtune=generic'
/usr/libexec/gcc/x86_64-redhat-linux/4.4.7/collect2 --eh-frame-hdr --build-id -m elf_x86_64 --hash-style=gnu -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../crt1.o /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../crti.o /usr/lib/gcc/x86_64-redhat-linux/4.4.7/crtbegin.o -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../.. /tmp/cc4clAj5.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-redhat-linux/4.4.7/crtend.o /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../crtn.o
/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../crt1.o: could not read symbols: File in wrong format
collect2: ld returned 1 exit status
$ file /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../crt1.o
/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../crt1.o: ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.18, not stripped
This error message:
/usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../crt1.o: could not read symbols: File in wrong format collect2: ld returned 1 exit status
Indicates that the linker found a 32-bit object file where it expected a 64-bit one, or vise versa. You can determine which by examining the file in question with the file command. readelf is a little bit overkill for this purpose, but it gets the job done, too. We know as a result that yours is a 32-bit ELF object file, so presumably the compiler expected a 64-bit one.
The path you present canonicalizes to /usr/lib/crt1.o. On RHEL6, that belongs to the 32-bit glibc-devel package (the 64-bit analog lives in /usr/lib64). A 32-bit ELF object is exactly what that file should be, so if the compiler expected a 64-bit object then something is seriously wrong with it.
My best guess is that an installation from source or from a package for some other RPM-based distribution (maybe even a different version of RHEL) has produced an unusable environment for you. If you have an installed-from-source copy of GCC somewhere other than /usr/bin, then removing that (from the path, at least) might be sufficient to solve your problem. The command which gcc might be useful in determining whether you're getting a rogue gcc instead of RHEL's standard one.
Otherwise, your best bet is to remove all copies of gcc and your development libraries, and then reinstall from the official package repositories. The main packages to remove and reinstall are 'gcc' and 'glibc-devel'; removing these may cause other tools and development libraries to be removed as well. I recommend also removing any 'compat-gcc' package. Do not attempt to remove the main 'glibc' package, however, for if you somehow managed to do that, it would render your system inoperable.
I am trying to compile a large C file (specifically for MATLAB mexing). The C file is around 20 MB (available from the GCC bug tracker if you want to play around with it).
Here is the command I am running and the output to screen, below. This has been running for hours, and as you can see, optimization is already disabled (-O0). Why is this so slow? Is there a way I can make this faster?
(For reference: Ubuntu 12.04 (Precise Pangolin) 64 bit and GCC 4.7.3)
/usr/bin/gcc -c -DMX_COMPAT_32 -D_GNU_SOURCE -DMATLAB_MEX_FILE -I"/usr/local/MATLAB/R2015a/extern/include" -I"/usr/local/MATLAB/R2015a/simulink/include" -ansi -fexceptions -fPIC -fno-omit-frame-pointer -pthread -O0 -DNDEBUG path/to/test4.c -o /tmp/mex_198714460457975_3922/test4.o -v
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.7.3-2ubuntu1~12.04' --with-bugurl=file:///usr/share/doc/gcc-4.7/README.Bugs --enable-languages=c,c++,go,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.7 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.7 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --with-system-zlib --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.7.3 (Ubuntu/Linaro 4.7.3-2ubuntu1~12.04)
COLLECT_GCC_OPTIONS='-c' '-D' 'MX_COMPAT_32' '-D' '_GNU_SOURCE' '-D' 'MATLAB_MEX_FILE' '-I' '/usr/local/MATLAB/R2015a/extern/include' '-I' '/usr/local/MATLAB/R2015a/simulink/include' '-ansi' '-fexceptions' '-fPIC' '-fno-omit-frame-pointer' '-pthread' '-O0' '-D' 'NDEBUG' '-o' '/tmp/mex_198714460457975_3922/test4.o' '-v' '-mtune=generic' '-march=x86-64'
/usr/lib/gcc/x86_64-linux-gnu/4.7/cc1 -quiet -v -I /usr/local/MATLAB/R2015a/extern/include -I /usr/local/MATLAB/R2015a/simulink/include -imultilib . -imultiarch x86_64-linux-gnu -D_REENTRANT -D MX_COMPAT_32 -D _GNU_SOURCE -D MATLAB_MEX_FILE -D NDEBUG path/to/test4.c -quiet -dumpbase test4.c -mtune=generic -march=x86-64 -auxbase-strip /tmp/mex_198714460457975_3922/test4.o -O0 -ansi -version -fexceptions -fPIC -fno-omit-frame-pointer -fstack-protector -o /tmp/ccxDOA5f.s
GNU C (Ubuntu/Linaro 4.7.3-2ubuntu1~12.04) version 4.7.3 (x86_64-linux-gnu)
compiled by GNU C version 4.7.3, GMP version 5.0.2, MPFR version 3.1.0-p3, MPC version 0.9
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
ignoring nonexistent directory "/usr/local/include/x86_64-linux-gnu"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../x86_64-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/local/MATLAB/R2015a/extern/include
/usr/local/MATLAB/R2015a/simulink/include
/usr/lib/gcc/x86_64-linux-gnu/4.7/include
/usr/local/include
/usr/lib/gcc/x86_64-linux-gnu/4.7/include-fixed
/usr/include/x86_64-linux-gnu
/usr/include
End of search list.
GNU C (Ubuntu/Linaro 4.7.3-2ubuntu1~12.04) version 4.7.3 (x86_64-linux-gnu)
compiled by GNU C version 4.7.3, GMP version 5.0.2, MPFR version 3.1.0-p3, MPC version 0.9
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: c119948b394d79ea05b6b3986ab084cf
EDIT: a follow-on: I followed chqrlie's advice and tcc compiled my function in <5 seconds (I had to remove the -ansi flag only and turn "gcc" to "tcc"), which is pretty remarkable, really. I can only imagine the complexity of GCC.
When trying to then mex it, however, there is one other command mex typically needs. The second command is typically:
/usr/bin/gcc -pthread -Wl,--no-undefined -Wl,-rpath-link,/usr/local/MATLAB/R2015a/bin/glnxa64 -shared -O -Wl,--version-script,"/usr/local/MATLAB/R2015a/extern/lib/glnxa64/mexFunction.map" /tmp/mex_61853296369424_4031/test4.o -L"/usr/local/MATLAB/R2015a/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -o test4.mexa64
I cannot run this with tcc as some of these flags are not compatible. If I try to run this second compilation step with GCC, I get:
/usr/bin/ld: test4.o: relocation R_X86_64_PC32 against undefined symbol `mxGetPr' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
EDIT: The solution appears to be clang. tcc can compile the file, but the arguments in the second step in mexing are incompatible with tcc's argument options. Clang is very fast and produces a nice, small, optimized file.
Nearly the entire file is one expression, the assignment to double f[24] = .... That's going to generate a gigantic abstract syntax tree tree. I'd be surprised if anything but a specialized compiler could handle that efficiently.
The 20 megabyte file itself may be fine, but the one giant expression may be what is causing the issue. Try as a preliminary step, splitting the line into double f[24] = {0} and then 24 assignments of f[0] = ...; f[1] = ... and see what happens. Worst case, you can split the 24 assignments into 24 functions, each in its own .c file, and compile them separately. This won't reduce the size of the AST, it will just reorganize it, but GCC is probably more optimized at handling many statements which together add up to a lot of code, vs. one huge expression.
The ultimate approach would be to generate the code in a more optimized manner. If I search for s4*s5*s6, for example, I get 77,783 hits. These s[4-6] variables don't change. You should generate a temporary variable, double _tmp1 = s4*s5*s6; and then use that instead of repeating the expression. You've just eliminated 311,132 nodes from your abstract syntax tree (assuming s4*s5*s6 is 5 nodes and _tmp1 is one node). That's that much less processing GCC has to do. This should also generate faster code (you won't repeat the same multiplication 77,783 times).
If you do this in a smart way in a recursive manner (e.g. s4*s5*s6 --> _tmp1, (c4*c6+s4*s5*s6) --> (c4*c6+_tmp1) --> _tmp2, c5*s6*(c4*c6+s4*s5*s6) --> c5*s6*_tmp2 -> _tmp3, etc...), you can probably eliminate most of the size of the generated code.
Upon testing, I found that the Clang compiler seems to have less problems compiling large files. Although Clang consumed almost a gigabyte of memory during compilation, it successfully turned OP's source code form into a 70 kB object file. This works for all optimization levels I tested.
gcc was also able to compile this file quickly and without consuming too much memory if optimization is turned on. This bug in gcc comes from the large expression in OPs code which places a huge burden on the register allocator. With optimizations turned on, the compiler performs an optimization called common subexpression elimination which is able to remove a lot of redundancy from OPs code, reducing both compilation time and object file size to manageable values.
Here are some tests with the testcase from the aforementioned bug report:
$ time gcc5 -O3 -c -o testcase.gcc5-O3.o testcase.c
real 0m39,30s
user 0m37,85s
sys 0m1,42s
$ time gcc5 -O0 -c -o testcase.gcc5-O0.o testcase.c
real 23m33,34s
user 23m27,07s
sys 0m5,92s
$ time tcc -c -o testcase.tcc.o testcase.c
real 0m2,60s
user 0m2,42s
sys 0m0,17s
$ time clang -O3 -c -o testcase.clang-O3.o testcase.c
real 0m13,71s
user 0m12,55s
sys 0m1,16s
$ time clang -O0 -c -o testcase.clang-O0.o testcase.c
real 0m17,63s
user 0m16,14s
sys 0m1,49s
$ time clang -Os -c -o testcase.clang-Os.o testcase.c
real 0m14,88s
user 0m13,73s
sys 0m1,11s
$ time clang -Oz -c -o testcase.clang-Oz.o testcase.c
real 0m13,56s
user 0m12,45s
sys 0m1,09
This is the resulting object file size:
text data bss dec hex filename
39101286 0 0 39101286 254a366 testcase.clang-O0.o
72161 0 0 72161 119e1 testcase.clang-O3.o
72087 0 0 72087 11997 testcase.clang-Os.o
72087 0 0 72087 11997 testcase.clang-Oz.o
38683240 0 0 38683240 24e4268 testcase.gcc5-O0.o
87500 0 0 87500 155cc testcase.gcc5-O3.o
78239 0 0 78239 1319f testcase.gcc5-Os.o
69210504 3170616 0 72381120 45072c0 testcase.tcc.o
Try Fabrice Bellard's tiny C compiler tcc from http://tinycc.org:
chqrlie$ time tcc -c test4.c
real 0m1.336s
user 0m1.248s
sys 0m0.084s
chqrlie$ size test4.o
text data bss dec hex filename
38953877 3170632 0 42124509 282c4dd test4.o
Yes, that's 1.336 seconds on a pretty basic PC!
Of course I cannot test the resulting executable, but the object file should be linkable with the rest of your program and libraries.
For this test, I used a dummy version of file mex.h:
typedef struct mxArray mxArray;
double *mxGetPr(const mxArray*);
enum { mxREAL = 0 };
mxArray *mxCreateDoubleMatrix(int nx, int ny, int type);
gcc still has not finished compiling...
EDIT: gcc managed to hog my Linux box so badly I cannot connect to it anymore:(
So what I am trying to do is on Ubuntu 14.04 (x86_64) I want to set up musl-libc based on the latest released 1.1.11 version which is available at this moment.
What I did was to:
Install multilib support for GCC: sudo apt-get --no-install-recommends install gcc-multilib
Configure the libraries for 32-bit and 64-bit respectively and install them into separate folders:
CFLAGS=-m32 ./configure --prefix=$HOME/bin/musl-32-bit --disable-shared --target=i386-linux-gnu && make && make install
CFLAGS=-m64 ./configure --prefix=$HOME/bin/musl-64-bit --disable-shared --target=x86_64-linux-gnu
Then in order to build a statically linked premake4, I invoke GNU make like this on the Makefile generated by premake4:
make -j 8 CC=$HOME/bin/musl-32-bit/bin/musl-gcc ARCH=-m32 LDFLAGS="-v -static" verbose=1
This appears to work up to the linking step, which bombs with:
Linking Premake4
$HOME/bin/musl-32-bit/bin/musl-gcc -o bin/release/premake4 intermediate/gmake__/premake.o intermediate/gmake__/os_uuid.o intermediate/gmake__/os_pathsearch.o intermediate/gmake__/os_match.o intermediate/gmake__/os_chdir.o intermediate/gmake__/os_mkdir.o intermediate/gmake__/os_stat.o intermediate/gmake__/os_getversion.o intermediate/gmake__/premake_main.o intermediate/gmake__/os_isdir.o intermediate/gmake__/string_endswith.o intermediate/gmake__/os_isfile.o intermediate/gmake__/scripts.o intermediate/gmake__/path_isabsolute.o intermediate/gmake__/os_rmdir.o intermediate/gmake__/os_getcwd.o intermediate/gmake__/os_is64bit.o intermediate/gmake__/os_copyfile.o intermediate/gmake__/lstate.o intermediate/gmake__/ltable.o intermediate/gmake__/lgc.o intermediate/gmake__/lobject.o intermediate/gmake__/lcode.o intermediate/gmake__/lmathlib.o intermediate/gmake__/lbaselib.o intermediate/gmake__/lmem.o intermediate/gmake__/lfunc.o intermediate/gmake__/lparser.o intermediate/gmake__/ldblib.o intermediate/gmake__/lzio.o intermediate/gmake__/lstrlib.o intermediate/gmake__/lvm.o intermediate/gmake__/lauxlib.o intermediate/gmake__/llex.o intermediate/gmake__/lstring.o intermediate/gmake__/ldump.o intermediate/gmake__/ldebug.o intermediate/gmake__/loadlib.o intermediate/gmake__/lopcodes.o intermediate/gmake__/linit.o intermediate/gmake__/ldo.o intermediate/gmake__/lapi.o intermediate/gmake__/liolib.o intermediate/gmake__/loslib.o intermediate/gmake__/lundump.o intermediate/gmake__/ltm.o intermediate/gmake__/ltablib.o -v -static -L. -s -rdynamic -lm -ldl
Using built-in specs.
Reading specs from $HOME/bin/musl-32-bit/lib/musl-gcc.specs
rename spec cpp_options to old_cpp_options
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.8/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.8.4-2ubuntu1~14.04' --with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.8 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.8 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libmudflap --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04)
COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib32/:/lib/../lib32/:/usr/lib/../lib32/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-m32' '-o' 'bin/release/premake4' '-v' '-static' '-L.' '-s' '-rdynamic' '-specs=$HOME/bin/musl-32-bit/lib/musl-gcc.specs' '-mtune=generic' '-march=i686'
/usr/lib/gcc/x86_64-linux-gnu/4.8/collect2 -dynamic-linker /lib/ld-musl-i386.so.1 -nostdlib -static -export-dynamic -z relro -o bin/release/premake4 -s $HOME/bin/musl-32-bit/lib/crt1.o $HOME/bin/musl-32-bit/lib/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbegin.o -L. -L$HOME/bin/musl-32-bit/lib -L /usr/lib/gcc/x86_64-linux-gnu/4.8/. intermediate/gmake__/premake.o intermediate/gmake__/os_uuid.o intermediate/gmake__/os_pathsearch.o intermediate/gmake__/os_match.o intermediate/gmake__/os_chdir.o intermediate/gmake__/os_mkdir.o intermediate/gmake__/os_stat.o intermediate/gmake__/os_getversion.o intermediate/gmake__/premake_main.o intermediate/gmake__/os_isdir.o intermediate/gmake__/string_endswith.o intermediate/gmake__/os_isfile.o intermediate/gmake__/scripts.o intermediate/gmake__/path_isabsolute.o intermediate/gmake__/os_rmdir.o intermediate/gmake__/os_getcwd.o intermediate/gmake__/os_is64bit.o intermediate/gmake__/os_copyfile.o intermediate/gmake__/lstate.o intermediate/gmake__/ltable.o intermediate/gmake__/lgc.o intermediate/gmake__/lobject.o intermediate/gmake__/lcode.o intermediate/gmake__/lmathlib.o intermediate/gmake__/lbaselib.o intermediate/gmake__/lmem.o intermediate/gmake__/lfunc.o intermediate/gmake__/lparser.o intermediate/gmake__/ldblib.o intermediate/gmake__/lzio.o intermediate/gmake__/lstrlib.o intermediate/gmake__/lvm.o intermediate/gmake__/lauxlib.o intermediate/gmake__/llex.o intermediate/gmake__/lstring.o intermediate/gmake__/ldump.o intermediate/gmake__/ldebug.o intermediate/gmake__/loadlib.o intermediate/gmake__/lopcodes.o intermediate/gmake__/linit.o intermediate/gmake__/ldo.o intermediate/gmake__/lapi.o intermediate/gmake__/liolib.o intermediate/gmake__/loslib.o intermediate/gmake__/lundump.o intermediate/gmake__/ltm.o intermediate/gmake__/ltablib.o -lm -ldl --start-group /usr/lib/gcc/x86_64-linux-gnu/4.8/libgcc.a /usr/lib/gcc/x86_64-linux-gnu/4.8/libgcc_eh.a -lc --end-group /usr/lib/gcc/x86_64-linux-gnu/4.8/crtend.o $HOME/bin/musl-32-bit/lib/crtn.o
/usr/bin/ld: skipping incompatible $HOME/bin/musl-32-bit/lib/libc.a when searching for -lc
/usr/bin/ld: cannot find -lc
collect2: error: ld returned 1 exit status
make[1]: *** [bin/release/premake4] Error 1
make: *** [Premake4] Error 2
The relevant line is:
/usr/bin/ld: skipping incompatible $HOME/bin/musl-32-bit/lib/libc.a when searching for -lc
Now the part I don't understand about this is, that when I ar x the libc.a (into a folder $HOME/bin/musl-32-bit/lib/libc) generated during the build step of musl-libc (see above), it proves that all of the objects included seem to be of the correct target architecture (all show ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped) as I can prove from coming up empty when issuing the following command:
find $HOME/bin/musl-32-bit/lib -name '*.o' -exec file {} +|grep -v 'ELF 32-bit LSB relocatable, Intel 80386, version 1 (SYSV), not stripped'
And in fact this gives no output. Similarly when looking inside the build directory using the same method, I cannot find any object file that doesn't match my expectation.
For good measure I decided to also task objdump to tell me more about the libc.a in question and came up with the same result:
objdump -a $HOME/bin/musl-32-bit/lib/libc.a|grep 'file format'|grep -v 'file format elf32-i386'
So my question is twofold:
what disqualifies a static library as "incompatible" when GCC is asked to link it?
what could be the particular issue I am seeing?
The first is what I am really interested in, but with the second I am asking to share your experience with trouble-shooting like this. Which verification steps have I missed, for example?
Please note that the "native" premake4 builds just fine with:
make -j 8 CC=$HOME/bin/musl-64-bit/bin/musl-gcc ARCH=-m64 LDFLAGS=-static verbose=1
From the output when adding the -v flag to LDFLAGS it appears as if the target always stays at x86_64-linux-gnu. I have yet to come up with a method to fix this, though.
In short, the musl-gcc wrapper script setup is not well-suited to use with -m32. I think what's happening is that the actual compiler is getting invoked in the default (64-bit) mode by musl-gcc, then the resulting object files are not compatible with the (intended, 32-bit) libc.
It may work if you put -m32 in the generated wrapper script. This will happen automatically with recent versions if you put the -m32 in $CC (i.e. CC="gcc -m32") rather than putting it in $CFLAGS.
Update: As noted in the discussion that was moved to chat, adding -Wl,-melf_i386 is probably also needed (due to flaws in the spec file used by the musl-gcc wrapper that don't account for -m32 support) but still does not seem to be sufficient.
The solution
It turns out the solution is rather simple.
We need to tell both the linker and the driver to use -m32 ... I was that far before. However, it turns out that the missing piece was to pass the linker option to the driver via CFLAGS like this -Wl,-melf_i386.
I am finally able to build and link the 32-bit executable on a multilib-enabled 64-bit host.
NB: Below information is left in place for those who want to learn how I investigated the issue.
Alright, so I investigated the issue a little further and the output gets more enlightening once you extract the object files. To reproduce what I am doing you may have to use Bash or a similar shell that allows for $(...) or you need to adjust the command lines accordingly.
First off it's important to have gcc-multilib and friends installed in order to target -m32 (i386-linux-gnu which happens to be an alias for i686-linux-gnu here).
The code and the make file
I had the following make file:
CC?=$(HOME)/bin/musl/bin64/musl-gcc
BLDARCH?=-m64
CFLAGS+=-v $(BLDARCH)
LDFLAGS+=-v -static $(BLDARCH)
all: helloworld
helloworld: helloworld.c
clean:
rm -f helloworld
rebuild: clean all
.PHONY: clean rebuild
.NOTPARALLEL: rebuild
and the following small helloworld.c:
#include <stdio.h>
int main(int argc, char** argv)
{
printf("Hello world!\n");
return 0;
}
and my 32-bit musl-libc was installed to $HOME/bin/musl/{bin,include,lib}32 respectively. The 64-bit one was installed to $HOME/bin/musl/{bin,include,lib}64 respectively.
Attempting to build with:
make CC=$HOME/bin/musl/bin32/musl-gcc BLDARCH=-m32 rebuild
always failed with the same meaningless lines:
/usr/bin/ld: skipping incompatible ~/bin/musl/lib32/libc.a when searching for -lc
/usr/bin/ld: cannot find -lc
collect2: error: ld returned 1 exit status
make: *** [helloworld] Error 1
Digging into the details
So after some contemplation I decided to re-do the steps done by the gcc driver manually.
This meant to run roughly (I replaced all occurrences of my home folder with a ~):
Compile: /usr/lib/gcc/x86_64-linux-gnu/4.8/cc1 -quiet -imultilib 32 -imultiarch i386-linux-gnu helloworld.c -nostdinc -isystem ~/bin/musl/include32 -isystem /usr/lib/gcc/x86_64-linux-gnu/4.8/include -quiet -dumpbase helloworld.c -m32 -mtune=generic -march=i686 -auxbase helloworld -version -fstack-protector -Wformat -Wformat-security -o /tmp/ccGmMuR1.s
Assemble: as -v -v --32 -o /tmp/ccgRGlqf.o /tmp/ccGmMuR1.s
Link: env COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/ LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.8/32/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib32/:/lib/../lib32/:/usr/lib/../lib32/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../:/lib/:/usr/lib/ COLLECT_GCC_OPTIONS="-v -v -static -m32 -o helloworld -specs=~/bin/musl/lib32/musl-gcc.specs -mtune=generic -march=i686" /usr/lib/gcc/x86_64-linux-gnu/4.8/collect2 -dynamic-linker /lib/ld-musl-i386.so.1 -nostdlib -static -z relro -o helloworld ~/bin/musl/lib32/crt1.o ~/bin/musl/lib32/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/32/crtbegin.o -L~/bin/musl/lib32 -L /usr/lib/gcc/x86_64-linux-gnu/4.8/32/. /tmp/ccpL09mJ.o --start-group /usr/lib/gcc/x86_64-linux-gnu/4.8/32/libgcc.a /usr/lib/gcc/x86_64-linux-gnu/4.8/32/libgcc_eh.a -lc --end-group /usr/lib/gcc/x86_64-linux-gnu/4.8/32/crtend.o ~/bin/musl/lib32/crtn.o
Obviously this didn't change the error message a bit just yet:
/usr/bin/ld: skipping incompatible ~/bin/musl/lib32/libc.a when searching for -lc
/usr/bin/ld: cannot find -lc
collect2: error: ld returned 1 exit status
So I took out the -lc from the command line for collect2 and created a folder ~/bin/musl/lib32/archive into which I extracted the whole libc.a generated by my musl-libc build attempt. I then instructed collect2 where to find the object files like this:
Link: env COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/ LIBRARY_PATH=/usr/lib/gc
c/x86_64-linux-gnu/4.8/32/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib32/:/lib/../lib32/:/usr/lib/../lib32/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../:/lib/:/usr/lib/ COLLECT_GCC_OPTIONS="-v -v -
static -m32 -o helloworld -specs=~/bin/musl/lib32/musl-gcc.specs -mtune=generic -march=i686" /usr/lib/gcc/x86_64-linux-gnu/4.8/collect2 -dynamic-linker /lib/ld-musl-i386.so.1 -nostdlib -static -z relro -o helloworld $HOME/bin/musl/lib32/crt1.o ~/bin/musl/lib32/crti.o /usr/lib/gcc/x86_64-linux-gnu/4.8/32/crtbegin.o -L~/bin/musl/lib32 -L /usr/lib/gcc/x86_64-linux-gnu/4.8/32/. /tmp/ccpL09mJ.o --start-group /usr/lib/gcc/x86_64-linux-g
nu/4.8/32/libgcc.a /usr/lib/gcc/x86_64-linux-gnu/4.8/32/libgcc_eh.a --end-group /usr/lib/gcc/x86_64-linux-gnu/4.8/32/crtend.o ~/bin/musl/lib32/crtn.o $(find ~/bin/musl/lib32/archive -type f -name '*.o')
This gist is taking out -lc and appending $(find ~/bin/musl/lib32/archive -type f -name '*.o').
Which gave me a whole bunch of new, but more meaningful errors similar to the following ones:
/usr/bin/ld: Warning: size of symbol `__init_ssp' changed from 1 in ~/bin/musl/lib32/archive/__libc_start_main.o to 65 in ~/bin/musl/lib32/archive/__stack_chk_fail.o
/usr/bin/ld: Warning: size of symbol `__funcs_on_exit' changed from 126 in ~/bin/musl/lib32/archive/atexit.o to 1 in ~/bin/musl/lib32/archive/exit.o
# more of those
/usr/bin/ld: i386 architecture of input file `~/bin/musl/lib32/crt1.o' is incompatible with i386:x86-64 output
/usr/bin/ld: i386 architecture of input file `~/bin/musl/lib32/crti.o' is incompatible with i386:x86-64 output
# more of those
The plot thickens. Apparently the collect2 command gets the wrong idea about what to build. Which is odd considering the output for the governing environment variables:
COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.8/32/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib32/:/lib/../lib32/:/usr/lib/../lib32/:/usr/lib/gcc/x86_64-linux-gnu/4.8/:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-v' '-v' '-static' '-m32' '-o' 'helloworld' '-specs=~/bin/musl/lib32/musl-gcc.specs' '-mtune=generic' '-march=i686'
... which I passed using env in my attempt to reproduce the conditions encountered by the collect2 wrapper when linking the libc.a.
In order to find out more about the internals of the GNU compilers one needs to read https://gcc.gnu.org/onlinedocs/gccint/
Unfortunately the part about collect2 doesn't help us here. But the lengthy output of /usr/lib/gcc/x86_64-linux-gnu/4.8/collect2 --help looks promising.
Just how could we smuggle our command line option to collect2?
For now I was going to settle for whatever I could run manually. So I attempted to tell the linker which output format I expected. Based on the list of supported targets I was interested in elf_i386.
Passing -melf_386 at the end of the previous line gave an interesting error output. Numerous referenced functions such as __vsyscall, __moddi3 and __divdi3 were undefined. That indicated they simply didn't exist in the object files from the static lib or in any of the startup .o files for that matter:
~/bin/musl/lib32/archive/aio.o: In function `cleanup':
aio.c:(.text+0x5ad): undefined reference to `__vsyscall'
aio.c:(.text+0x5bb): undefined reference to `__vsyscall'
aio.c:(.text+0x5e8): undefined reference to `__vsyscall'
aio.c:(.text+0x5f6): undefined reference to `__vsyscall'
aio.c:(.text+0x61d): undefined reference to `__vsyscall'
~/bin/musl/lib32/archive/aio.o:aio.c:(.text+0x62b): more undefined references to `__vsyscall' follow
~/bin/musl/lib32/archive/cpow.o: In function `cpow':
cpow.c:(.text+0x4f): undefined reference to `__mulxc3'
~/bin/musl/lib32/archive/cpowf.o: In function `cpowf':
cpowf.c:(.text+0x47): undefined reference to `__mulxc3'
~/bin/musl/lib32/archive/cpowl.o: In function `cpowl':
cpowl.c:(.text+0x4c): undefined reference to `__mulxc3'
~/bin/musl/lib32/archive/sysconf.o: In function `sysconf':
sysconf.c:(.text+0xcc): undefined reference to `__vsyscall'
~/bin/musl/lib32/archive/__getdents.o: In function `__getdents':
__getdents.c:(.text+0x13): undefined reference to `__vsyscall'
~/bin/musl/lib32/archive/opendir.o: In function `opendir':
opendir.c:(.text+0x37): undefined reference to `__vsyscall'
~/bin/musl/lib32/archive/readdir.o: In function `readdir':
readdir.c:(.text+0x1f): undefined reference to `__vsyscall'
~/bin/musl/lib32/archive/__init_tls.o: In function `__init_tls':
__init_tls.c:(.text+0x136): undefined reference to `__vsyscall6'
__init_tls.c:(.text+0x16e): undefined reference to `__vsyscall'
As I had already clarified in my question, the object files from the archive all stated that they were elf32-i386.
The functions __vsyscall and __vsyscall6 should end up in a file called syscall.o given the source file for i386 in musl-libc: src/internal/i386/syscall.s. Let's verify that first. Since there is also a file src/misc/syscall.c the name might be different though. Four files have syscall in the file name:
__syscall_cp.o
syscall_cp.o
syscall.o
syscall_ret.o
Querying those files using nm gave:
$ nm -s $(ls |grep syscall)
__syscall_cp.o:
00000000 t sccp
U __syscall
00000005 T __syscall_cp
00000000 W __syscall_cp_c
syscall_cp.o:
U __cancel
00000008 T __cp_begin
00000035 T __cp_cancel
00000030 T __cp_end
00000000 T __syscall_cp_asm
syscall.o:
00000000 T syscall
U __syscall_ret
U __vsyscall6
syscall_ret.o:
U __errno_location
00000000 T __syscall_ret
Symbols with the symbol type U are undefined and therefore expected by the linker to come from outside (external to each object file).
A final $ nm --defined-only *.o ../*.o|grep vsyscall was what was needed to verify that those symbols were indeed missing from the libc.a.
So the cross-built libc.a what was faulty after all. Back to the drawing board.
I hope this description helps others to figure out similar issues and look behind the scenes in GCC.
The saga continues
I was really surprised to see:
$ nm --defined-only ../libc.a |grep -B 2 vsyscall
syscall.o:
0000004b T __syscall
00000000 T __vsyscall
00000031 T __vsyscall6
but for the extracted object files the corresponding command (nm --defined-only *.o ../*.o|grep -B 2 vsyscall) would yield no output.
So inside the libc.a somehow nm sees the two symbols, but after extracting them they disappear? Odd.
Let's look for syscall.o in the libc.a:
$ nm ../libc.a |grep ^syscall
syscall.o:
syscall_ret.o:
syscall.o:
syscall_cp.o:
Whoa? So syscall.o exists twice inside the static library? Well that looks like it may just be the error cause we're looking for. And it certainly does explain why the symbols disappear. Likely the latter syscall.o overwrites the one first extracted when running ar x ....
Confirming:
$ nm ../libc.a |grep -A 4 ^syscall\.o
syscall.o:
0000004b T __syscall
U __sysinfo
00000000 T __vsyscall
00000031 T __vsyscall6
--
syscall.o:
00000000 T syscall
U __syscall_ret
U __vsyscall6
and looking into the musl-libc source tree after doing the 32-bit build:
$ find . -type f -name 'syscall.o' -exec nm {} +
./src/internal/syscall.o:
0000004b T __syscall
U __sysinfo
00000000 T __vsyscall
00000031 T __vsyscall6
./src/misc/syscall.o:
00000000 T syscall
U __syscall_ret
U __vsyscall6
Copying the former into the lib32/archive directory under a name that doesn't collide with existing names gives more errors on other functions, suggesting other object files may also exist as duplicates inside the generated libc.a.
Which ones are duplicates?
$ diff <(nm libc.a|grep ':$'|cut -f 1 -d :|sort) <(nm libc.a|grep ':$'|cut -f 1 -d :|sort -u)
--- /dev/fd/63 2015-10-05 23:58:53.683804823 +0000
+++ /dev/fd/62 2015-10-05 23:58:53.683804823 +0000
## -131,7 +131,6 ##
clogl.o
clog.o
clone.o
-clone.o
closedir.o
close.o
cnd_broadcast.o
## -1115,7 +1114,6 ##
__syscall_cp.o
syscall_cp.o
syscall.o
-syscall.o
syscall_ret.o
sysconf.o
sysinfo.o
This way we see clone.o and syscall.o are affected as duplicates. Which indicates that some object files are missing altogether from the libc.a given undefined references to the following symbols:
__divdi3
__moddi3
__mulxc3
__tls_get_new
__udivdi3
__umoddi3
These names happen to coincide with the ones from the Integer library routines listed for GCC. Which makes me think I am missing one library provided by GCC which to link. Like libgcc?! ...
I have package lib32gcc-4.8-dev which means I should have the required file:
$ apt-file list lib32gcc-4.8-dev|grep -E 'libgcc.*\.a'
lib32gcc-4.8-dev: /usr/lib/gcc/x86_64-linux-gnu/4.8/32/libgcc.a
lib32gcc-4.8-dev: /usr/lib/gcc/x86_64-linux-gnu/4.8/32/libgcc_eh.a
After the previous step I decided to set up a x86_32 version on Ubuntu 14.04 that was essentially at the same patch level.
I then compared the libgcc.a from the x86_32 machine with that from the x86_64 machine. They turned out to be almost identical except for the "symbol value" (in nm lingo) of a handful of functions.
Also, since the symbols were in the static library I attempted to link against the static library again. This worked only with -melf_i386 on the linker (collect2) command line.
After trying to use LDFLAGS and noticing that those were also passed to cc1, I set appended -Wl,-melf_i386 to the CFLAGS and now it worked. Brilliant.
As a side-note: I also swapped the /usr/lib/gcc/x86_64-linux-gnu/4.8/collect2 in the command line for ld of which collect2 is supposed to be a wrapper. The error output was identical.