How to debug an experimental toolchain producing malformed executables - c

Question

I am working on cross compiling an experimental GNU free Linux toolchain using clang (instead of gcc), compiler-rt (instead of libgcc), libunwind (available at http://llvm.org/git/libunwind.git) (instead of libgcc_s), lld (instead of GNU ld), libcxx (instead of libstdc++), libcxxabi (instead of not sure, I'm unclear on the GNU distinction between libstdc++ and its ABI) and musl (instead of glibc).
Using a musl based gcc cross compiler and a few patches I've managed to successfully compile all of the above and sucessfully compile and link a simple hello world C program with it. Something seems to have gone wrong, however, as running the hello world program results in a segmentation fault:
$ ./hello
Segmentation fault
$
Normally I would simply debug it with gdb, but herein lies the problem:
$ gdb ./hello
Reading symbols from ./hello...Dwarf Error: Could not find abbrev number 5 in CU at offset 0x52 [in module /home/main/code/main/asm/hello]
(no debugging symbols found)...done.
(gdb) start
Temporary breakpoint 1 at 0x206
Starting program: /hello
During startup program terminated with signal SIGSEGV, Segmentation fault.
(gdb)
I can't seem to step through the program in any way, I'm guessing because the error is occuring somewhere in early C runtime startup. I can't even step through the assembly using layout asm and stepi, so I really don't know how to find out where exactly the error is occuring (to debug my toolchain).
I have confirmed that the problem resides with lld by using a GNU binutils ld to successfully link the hello world object (statically) using the cross compiled libraries and object files, which results in a functional hello world program. Since lld successfully links, however, I can't pinpoint where failure is occuring.
Note I compiled hello as a static executable and used the -v gcc/clang option to verify that all the correct libraries and object files were linked it.
Note online GDB documentation has the following to say about the above error:
On Unix systems, by default, if a shell is available on your target, gdb) uses it to start your program. Arguments of the run command are passed to the shell, which does variable substitution, expands wildcard characters and performs redirection of I/O. In some circumstances, it may be useful to disable such use of a shell, for example, when debugging the shell itself or diagnosing startup failures such as:
(gdb) run
Starting program: ./a.out
During startup program terminated with signal SIGSEGV, Segmentation fault.
which indicates the shell or the wrapper specified with ‘exec-wrapper’ crashed, not your program.
I don't think this is true, considering what I'm working with and that the problem doesn't happen when I use GNU ld, and because the suggested solution (set startup-with-shell off) doesn't work.

Answer 1

The croscompilling means that the compilation is done on a host machine, and the output of the compilation is the binary which shall run on a target machine. Therefore the compiled binary is not compatible with your host CPU. Instead, if your target supports this, you could run the binary there and use the debugger from your toolchain to connect to the running binary remotely if supported. Or alternatively, the debugger may also be available at the target and you can debug the binary already at place.
Just to get more feeling, try to use command file for the compiled binary, and some other binaries of your host to see possible differences.