no symbols (just question marks) in stack trace in remote gdb - c

Question

I have a relatively simple application (that links in another simple library) that refuses to let me remotely debug it with gdb. I've checked that gdb and gdbserver versions match. It's actually even the same OS (ubuntu) on both machines. It seems to be happily loading symbols from the executable. So I'm at a bit of a loss for what could be wrong. Any suggestions appreciated. Here is the transcript from gdb:
dev:/fast/git/archive/foo$ gdb /fast/git/foo
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /fast/git/foo...done.
(gdb) target remote test1:5000
Remote debugging using test1:5000
Reading /lib64/ld-linux-x86-64.so.2 from remote target...
warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.
Reading /lib64/ld-linux-x86-64.so.2 from remote target...
Reading symbols from target:/lib64/ld-linux-x86-64.so.2...Reading /lib64/ld-2.23.so from remote target...
Reading /lib64/.debug/ld-2.23.so from remote target...
(no debugging symbols found)...done.
0x00007ffff7dd7cc0 in ?? () from target:/lib64/ld-linux-x86-64.so.2
(gdb) bt
#0 0x00007ffff7dd7cc0 in ?? () from target:/lib64/ld-linux-x86-64.so.2
#1 0x0000000000000003 in ?? ()
#2 0x00007fffffffce02 in ?? ()
#3 0x00007fffffffce2f in ?? ()
#4 0x00007fffffffce32 in ?? ()
#5 0x0000000000000000 in ?? ()
Ah, so interesting. I'm still not sure why, but it only prints that (bogus stack trace) on connect. If I then "continue," it will happily print the right symbols if I trigger a break.
Odd...maybe this is partly user error, but I expected it to start running and break on the start on main when I connected.

Answer 1

I'm at a bit of a loss for what could be wrong
It could be that there is nothing wrong at all.
Here is what I get on a local machine:
gdbserver :0 /bin/date
Process /bin/date created; pid = 132826
Listening on port 57966
Remote debugging from host 127.0.0.1
In a separate window:
gdb -q /bin/date
(gdb) target remote localhost:57966
Remote debugging using localhost:57966
Loading symbols for shared libraries.
Loading symbols for shared libraries.
0x00007ffff7ddb2d0 in _start () at rtld.c:871
871 rtld.c: No such file or directory.
(gdb) bt
#0 0x00007ffff7ddb2d0 in _start () at rtld.c:871
#1 0x0000000000000001 in ?? ()
#2 0x00007fffffffe157 in ?? ()
#3 0x0000000000000000 in ?? ()
What's happening here is that gdbserver stopped the inferior (being debugged) process very early on. Before the loader zeroed out stack, and before entering main.
The symbols for main executable should already be loaded, and you should be able to set breakpoints on them. Set a breakpoint on main, and just continue from there, and you should soon see your breakpoint getting hit.
Update:
I expected it to start running and break on the start on main when I connected.
Your expectation is incorrect.
In a typical dynamically-linked binary, there are 1000s of instructions between _start and getting to main. Sometimes these instructions are the ones where a crash happens. If GDB automatically continued to main without giving you a chance to set breakpoints or watchpoints, then debugging such crashes would be much more difficult than it is now.

Answer 2

A few ideas:
Make sure the remote program and shared libraries are compiled with -g to add debug symbols. The (no debugging symbols found) message may mean that the debug symbols are missing from the whole executable.
Make sure the local and remote program are the same image. You can do a 'sum' on each. This may be the problem as the #1 0x0000000000000003 in ?? () line looks to be corrupt.
Are you able to manually debug the target on the remote box? If so, check for symbols to see if it something about the remote/local debug session. You can check for debug symbols by doing a list <function>
The address ranges are often mapped the same on local system versus remote. You can disassemble an address on a local debug session and see what function/method it maps to. This can be helpful if the remote program is crashing, at least you can get an idea of where the trouble is.

Answer 3

When debugging remote, gdb client does not know where to load symbols from. You have two options:
1. specify executable when starting gdb
gdb <executable>
(gdb) target remote <IP>:<port>
(gdb) load <executable>
gdb should know symbols now
(gdb) b main
(gdb) mon reset
(gdb) contnue
it should break at main
(gdb) bt
2. use file command to tell about the symbols.
gdb
(gdb) target remote <IP>:<port>
(gdb) load <executable>
(gdb) file <executable>
gdb should know symbols now
(gdb) b main
(gdb) mon reset
(gdb) contnue
it should break at main
(gdb) bt