My program is crashing and this is the segfault message. python3[26871]: segfault at 4cac ip 00007fe49938248a sp 00007fe498a64820 error 4 in libswsscommon.so.0.0.0[7fe499359000+7a000]
address should be (IP - loaded address of shared lib). (7fe49938248a - 7fe499359000) = 0x2948A.
When I try to get the info, gdb says 'no line number info'
Reading symbols from /usr/lib/x86_64-linux-gnu/libswsscommon.so.0.0.0...Reading symbols from /usr/lib/debug/.build-id/94/d57d3ce6dd6901ddf7f7d8985a8334c7622fc6.debug...done.
done.
(gdb) info line *0x2948A
No line number information available for address 0x2948a
How is it possible? Does it have anything to do with the start address which is greater than 0x2948A, as shown by objdump -f
objdump -f /usr/lib/x86_64-linux-gnu/libswsscommon.so.0.0.0
/usr/lib/x86_64-linux-gnu/libswsscommon.so.0.0.0: file format elf64-x86-64
architecture: i386:x86-64, flags 0x00000150:
HAS_SYMS, DYNAMIC, D_PAGED
start address 0x0000000000034d60
Related
Question
I have a test program called scan (see code section) that I run in one terminal. The program starts running and asks the user to enter a letter:
$ ./scan
Enter a letter
In another terminal window, I run gdb and attach to the scan process:
$ sudo gdb -p `pidof scan`
(gdb)
I would like to send contents of a binary file called payload as input to the scan process. How can I accomplish this in gdb while the process is attached? Is this possible?
Background
I want to attach to the scan process because I want to examine the program's ASLR address space while running. If I run the process directly with gdb ($ gdb scan), gdb turns off ASLR and I get consistent addressing.
For example, running scan directly with gdb (please note that ... is truncated output):
$ gdb scan
(gdb) info functions
...
0x00000000000005a8 _init
0x00000000000005d0 __stack_chk_fail#plt
0x00000000000005e0 printf#plt
0x00000000000005f0 __isoc99_scanf#plt
...
The addresses shown above are the same every time I run gdb this way, which indicates that gdb turns off ASLR. However, when attaching to a process, the addresses always change. On one run:
$ sudo gdb -p `pidof scan`
(gdb) info functions
...
0x00005598706305a8 _init
0x00005598706305d0 __stack_chk_fail#plt
0x00005598706305e0 printf#plt
0x00005598706305f0 __isoc99_scanf#plt
On another run:
0x000055813ccf65a8 _init
0x000055813ccf65d0 __stack_chk_fail#plt
0x000055813ccf65e0 printf#plt
0x000055813ccf65f0 __isoc99_scanf#plt
The file payload contains a binary payload. I know how to send file contents as input when running gdb normally (e.g. (gdb) run < payload), but not when running with an attached process.
I do not want to copy/paste these file contents into the terminal that is runningscan. I also do not want to turn off ASLR.
Things I have tried/read
I have read the gdb manual and gdb help commands:
(gdb) help
(gdb) help target
(gdb) help attach
(gdb) help obscure
Other StackOverflow questions do not ask about sending input to an attached process:
Pass File Input and Stdin to gdb
How to debug a program that takes user input from stdin with GDB?
How to passing input data in GDB mode for programming C. Already passed parameters and run program
Code
My entire scan program is this:
#include <stdio.h>
int main(int argc, char **argv)
{
char letter[1];
char buffer[8];
printf("Enter a letter: ");
int result = scanf("%s", letter);
printf("You entered: %s\n", letter);
}
I'm facing a segmentation fault. Syslog reports the following:
segfault at 0 ip 00000000f71ff256 sp 00000000f44fee50 error 4 in libprotobuf-c.so.0.0.0[f71f8000+f000]
So, libprotobuf-c.so is loaded at 0xf71f8000. When I loaded the respective core file in gdb and tried info sharedlibrary, it shows FROM address as 0xf71f9f70 which is different from what syslog showed. I'm not able to understand this mismatch. Could someone please help?
0xf71f9f70 0xf7204028 Yes (*) /usr/lib/libprotobuf-c.so.0
So, libprotobuf-c.so is loaded at 0xf71f8000. When I loaded the respective core file in gdb and tried info sharedlibrary, it shows FROM address as 0xf71f9f70 which is different from what syslog showed.
Actually they are the same. GDB shows start of .text as the From address.
If you do readelf -WS /usr/lib/libprotobuf-c.so.0 | grep '\.text', you'll discover that .text starts at 0xf71f9f70 - 0xf71f8000 == 0x1f70.
Are there any tools or libraries one can use on Linux to get the original (source) instruction only from the PID and the current instruction pointer address, even if the IP currently points into a shared library?
AFAIK it should be possible, since the location of the library mapping is available through /proc/[PID]/maps, though I haven't found any applications or examples doing so.
Any suggestions?
EDIT: an assembly instruction or the nearest symbol suffice (source code line is not necessarily needed)
I found a way to do this with GDB:
Interactive:
$ gdb --pid 1566
(gdb) info symbol 0x7fe28b8a2b79
pselect + 89 in section .text of /lib/x86_64-linux-gnu/libc.so.6
(gdb) info symbol 0x5612550f14a4
copy_word_list + 20 in section .text of /usr/bin/bash
(gdb) info symbol 0x7fe28b878947
execve + 7 in section .text of /lib/x86_64-linux-gnu/libc.so.6
Shows exactly what I wanted!
It can also be scripted:
gdb -q --pid PID --batch -ex 'info symbol HEX_SYMBOL_ADDR'
I have been running my application successfully in CentOs6.6. Recently, the hardware(motherboard and RAM) was updated and my application is getting killed now without any reason at all.
[root#localhost PktBlaster]# ./PktBlaster
Killed
File and ldd output
[root#localhost PktBlaster]# file PktBlaster
PktBlaster: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped
[root#localhost PktBlaster]# ldd PktBlaster
not a dynamic executable
Output of strace
[root#localhost PktBlaster]# strace ./PktBlaster
execve("./PktBlaster", ["./PktBlaster"], [/* 30 vars */] <unfinished ...>
+++ killed by SIGKILL +++
Killed
GDB
[root#localhost PktBlaster]# gdb PktBlaster
(gdb) break main
Breakpoint 1 at 0x43d664: file VTP.c, line 544.
(gdb) run
Starting program: /root/Veryx/PktBlaster/PktBlaster
During startup program terminated with signal SIGKILL, Killed.
While debugging, observed that the bss memory is huge(~6GB). The system has 4GB RAM and I think this could be the reason for the issue.
[root#localhost PktBlaster_1Gig]# size build/unix/bin/PktBlaster
text data bss dec hex filename
375551 55936 6747541120 6747972607 19235e3ff build/unix/bin/PktBlaster
The application contains many .h files and many datastructures and so it is difficult for me to identify why BSS is been raised to 6GB.
Could anyone please suggest how to identify which file is causing this? or any other easier way to debug this?
It seems that problem really is huge BSS size.
I have asked you to show output of LD_DEBUG=all /lib64/ld-linux-x86-64.so.2 /path/to/exe in comments.
/lib64/ld-linux-x86-64.so.2 is runtime linker which is used by OS to load your binary in process memory during execve system call. Runtime linker is responsible for parsing executable format, loading all sections and dependencies in memory, performing all required relocations and so on.
Setting environment variable LD_DEBUG to all we instruct runtime linker to generate debug output.
[root#localhost PktBlaster]# LD_DEBUG=all /lib64/ld-linux-x86-64.so.2
/root/Veryx/PktBlaster/PktBlaster
851: file=/root/Veryx/PktBlaster/PktBlaster [0]; generating link map
/root/Veryx/PktBlaster/PktBlaster: error while loading shared
libraries: /root/Veryx/PktBlaster/PktBlaster: cannot map zero-fill
pages: Cannot allocate memory
Searching for this error message in source code of runtime linker(glibc-2.17 elf/dl-load.c, lines ~1400) we see:
1393 if (zeroend > zeropage)
1394 {
1395 /* Map the remaining zero pages in from the zero fill FD. */
1396 caddr_t mapat;
1397 mapat = __mmap ((caddr_t) zeropage, zeroend - zeropage,
1398 c->prot, MAP_ANON|MAP_PRIVATE|MAP_FIXED,
1399 -1, 0);
1400 if (__builtin_expect (mapat == MAP_FAILED, 0))
1401 {
1402 errstring = N_("cannot map zero-fill pages");
1403 goto call_lose_errno;
1404 }
dl-loader is in process of loading BSS segment, which by optimizations is stored in binary format as just number of bytes, that must be initialized to zero. Loader tries to allocate through mmap zero initialized memory block(MAP_ANONYMOUS) and get error from the OS:
15 #define ENOMEM 12 /* Out of memory */
From man 2 mmap:
ENOMEM No memory is available, or the process's maximum number of
mappings would have been exceeded.
So it seems that for whatever reason OS cannot fulfill loader request for memory. Either some limits are used(systemd, process limit, some security LKM, whatever) or simply there are not enough free memory in kernel.
To determine what object file generates most part of the BSS - use
objdump -j '.bss' -t *.o
I first noticed it while playing with GDB's rbreak ., and then made a minimal example:
(gdb) file hello_world.out
Reading symbols from hello_world.out...done.
(gdb) b _init
Breakpoint 1 at 0x4003e0
(gdb) b _start
Breakpoint 2 at 0x400440
(gdb) run
Starting program: /home/ciro/bak/git/cpp/cheat/gdb/hello_world.out
Breakpoint 1, _init (argc=1, argv=0x7fffffffd698, envp=0x7fffffffd6a8) at ../csu/init-first.c:52
52 ../csu/init-first.c: No such file or directory.
(gdb) continue
Continuing.
Breakpoint 2, 0x0000000000400440 in _start ()
(gdb) continue
Continuing.
Breakpoint 1, 0x00000000004003e0 in _init ()
(gdb) info breakpoints
Num Type Disp Enb Address What
1 breakpoint keep y <MULTIPLE>
breakpoint already hit 2 times
1.1 y 0x00000000004003e0 <_init>
1.2 y 0x00007ffff7a36c20 in _init at ../csu/init-first.c:52
2 breakpoint keep y 0x0000000000400440 <_start>
breakpoint already hit 1 time
Note that there are 2 _init: one in csu/init-first.c, and the other seems to come from sysdeps/x86_64/crti.S. I'm talking about the csu one.
Isn't _start supposed to be the entry point set by the linker, and stored in the ELF header? What mechanism makes _init run first? What is its purpose?
Tested on GCC 4.8, glibc 2.19, GDB 7.7.1 and Ubuntu 14.04.
Where the debugger halts first in your example isn't the real beginning of the process.
In the ELF header there is an entry for the program interpreter (dynamic linker). On Linux 64 bit its value is /lib64/ld-linux-x86-64.so.2. The kernel sets the initial instruction pointer to the entry point of this program interpreter. The symbol name of it is _start too, like the programs _start.
After the dynamic linker has done its work, calling also functions in the program, like _init in glibc, it calls the entry point of the program.
The breakpoint at _start doesn't work for the dynamic linker because it takes only the address of the program's _start.
You can find the entry point address with readelf -h /lib64/ld-linux-x86-64.so.2.
You could also set a breakpoint at _dl_start and print a backtrace to see that this function is called from dynamic linker's _start.
If you download glibc's current source code you can find the entry point of the dynamic loader at glibc-2.21/sysdeps/x86_64/dl-machine.h starting on line 121.