When reading the /proc/$PID/maps you get the mapped memory regions.
Is ther a way to dump one of this regions?
$ cat /proc/18448/maps
...[snip]...
0059e000-005b1000 r-xp 00000000 08:11 40 /usr/local/lib/libgstlightning.so.0.0.0
005b1000-005b2000 r--p 00012000 08:11 40 /usr/local/lib/libgstlightning.so.0.0.0
005b2000-005b3000 rw-p 00013000 08:11 40 /usr/local/lib/libgstlightning.so.0.0.0
...[snip]...
Thanks
Nah! Call ptrace() with PTRACE ATTACH. Then open /proc/<pid>/mem, seek to the region offset, and read the length of the region as given in /proc</pid>/maps.
Here's a program I wrote that does it in C. Here's a module I wrote that does it in Python (and the ptrace binding). For the finish, a program that dumps all regions of a process to files.
Enjoy!
You can attach gdb to the process then dump memory region of length X words starting at location L with this: x/Xw L.
Attaching gdb when you start your process is simple: gdb ./executable then run. If you need to attach to a running process, start gdb then gdb attach pid where pid is is the process ID you care about.
Using dd(1):
sudo dd if=/dev/mem bs=1 skip=$(( 16#0059e000 - 1 )) \
count=$(( 16#005b1000 - 16#0059e000 + 1)) | hexdump -C
Related
I learned from this link Why is address 0x400000 chosen as a start of text segment in x86_64 ABI? that 64-bit Linux process start address by default should be 0x400000, but on my Ubuntu, I only found my bash process starts from a very high base address (0x55971cea6000).
Any one knows why? and how does dynamic linker choose the start address for a 64-bit process?
$ uname -r
5.15.0-25-generic
$ cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS"
...
$ file /usr/bin/bash
/usr/bin/bash: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=33a5554034feb2af38e8c75872058883b2988bc5, for GNU/Linux 3.2.0, stripped
$ ld -verbose | grep -i text-segment
PROVIDE (__executable_start = SEGMENT_START("text-segment", 0x400000)); . = SEGMENT_START("text-segment", 0x400000) + SIZEOF_HEADERS;
$ cat maps
55971ce77000-55971cea6000 r--p 00000000 08:02 153 /usr/bin/bash
55971cea6000-55971cf85000 r-xp 0002f000 08:02 153 /usr/bin/bash
55971cf85000-55971cfbf000 r--p 0010e000 08:02 153 /usr/bin/bash
55971cfc0000-55971cfc4000 r--p 00148000 08:02 153 /usr/bin/bash
55971cfc4000-55971cfcd000 rw-p 0014c000 08:02 153 /usr/bin/bash
55971cfcd000-55971cfd8000 rw-p 00000000 00:00 0
...
$ readelf -h /usr/bin/bash
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: DYN (Position-Independent Executable file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0x32eb0
Start of program headers: 64 (bytes into file)
Start of section headers: 1394600 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 13
Size of section headers: 64 (bytes)
Number of section headers: 30
Section header string table index: 29
I learned from this link Why is address 0x400000 chosen as a start of text segment in x86_64
That address is used for executables (ELF type ET_EXEC).
I only found my bash process starts from a very high base address (0x55971cea6000). Any one knows why?
Because your bash is (newer) position-independent executable (ELF type ET_DYN). It behaves much like a shared library, and is relocated to random address at runtime.
The 0x55971cea6000 address you found will vary from one execution to another. In contrast, ET_EXEC executables can only run correctly when loaded at their "linked at" address (typically 0x400000).
how does dynamic linker choose the start address for a 64-bit process?
The dynamic linker doesn't choose the start address of the executable -- the kernel does (by the time the dynamic linker starts running, the executable has already been mmaped into memory).
The kernel looks at the .e_type in the ELF header and .p_vaddr field of the first program header and goes from there. IFF .e_type == ET_EXEC, then the kernel maps executable segments at their .p_vaddr addresses. For ET_DYN, if ASLR is in effect, the kernel performs mmaps at a random address.
I have been running my application successfully in CentOs6.6. Recently, the hardware(motherboard and RAM) was updated and my application is getting killed now without any reason at all.
[root#localhost PktBlaster]# ./PktBlaster
Killed
File and ldd output
[root#localhost PktBlaster]# file PktBlaster
PktBlaster: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped
[root#localhost PktBlaster]# ldd PktBlaster
not a dynamic executable
Output of strace
[root#localhost PktBlaster]# strace ./PktBlaster
execve("./PktBlaster", ["./PktBlaster"], [/* 30 vars */] <unfinished ...>
+++ killed by SIGKILL +++
Killed
GDB
[root#localhost PktBlaster]# gdb PktBlaster
(gdb) break main
Breakpoint 1 at 0x43d664: file VTP.c, line 544.
(gdb) run
Starting program: /root/Veryx/PktBlaster/PktBlaster
During startup program terminated with signal SIGKILL, Killed.
While debugging, observed that the bss memory is huge(~6GB). The system has 4GB RAM and I think this could be the reason for the issue.
[root#localhost PktBlaster_1Gig]# size build/unix/bin/PktBlaster
text data bss dec hex filename
375551 55936 6747541120 6747972607 19235e3ff build/unix/bin/PktBlaster
The application contains many .h files and many datastructures and so it is difficult for me to identify why BSS is been raised to 6GB.
Could anyone please suggest how to identify which file is causing this? or any other easier way to debug this?
It seems that problem really is huge BSS size.
I have asked you to show output of LD_DEBUG=all /lib64/ld-linux-x86-64.so.2 /path/to/exe in comments.
/lib64/ld-linux-x86-64.so.2 is runtime linker which is used by OS to load your binary in process memory during execve system call. Runtime linker is responsible for parsing executable format, loading all sections and dependencies in memory, performing all required relocations and so on.
Setting environment variable LD_DEBUG to all we instruct runtime linker to generate debug output.
[root#localhost PktBlaster]# LD_DEBUG=all /lib64/ld-linux-x86-64.so.2
/root/Veryx/PktBlaster/PktBlaster
851: file=/root/Veryx/PktBlaster/PktBlaster [0]; generating link map
/root/Veryx/PktBlaster/PktBlaster: error while loading shared
libraries: /root/Veryx/PktBlaster/PktBlaster: cannot map zero-fill
pages: Cannot allocate memory
Searching for this error message in source code of runtime linker(glibc-2.17 elf/dl-load.c, lines ~1400) we see:
1393 if (zeroend > zeropage)
1394 {
1395 /* Map the remaining zero pages in from the zero fill FD. */
1396 caddr_t mapat;
1397 mapat = __mmap ((caddr_t) zeropage, zeroend - zeropage,
1398 c->prot, MAP_ANON|MAP_PRIVATE|MAP_FIXED,
1399 -1, 0);
1400 if (__builtin_expect (mapat == MAP_FAILED, 0))
1401 {
1402 errstring = N_("cannot map zero-fill pages");
1403 goto call_lose_errno;
1404 }
dl-loader is in process of loading BSS segment, which by optimizations is stored in binary format as just number of bytes, that must be initialized to zero. Loader tries to allocate through mmap zero initialized memory block(MAP_ANONYMOUS) and get error from the OS:
15 #define ENOMEM 12 /* Out of memory */
From man 2 mmap:
ENOMEM No memory is available, or the process's maximum number of
mappings would have been exceeded.
So it seems that for whatever reason OS cannot fulfill loader request for memory. Either some limits are used(systemd, process limit, some security LKM, whatever) or simply there are not enough free memory in kernel.
To determine what object file generates most part of the BSS - use
objdump -j '.bss' -t *.o
I am exploring the possibility to back the text and data segment with hugepages following the guide in https://github.com/libhugetlbfs/libhugetlbfs/blob/master/HOWTO.
I have relinked the application as suggested by adding "-B/usr/share/libhugetlbfs -Wl,--hugetlbfs-align" and started the application with "hugectl --text --data --heap --bss /path/to/my/application".
But I am not very sure how to verify whether the data and text segments are indeed copied to the files on the hugetlbfs filesystem.
Checking the /proc/{pid}/maps, it could be seen that the hugepages are used for heap but not for text and data segments as the first two address ranges are mapped to the application but not the hugepage file system.
Is my understanding correct? Actually I suspect my conclusion that hugepages are used for heap from /proc/{pid}/maps is also incorrect.
How should I verify whether the data and text segments are backed in the hugepages? I know that data and text segments will be copied to hugetlbfs filesystem if successful but how to verify it?
Thanks!
output of /proc/{pid}/maps
00400000-00d2c000 r-xp 00000000 fd:02 46153351 /path/to/my/application
00f2b000-00fa3000 rw-p 0092b000 fd:02 46153351 /path/to/my/application
00fa3000-00fbb000 rw-p 00000000 00:00 0
02a0c000-02a2d000 rw-p 00000000 00:00 0 [heap]
40000000-80000000 rw-p 00000000 00:15 2476090 /dev/hugepages-1G/libhugetlbfs.tmp.nS7exn (deleted)
check
/proc/$pid/numa_maps
contains information about each memory area used by a given process allowing--among other information--the determination of which nodes were used for the pages.
for formar see http://linux.die.net/man/5/numa_maps
If you set HUGETLB_DEBUG=1 variable, it will tell you whole lot of useful information. One of them is this:
INFO: Segment 2's aligned memsz is too small: 0x864 < 0xffffffffffffffff
If it is successful it looks like:
libhugetlbfs [zupa:154297]: INFO: Segment 0 (phdr 2): 0x400000-0x400864 (filesz=0x864) (prot = 0x5)
libhugetlbfs [zupa:154297]: INFO: Segment 1 (phdr 3): 0x600868-0x600af8 (filesz=0x27c) (prot = 0x3)
libhugetlbfs [zupa:154297]: DEBUG: Total memsz = 0xaf4, memsz of largest segment = 0x864
libhugetlbfs [zupa:154297]: INFO: libhugetlbfs version: 2.16 (modified)
libhugetlbfs [zupa:154951]: INFO: Mapped hugeseg at 0x2aaaaac00000. Copying 0x864 bytes and 0 extra bytes from 0x400000...done
libhugetlbfs [zupa:154297]: INFO: Prepare succeeded
libhugetlbfs [zupa:154952]: INFO: Mapped hugeseg at 0x2aaaaac00000. Copying 0x27c bytes and 0 extra bytes from 0x600868...done
libhugetlbfs [zupa:154297]: INFO: Prepare succeeded
I am using GDB to debug a program that uses libpthread. There is an error
happening in pthread_create and need to step into that function. Unfortunately when I am debugging my program, it does not load the shared library symbols properly so I can't step over the source code and examine program behaviour meaningfully.. This is the output as soon as I start gdb.
Remote debugging using 127.0.0.1:21293
warning: limiting remote suggested packet size (206696 bytes) to 16384
Failed to read a valid object file image from memory.
So I believe the last message is related to the failure to read debugging symbols. This is despite having the libc6-dbg package installed. This is the truncated output of "where" at a point just before a SIGSEGV is encountered (in pthread_create, the function I want to examine in the debugger)
#0 0x68200ce2 in ?? ()
#1 0x68403cbf in ?? ()
#2 0x687571b0 in ?? ()
#3 0x6874c638 in ?? ()
#4 0x68867a72 in ?? ()
....
The process' /proc/.../maps shows where libpthread is mapped into memory.
683f8000-68410000 r-xp 00000000 08:01 3017052 /lib/i386-linux-gnu/i686/cmov/libpthread-2.19.so
68410000-68411000 r--p 00017000 08:01 3017052 /lib/i386-linux-gnu/i686/cmov/libpthread-2.19.so
68411000-68412000 rw-p 00018000 08:01 3017052 /lib/i386-linux-gnu/i686/cmov/libpthread-2.19.so
I believe that if I could only manually load the debugging symbols into gdb, then I will be able to step over the source code and find the source of my memory error. However I am unsure about how to do this.
I am debugging a 32 bit program on x86_64 Debian. What should I do to load libpthread symbols into GDB so that I can debug it meaningfully?
Loading debug symbols for a shared library
If the shared library is stripped, and the debug symbols are provided as a separate file, you need to load them after the shared library was loaded by the linker. The symbols should be loaded on the memory address where the shared library is loaded.
Following is an example of loading the symbols:
start gdb
~$ gdb a.out
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
(gdb) info sharedlibrary
No shared libraries loaded at this time.
Create break point (on main or any other place) and start the debug
(gdb) b 35
Breakpoint 1 at 0xd7c: file test.c, line 35.
(gdb) r
Starting program: /root/testing/lib-test/a.out
Find the memory location where the shared library that you want to debug was loaded (in this example the library is libtest.so.1
(gdb) info sharedlibrary
From To Syms Read Shared Object Library
0x0000fffff7fcd0c0 0x0000fffff7fe5468 Yes (*) /lib/ld-linux-aarch64.so.1
0x0000fffff7f9f890 0x0000fffff7fb65c0 Yes (*) /usr/local/lib/libtest.so.1
0x0000fffff7e4bbc0 0x0000fffff7f3b190 Yes /lib/aarch64-linux-gnu/libc.so.6
0x0000fffff7dfea50 0x0000fffff7e0ddec Yes /lib/aarch64-linux-gnu/libpthread.so.0
so, the library is loaded starting from memory address 0x0000fffff7f9f890
Load the symbol file with the address from the share library
(gdb) add-symbol-file ./libsrc/libtest.dbg 0x0000fffff7f9f890
add symbol table from file "./libsrc/libtest.dbg" at
.text_addr = 0xfffff7f9f890
(y or n) y
Reading symbols from ./libsrc/libtest.dbg...
After this, you can trace the execution flow inside the library, list the lines of the source code, inspect variables by name, etc.
Here goes what I did.
Download MESA and build with --enable-debug configuration.
The build results are all in ${MESA_SRC}/lib directory.
I set the LIBGL_DRIVERS_PATH and LD_LIBRARY_PATH to the ${MESA_SRC}/lib directory to use the built results rather than the original library in my local PC. I checked that OpenGL apps are using the libraries in ${MESA_SRC}/lib by using ldd as follows:
~/work/mesa$ ldd /usr/bin/glxgears | grep libGL
libGL.so.1 => lib/libGL.so.1 (0x00007f81aa9cc000)
where lib/libGL.so.1 is the build results from mesa source code.
Now, I wrote a very simple OpenGL app. Let say A. And run gdb and breakpoint to main.
work/mesa$ gdb A
GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
(gdb) b main
Breakpoint 1 at 0x4015af: file A.c, line 168.
(gdb) r
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Breakpoint 1, main (argc=1, argv=0x7fffffffdee8) at compositor-pbo.c:168
168 {
(gdb)
After that, I tried to load solib-search-path
(gdb) set solib-search-path /home/jyyoo/work/mesa/lib
where /home/jyyoo/work/mesa is the ${MESA_SRC}. But, I don't see any so files loading. As far as I know, if I issue solib-search-path, the loading of so files in the corresponding directory should be listed with. But, I don't see any. I know that there is a list of so files as follows:
$ ls /home/jyyoo/work/mesa/lib/*.so
/home/jyyoo/work/mesa/lib/i965_dri.so
/home/jyyoo/work/mesa/lib/libglapi.so
/home/jyyoo/work/mesa/lib/libGL.so
/home/jyyoo/work/mesa/lib/libEGL.so
/home/jyyoo/work/mesa/lib/libGLESv1_CM.so
/home/jyyoo/work/mesa/lib/mesa_dri_drivers.so
/home/jyyoo/work/mesa/lib/libgbm.so
/home/jyyoo/work/mesa/lib/libGLESv2.so
So, if I try to 's' in gl functions, it does not step into...
Additionally, if I see the address of glBindBuffer, it shows
(gdb) p glBindBuffer
$1 = {<text variable, no debug info>} 0x7ffff7969f60 <glBindBufferARB>
But, the maps of this process does not load anything in that address:
/proc/<pid of gdb>$ cat maps
... snip ...
7fff5f652000-7fff5f673000 rw-p 00000000 00:00 0 [stack]
7fff5f74d000-7fff5f74f000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Seems the apis are written in assembly language, so no source can be displayed. Just set a breakpoint at glBindBuffer, then run, then the program will pause at the breakpoint, then type 'step' or 'next', you will step into the real function behind the api.