I'm trying to get the line of source code from addr2line for the rasbian 5.4.y kernel.
My host environment is ubuntu18.04.2 on virtualbox, and I'm compiling the kernel with arm-linux-gnueabihf- cross-compiler.
I compiled the kernel with bcm2711_defconfig configuration since the target machine is Raspberry Pi 4, following the official guide, https://www.raspberrypi.org/documentation/linux/kernel/building.md, with 32-bit arm arch configuration. I didn't modify any kernel configuration at all.
I obtained the address of a function (_local_bh_enable here, for an instance) from vmlinux by using objdump as below,
$ arm-linux-gnueabihf-objdump -x linux/vmlinux | grep _local_bh_enable
...
c0227a28 g F .text 00000098 _local_bh_enable
As you can see above, I got the address for _local_bh_enable as 0xc0227a28.
Then I ran addr2line to get the line of the address but some strange result I've got as below.
$ arm-linux-gnueabihf-addr2line -fe linux/vmlinux -a 0xc0227a28
0xc0227a28
_local_bh_enable
.tmp_vmlinux.kallsyms2.o:?
I don't get what that means. Isn't it supposed to give the source file name with the line number on it?
I've also tried with many other functions but ended up all the same with the ".tmp_vmlinux.kallsyms2.o:?", not the code line I'm expecting to get.
Am I missing something here? Please give me any help for this.
Thanks in advance.
I checked the kernel configuration, and I've found out that the CONFIG_DEBUG_INFO was not set. It seems the cause.
I'm compiling the kernel with CONFIG_DEBUG_INFO set, let me see whether it's working.
Related
I am looking for help with GDB to reverse engineer shared library written in C that is preloaded in /etc/ld.so.preload.
Current library hooks accept() call if source port is correct it returns reverse shell back to user.
Strings command doesn't give out source port, so my target is to try to find it within GDB.
Program consist of two files headers.h where I have my definitions and variable #define SECRET_PORT 11111
source.c contains accept hook with reverse shell.
My problem is I cannot figure out a way how to retrieve PORT within GDB - I can load mylib.so within gdb and run: info functions to see whats inside - I can see accept function but when I try to disass accept I only get instructions that I barely can understand.
Problem when I run mylib it gives out SIGSEGV (maybe thats the reason I cannot see variables) there is no main function where to set break and if I do set it on function accept is still gives SIGSEGV error.
I tested with starti instead of run then I got Program stopped 0xSOMEADRESGOESHERE in deregister_tm_clones() I don't even know if this is correct way to test .so file. maybe there are some oser switches.
Im thinking I need to find a way how to set BP in HTONS() checking function where if statement compares source port and extract values from there but so far no luck.
p.s. when mylib is loaded in gdb there is message No debugging symbols found. So I cannot run like list accept or anything like that to view a source.
Compilation code gcc -Wall -shared -fPIC mylib.c -o mylib.so -ldl
Im thinking I need to find a way how to set BP in HTONS() checking function where if statement compares source port and extract values from there
You don't need to do that -- the instructions will be the same whether you run the application, or disassemble the function without running.
Compilation code ...
So you are trying to reverse-engineer the library for which you have a source?
That makes it very easy to find the constant you are looking for.
Start by setting the constant to easily recognizable value, e.g. 0x12131415. Compile the library and disassemble it. Look for your constant.
If you don't see it, save the disassembled output, and rebuild the library with a different value, e.g. 0xA1B1C1D1. Disassemble it again and compare to previous disassembled output. It should be easy to spot the difference.
P.S. If you really want to debug this library with a live process, do this:
gdb ./myprog
(gdb) set env LD_PRELOAD /path/to/mylib.so
(gdb) run
At this point, you should be able to set breakpoints and observe your library "in action".
Ok managed solve this with a help
when running GDB on shared library You will have to check hex value for 11111 and it should be 2B67 so in registers this will become something like 0x2b67 & it will be passed to htons() as check for source port.
So let's assume I didn't have the source code I could still run: gdb -q *.so
then: info functions and see with disass functionNameGoesHere where some accept / htons calls are made. Correct value should be found right above htons line.
Then decoded hex to dec and thats how You can find it.
This took some while to figure out as I coudn't set BP's.
Again thanks for input from community! Cheers
I use gdb (ddd) to debug my C/C++ projects.
Whenever an assert fails, I can debug the program as normal and backtrace to the assert which failed, but first I get an annoying popup
I assume raise.c defines assert, but ddd is looking within my home directory instead of /usr/... or something like that.
I may or may not have the debugging packages installed (I'm on Ubuntu), but the main question is: why is gdb looking within $HOME for this source?
Since the glibc debuginfo package libc6-dbg is installed on your system, gdb will look in /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.23.so to map instruction addresses into source file names and line numbers.
The section of that file that contains the info for __GI_raise has the following attributes that indicate where the source code might be found (on Ubuntu 16.04):
<0><688bd>: Abbrev Number: 1 (DW_TAG_compile_unit)
<688c3> DW_AT_name : (indirect string, offset: 0x9137): ../sysdeps/unix/sysv/linux/raise.c
<688c7> DW_AT_comp_dir : (indirect string, offset: 0x9010): /build/glibc-Cl5G7W/glibc-2.23/signal
Ubuntu doesn't ship source code in the base distribution, so your system doesn't have any glibc source in /build/glibc-Cl5G7W/glibc-2.23, so gdb looks (unsuccessfully) for raise.c in a few other directories according to its rules Specifying Source Directories and eventually gives up, with the error message
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
Then, ddd calls SourceView::full_path("../sysdeps/unix/sysv/linux/raise.c") to canonicalize the pathname, taking your working directory into account, and displays the error dialog box in your question.
See GDB complaining about missing raise.c for how to install glibc source on Ubuntu.
I have recently been trying to compile dropbear for 64bit arch android devices. I got the program to compile and run, but when I log in to the server it kicks me off with an error.
I configured & compiled like so:
cp ../config.guess ../config.sub .
./configure --build=x86_64-windows --host=arm-linux-androideabi \
--disable-zlib --disable-largefile --disable-loginfunc \
--disable-shadow --disable-utmp --disable-utmpx --disable-wtmp \
--disable-wtmpx --disable-pututline --disable-pututxline --disable-lastlog \
CFLAGS='-Os -W -Wall -fPIE' LDFLAGS='-fPIE -pie'
make dropbear
The error:
CANNOT LINK EXECUTABLE: "/system/lib/libc++.so" is 32-bit instead of 64-bit
page record for 0x7f800b8010 was not found (block_size=32)
I believe the error is caused by dropbear linking sh and other executables to the library in wrong location.
The device has two locations for libc++
#1 /system/lib/libc++.so
#2 /system/lib64/libc++.so
How can I tell/force the compiled binary to use the lib(s) in
/sytem/lib64
If giving an example please show me exactly what needs to be added, so I can apply the information correctly. I'm almost sure the fix is simple but I'm not a programmer. I do this stuff when I'm desperate.
Turns out there was a hard link in srv-chansession.c that needed to be modified
From:
addnewvar("LD_LIBRARY_PATH", "/system/lib");
To:
addnewvar("LD_LIBRARY_PATH", "/system/lib64");
It may actually be better to comment out that line and let androids linkers take care of it.
Now I have working a working dropbear ssh server on my 64 bit android device.
Consider the following Linux kernel dump stack trace; e.g., you can trigger a panic from the kernel source code by calling panic("debugging a Linux kernel panic");:
[<001360ac>] (unwind_backtrace+0x0/0xf8) from [<00147b7c>] (warn_slowpath_common+0x50/0x60)
[<00147b7c>] (warn_slowpath_common+0x50/0x60) from [<00147c40>] (warn_slowpath_null+0x1c/0x24)
[<00147c40>] (warn_slowpath_null+0x1c/0x24) from [<0014de44>] (local_bh_enable_ip+0xa0/0xac)
[<0014de44>] (local_bh_enable_ip+0xa0/0xac) from [<0019594c>] (bdi_register+0xec/0x150)
In unwind_backtrace+0x0/0xf8 what does +0x0/0xf8 stand for?
How can I see the C code of unwind_backtrace+0x0/0xf8?
How to interpret the panic's content?
It's just an ordinary backtrace, those functions are called in reverse order (first one called was called by the previous one and so on):
unwind_backtrace+0x0/0xf8
warn_slowpath_common+0x50/0x60
warn_slowpath_null+0x1c/0x24
ocal_bh_enable_ip+0xa0/0xac
bdi_register+0xec/0x150
The bdi_register+0xec/0x150 is the symbol + the offset/length there's more information about that in Understanding a Kernel Oops and how you can debug a kernel oops. Also there's this excellent tutorial on Debugging the Kernel
Note: as suggested below by Eugene, you may want to try addr2line first, it still needs an image with debugging symbols though, for example
addr2line -e vmlinux_with_debug_info 0019594c(+offset)
Here are two alternatives for addr2line. Assuming you have the proper target's toolchain, you can do one of the following:
Use objdump:
locate your vmlinux or the .ko file under the kernel root directory, then disassemble the object file :
objdump -dS vmlinux > /tmp/kernel.s
Open the generated assembly file, /tmp/kernel.s. with a text editor such as vim. Go to
unwind_backtrace+0x0/0xf8, i.e. search for the address of unwind_backtrace + the offset. Finally, you have located the problematic part in your source code.
Use gdb:
IMO, an even more elegant option is to use the one and only gdb. Assuming you have the suitable toolchain on your host machine:
Run gdb <path-to-vmlinux>.
Execute in gdb's prompt: list *(unwind_backtrace+0x10).
For additional information, you may checkout the following resources:
Kernel Debugging Tricks.
Debugging The Linux Kernel Using Gdb
In unwind_backtrace+0x0/0xf8 what the +0x0/0xf8 stands for?
The first number (+0x0) is the offset from the beginning of the function (unwind_backtrace in this case). The second number (0xf8) is the total length of the function. Given these two pieces of information, if you already have a hunch about where the fault occurred this might be enough to confirm your suspicion (you can tell (roughly) how far along in the function you were).
To get the exact source line of the corresponding instruction (generally better than hunches), use addr2line or the other methods in other answers.
I'm trying to debug kernel module. I suspect to have there some memory leaks. To check it I have prepared build with enabled Memory leak debugging for kernel and modules. And I got some warning from that:
[11839.429168] slab error in verify_redzone_free(): cache `size-64': memory outside object was overwritten
[11839.438659] [<c005575c>] (unwind_backtrace+0x0/0x164) from [<c0116ca0>] (kfree+0x278/0x4d8)
[11839.447357] [<c0116ca0>] (kfree+0x278/0x4d8) from [<bf083f48>] (some_function+0x18/0x1c [my_module])
[11839.457214] [<bf083f48>] (some_function+0x18/0x1c [my_module]) from [<bf08762c>] (some_function+0x174/0x718 [my_module])
[11839.470184] [<bf08762c>] (some_function+0x174/0x718 [my_module]) from [<bf0a56b8>] (some_function+0x12c/0x16c [my_module])
[11839.483917] [<bf0a56b8>] (some_function+0x12c/0x16c [my_module]) from [<bf085790>] (some_function+0x8/0x10 [my_module])
[11839.496368] [<bf085790>] (some_function+0x8/0x10 [my_module]) from [<bf07b74c>] (some_function+0x358/0x6d4 [my_module])
[11839.507476] [<bf07b74c>] (some_function+0x358/0x6d4 [my_module]) from [<c00a60f8>] (worker_thread+0x1e8/0x284)
[11839.517211] [<c00a60f8>] (worker_thread+0x1e8/0x284) from [<c00a9edc>] (kthread+0x78/0x80)
[11839.525543] [<c00a9edc>] (kthread+0x78/0x80) from [<c004f8fc>] (kernel_thread_exit+0x0/0x8)
There is no problem to translate addresses which points to kernel:
$ addr2line -f -e vmlinux.kmeml c0116ca0
verify_redzone_free
/[...]/kernel/mm/slab.c:2922
But I can't do that if addresses are from my_module:
$ addr2line -f -e vmlinux.kmeml bf0a56b8
??
??:0
I was also trying with module file:
$ addr2line -f -e my_module.ko bf0a56b8
??
??:0
How can I translate this addresses to files and line numbers?
I suppose the module is built with debug info included. If so, you can use gdb or objdump to find out which source file and line each address belongs to. Something like this:
$ gdb "$(modinfo -n my_module)"
(gdb) list *(some_function+0x12c)
Gdb will now tell the name of the source file and the line in it.
You can also do a similar thing with objdump but it is a bit more difficult. First, disassemble the module:
objdump -dSlr my_module.ko > my_module.disasm
When called with -S option, objdump will include the source lines in the resulting listing where appropriate.
You can now scroll the listing down to the code of some_function, find the instruction at offset 0x12c from the beginning of the function. The source line will be indicated above it.
EDIT:
After many experiments, I found that although addr2line can indeed be used for kernel modules, eu-addr2line (a similar tool from elfutils) seems to be more reliable. That is, sometimes addr2line output incorrect source lines but eu-add2line did things right.
To use eu-addr2line, one may need to install libdw and libebl libraries if they are not already installed along with elfutils.
The usage is similar to that of addr2line:
eu-addr2line -f -e <path_to_the_module> -j <section_name> <offset_in_section>
If the debug information for a kernel module is stored in separate file (this is often the case for the kernels provided by the major Linux distros), the path to that file should be used as <path_to_the_module>.
You indeed need to run addr2line on your kernel module and not kernel but there is a twist -
the kernel module file uses relative addresses, the crash address you have is actually composed of:
offset inside module + module load address is memory.
So what you need to do is find the kernel moduel load address is memory first by doing cat /proc/modules, finding to what module that address belongs to, in case you don't know, subtract the module load address from the crash address and feed that to addr2line
good luck
Maybe you should use -g parameter to compile the module.