I am using Eclipse to develop and remotely debug some software for an ARM Processor. Unfortunately the software I am writing is multi-threaded and I am unable to debug it. If I place a break-point in the thread code, i get the following message:
Child terminated with signal = 5
Child terminated with signal = 0x5
GDBserver exiting
After doing quite a bit of Googling, I found a "solution" that proposed using this:
strip --strip-debug libpthread.so.0
Unfortunately, I still get the termination errors.
I would really appreciate your help in getting this figured out!
Thanks!
First, this (and subsequent) error(s):
cc1.exe: error: unrecognized command line option "-fstrip-debug"
is caused by adding strip --strip-debug etc. to the GCC command line. That is obviously bogus thing to do, and not at all what your googling suggested. (You might want to clean up your question to remove references to these errors; they have nothing to do with your problem.)
What it did (or should have) suggested, is using strip --strip-debug libpthread.so.0 instead of using strip libpthread.so.0.
This is because GDB can not work with threads if your libpthread.so.0 is fully stripped.
It can be stripped of debug symbols (which is what strip --strip-debug libpthread.so.0 does), but stripping it of all symbols (which is what strip libpthread.so.0 does) is a bad idea(TM).
Since you are (apparently) not yourself building libpthread.so.0, you shouldn't need to strip it either.
You should however verify that the provider of your toolchain did not screw it up. The following command should not report no symbols, and should in fact print a matching nptl_version (as a defined symbol):
nm /path/to/target/libpthread.so.0 | grep nptl_version
Assuming all is well so far, we can now diagnose your problem, except ... you didn't provide sufficient info ;-( In particular, when you run GDB, it should print something like using /path/to/libthread_db.so.0. You might have to hunt for GDB console in Eclipse, or you might want to run GDB from command line, so you see exactly what it prints.
It is crucial that the version of libthread_db.so.0 (for host) matches the version of lipthread.so.0 (for target). They should both be provided by your toolchain vendor.
Your problem is most likely that either GDB can't find libthread_db.so.0 at all, or that it finds the wrong one.
Related
I am looking for help with GDB to reverse engineer shared library written in C that is preloaded in /etc/ld.so.preload.
Current library hooks accept() call if source port is correct it returns reverse shell back to user.
Strings command doesn't give out source port, so my target is to try to find it within GDB.
Program consist of two files headers.h where I have my definitions and variable #define SECRET_PORT 11111
source.c contains accept hook with reverse shell.
My problem is I cannot figure out a way how to retrieve PORT within GDB - I can load mylib.so within gdb and run: info functions to see whats inside - I can see accept function but when I try to disass accept I only get instructions that I barely can understand.
Problem when I run mylib it gives out SIGSEGV (maybe thats the reason I cannot see variables) there is no main function where to set break and if I do set it on function accept is still gives SIGSEGV error.
I tested with starti instead of run then I got Program stopped 0xSOMEADRESGOESHERE in deregister_tm_clones() I don't even know if this is correct way to test .so file. maybe there are some oser switches.
Im thinking I need to find a way how to set BP in HTONS() checking function where if statement compares source port and extract values from there but so far no luck.
p.s. when mylib is loaded in gdb there is message No debugging symbols found. So I cannot run like list accept or anything like that to view a source.
Compilation code gcc -Wall -shared -fPIC mylib.c -o mylib.so -ldl
Im thinking I need to find a way how to set BP in HTONS() checking function where if statement compares source port and extract values from there
You don't need to do that -- the instructions will be the same whether you run the application, or disassemble the function without running.
Compilation code ...
So you are trying to reverse-engineer the library for which you have a source?
That makes it very easy to find the constant you are looking for.
Start by setting the constant to easily recognizable value, e.g. 0x12131415. Compile the library and disassemble it. Look for your constant.
If you don't see it, save the disassembled output, and rebuild the library with a different value, e.g. 0xA1B1C1D1. Disassemble it again and compare to previous disassembled output. It should be easy to spot the difference.
P.S. If you really want to debug this library with a live process, do this:
gdb ./myprog
(gdb) set env LD_PRELOAD /path/to/mylib.so
(gdb) run
At this point, you should be able to set breakpoints and observe your library "in action".
Ok managed solve this with a help
when running GDB on shared library You will have to check hex value for 11111 and it should be 2B67 so in registers this will become something like 0x2b67 & it will be passed to htons() as check for source port.
So let's assume I didn't have the source code I could still run: gdb -q *.so
then: info functions and see with disass functionNameGoesHere where some accept / htons calls are made. Correct value should be found right above htons line.
Then decoded hex to dec and thats how You can find it.
This took some while to figure out as I coudn't set BP's.
Again thanks for input from community! Cheers
I wanted to debug printf function, so when I step inside the printf function (gdb debugger) it showed me this:
__printf (format=0x80484d0 " my name is Adam") at printf.c:28
28 printf.c: No such file or directory.
What is the meaning of this?
And when I again started step then there are a lot more statements like this.
Please help me to understand this.
I think it's pretty clear. There is a place where the gdb expects the source code to be, so download glibc's source code and put it there. I think the error message contains the full path.
If it's a linux distro it's fairly simple in fact because usually source packages are shipped too. Otherwise you need to find the source code yourself, note that it MUST be exactly the same that was used to compile the c library components, not just the same version because distributors often make changes to the sources.
Well, for the debugger to show you the code that was compiled into the binaries you're using, you need the original code somewhere.
You don't seem to have that, so your debugger can't find it.
Notice that you usually do not want to debug the source code of your std library functions, but only the way they are being called. For that, the usual "debug symbol" packages of your operating systems are optimal.
As others have answered, GDB was unable to find the source file.
For the C runtime libraries, Linux distributions may provide a debuginfo RPM that you can install, which may allow GDB to view the files. For example:
$ yum search glibc-debuginfo
...
glibc-debuginfo.x86_64 : Debug information for package glibc
glibc-debuginfo-common.x86_64 : Debug information for package glibc
...
The glibc package and the glibc-debuginfo are a matched pair. There is no explicit dependency, but glibc-debuginfo package won't work unless it is matched with the same version of glibc.
If you have the sources unpacked somewhere, but not where GDB is expecting them to be, you can attempt to use either the directory or the set substitute-path command to let GDB know where the sources are.
The directory command tells GDB to prepend a prefix ahead of any source file path it is attempting to find. For example, if the source tree is actually located under the /tmp, you could use:
(gdb) directory /tmp
The set substitute-path command is used to tell GDB to replace a matching prefix in a source file path with a different path prefix. For example, if the compiled source file was in /build/path/source.c, but in debugging the source file is actually in /usr/home/alice/release-1.1/source.c, then you could use:
(gdb) set substitute-path /build/path /usr/home/alice/release-1.1
The command assumes that you are only specifying a complete path names, so it won't perform the substitution on /build/pathological/source.c.
I am trying to navigate and understand whoami (and other coreutils) all the way down to the lowest level source code, just as an exercise.
My dive so far:
Where is the actual binary?
which whoami
/usr/bin/whoami
Where is it maintained?
http://www.gnu.org/software/coreutils/coreutils.html
How do I get source?
git clone git://git.sv.gnu.org/coreutils
Where is whoami source code within the repository?
# find . | grep whoami
./man/whoami.x
./man/whoami.1
./src/whoami.c
./src/whoami
./src/whoami.o
./src/.deps/src_libsinglebin_whoami_a-whoami.Po
./src/.deps/whoami.Po
relevant line (84):
uid = geteuid ();
This is approximately where my rabbit hole stops. geteuid() is mentioned in gnulib/lib/euidaccess.c, but not explicitly defined AFAICT. It's also referenced in /usr/local/unistd.h as extern but there's no heavy lifting related to grabbing a uid that I can see.
I got here by mostly grepping for geteuid within known system headers and includes as I'm having trouble backtracing its definition.
Question: How can I dive down further and explore the source code of geteuid()? What is the most efficient way to explore this codebase quickly without grepping around?
I'm on Ubuntu server 15.04 using Vim and some ctags (which hasn't been very helpful for navigating existing system headers). I'm a terrible developer and this is my method of learning, though I can't get through this roadblock.
Normally you should read the documentation for geteuid. You can either read GNU documentation, the specification from the Open Group or consult the man page.
If that doesn't help you could install the debug symbols for the c-library (it's called libc6-dbg or something similar) and download the source code for libc6) then you point out the path to the source file when you step into the library.
In this case I don't think this would take you much further, what probably happens in geteuid is that it simply issues an actual syscall and then it's into kernel space. You cannot debug that (kernel) code in the same way as you would debug a normal program.
So in your case you should better consult the documentation and read it carefully and try to figure out why geteuid doesn't return what you expect. Probably this will lead to you changing your expectation of what geteuid should return to match what's actually returned.
Consider the following Linux kernel dump stack trace; e.g., you can trigger a panic from the kernel source code by calling panic("debugging a Linux kernel panic");:
[<001360ac>] (unwind_backtrace+0x0/0xf8) from [<00147b7c>] (warn_slowpath_common+0x50/0x60)
[<00147b7c>] (warn_slowpath_common+0x50/0x60) from [<00147c40>] (warn_slowpath_null+0x1c/0x24)
[<00147c40>] (warn_slowpath_null+0x1c/0x24) from [<0014de44>] (local_bh_enable_ip+0xa0/0xac)
[<0014de44>] (local_bh_enable_ip+0xa0/0xac) from [<0019594c>] (bdi_register+0xec/0x150)
In unwind_backtrace+0x0/0xf8 what does +0x0/0xf8 stand for?
How can I see the C code of unwind_backtrace+0x0/0xf8?
How to interpret the panic's content?
It's just an ordinary backtrace, those functions are called in reverse order (first one called was called by the previous one and so on):
unwind_backtrace+0x0/0xf8
warn_slowpath_common+0x50/0x60
warn_slowpath_null+0x1c/0x24
ocal_bh_enable_ip+0xa0/0xac
bdi_register+0xec/0x150
The bdi_register+0xec/0x150 is the symbol + the offset/length there's more information about that in Understanding a Kernel Oops and how you can debug a kernel oops. Also there's this excellent tutorial on Debugging the Kernel
Note: as suggested below by Eugene, you may want to try addr2line first, it still needs an image with debugging symbols though, for example
addr2line -e vmlinux_with_debug_info 0019594c(+offset)
Here are two alternatives for addr2line. Assuming you have the proper target's toolchain, you can do one of the following:
Use objdump:
locate your vmlinux or the .ko file under the kernel root directory, then disassemble the object file :
objdump -dS vmlinux > /tmp/kernel.s
Open the generated assembly file, /tmp/kernel.s. with a text editor such as vim. Go to
unwind_backtrace+0x0/0xf8, i.e. search for the address of unwind_backtrace + the offset. Finally, you have located the problematic part in your source code.
Use gdb:
IMO, an even more elegant option is to use the one and only gdb. Assuming you have the suitable toolchain on your host machine:
Run gdb <path-to-vmlinux>.
Execute in gdb's prompt: list *(unwind_backtrace+0x10).
For additional information, you may checkout the following resources:
Kernel Debugging Tricks.
Debugging The Linux Kernel Using Gdb
In unwind_backtrace+0x0/0xf8 what the +0x0/0xf8 stands for?
The first number (+0x0) is the offset from the beginning of the function (unwind_backtrace in this case). The second number (0xf8) is the total length of the function. Given these two pieces of information, if you already have a hunch about where the fault occurred this might be enough to confirm your suspicion (you can tell (roughly) how far along in the function you were).
To get the exact source line of the corresponding instruction (generally better than hunches), use addr2line or the other methods in other answers.
I'm having some problems with a program causing a segmentation fault when run on a Mac. I'm putting together an entry for the IOCCC, which means the following things are true about my program:
It's a very small C program in a single file called prog.c
I won't post it here, because it won't help (and would probably render the contest entry invalid)
It compiles cleanly under gcc using "cc -o prog prog.c -Wall"
Despite (or, more accurately, because of) the fact it contains a bunch of really bizarre uses of C, it has been constructed extremely carefully. I don't know of any part of it which is careless with memory (which is not to say that there can't possibly be bugs, just that if there are they're not likely to be obvious ones)
I'm primarily a Windows user, but several years ago I successfully compiled and ran it on several windows machines, a couple of Macs and a Linux box, with no problems. The code hasn't changed since then, but I no longer have access to those machines.
I don't have a Linux machine to re-test on, but as one final test, I tried compiling and running it on a MacBook Pro - Mac OSX 10.6.7, Xcode 4.2 (i.e. GCC 4.2.1). Again, it compiles cleanly from the command line. It seems that on a Mac typing "prog" won't make the compiled program run, but "open prog" seems to. Nothing happens for about 10 seconds (my program takes about a minute to run when it's successful), but then it just says "Segmentation fault", and ends.
Here is what I've tried, to track down the problem, using answers mostly gleaned from this useful StackOverflow thread:
On Windows, peppered the code with _ASSERTE(_CrtCheckMemory()); - The code ran dog-slow, but ran successfully. None of the asserts fired (they do when I deliberately add horrible code to ensure that _CrtCheckMemory and _ASSERTE are working as expected, but not otherwise)
On the Mac, I tried Mudflap. I tried to build the code using variations of "g++ -fmudflap -fstack-protector-all -lmudflap -Wall -o prog prog.c", which just produces the error "cc1plus: error: mf-runtime.h: No such file or directory". Googling the matter didn't bring up anything conclusive, but there does seem to be a feeling that Mudflap just doesn't work on Macs.
Also on the Mac, I tried Valgrind. I installed and built it, and built my code using "cc -o prog -g -O0 prog.c". Running Valgrind with the command "valgrind --leak-check=yes prog" produces the error "valgrind: prog: command not found". Remembering you have you "open" an exectable on a Mac I tried "valgrind --leak-check=yes open prog", which appears to run the program, and also runs Valgrind, which finds no problems. However, Valgrind is failing to find problems for me even when I run it with programs which are designed specifically to make it trigger error messages. I this also broken on Macs?
I tried running the program in Xcode, with all the Diagnostics checkboxes ticked in the Product->Edit Scheme... menu, and with a symbolic breakpoint set in malloc_error_break. The breakpoint doesn't get hit, the code stops with a callstack containing one thing ("dlopen"), and the only thing of note that shows up in the output window is the following:
Warning: Unable to restore previously selected frame.
No memory available to program now: unsafe to call malloc
I'm out of ideas. I'm trying to get Cygwin set up (it's taking hours though) to see if any of the tools will work that way, but if that fails then I'm at a loss. Surely there must be SOME tools which are capable of tracking down the causes of Segmentation faults on a Mac?
For the more modern lldb flavor
$ lldb --file /path/to/program
...
(lldb) r
Process 89510 launched
...
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x726f00)
* frame #0: 0x00007fff73856e52 libsystem_platform.dylib`_platform_strlen + 18
...
Have you compiled with -g and run it inside gdb? Once the app crashes, you can get a backtrace with bt that should show you where the crash occurs
In many cases, macOS stores the recent program crash logs under ~/Library/Logs/DiagnosticReports/ folder.
Usually I will try the following steps when doing troubleshooting on macOS:
Clean the existing crash logs under the ~/Library/Logs/DiagnosticReports/
Run the program again to reproduce the issue
Wait for a few seconds, the crash log will appear under the folder. The crash log is named like {your_program}_{crashing_date}_{id}_{your_host}.crash
Open the crash log with your text editor, search for the keyword Crashed to locate the thread causing the crash. It will show you the stack trace during crash, and in many cases, the exact line of source code causing the crash will be recorded as well.
Some links:
[1] https://mac-optimization.bestreviews.net/analyze-mac-crash-reports/