Is there any way to dump the call stack in a running process in C or C++ every time a certain function is called? What I have in mind is something like this:
void foo()
{
print_stack_trace();
// foo's body
return
}
Where print_stack_trace works similarly to caller in Perl.
Or something like this:
int main (void)
{
// will print out debug info every time foo() is called
register_stack_trace_function(foo);
// etc...
}
where register_stack_trace_function puts some sort of internal breakpoint that will cause a stack trace to be printed whenever foo is called.
Does anything like this exist in some standard C library?
I am working on Linux, using GCC.
Background
I have a test run that behaves differently based on some commandline switches that shouldn't affect this behavior. My code has a pseudo-random number generator that I assume is being called differently based on these switches. I want to be able to run the test with each set of switches and see if the random number generator is called differently for each one.
Survey of C/C++ backtrace methods
In this answer I will try to run a single benchmark for a bunch of solutions to see which one runs faster, while also considering other points such as features and portability.
Tool
Time / call
Line number
Function name
C++ demangling
Recompile
Signal safe
As string
C
C++23 <stacktrace> GCC 12.1
7 us
y
y
y
y
n
y
n
Boost 1.74 stacktrace()
5 us
y
y
y
y
n
y
n
Boost 1.74 stacktrace::safe_dump_to
y
n
n
glibc backtrace_symbols_fd
25 us
n
-rdynamic
hacks
y
y
n
y
glibc backtrace_symbols
21 us
n
-rdynamic
hacks
y
n
y
y
GDB scripting
600 us
y
y
y
n
y
n
y
GDB code injection
n
n
y
libunwind
y
libdwfl
4 ms
n
y
libbacktrace
y
Empty cells mean "TODO", not "no".
us: microsecond
Line number: shows actual line number, not just function name + a memory address.
It is usually possible to recover the line number from an address manually after the fact with addr2line. But it is a pain.
Recompile: requires recompiling the program to get your traces. Not recompiling is better!
Signal safe: crucial for the important uses case of "getting a stack trace in case of segfault": How to automatically generate a stacktrace when my program crashes
As string: you get the stack trace as a string in the program itself, as opposed to e.g. just printing to stdout. Usually implies not signal safe, as we don't know the size of the stack trace string size in advance, and therefore requires malloc which is not async signal safe.
C: does it work on a plain-C project (yes, there are still poor souls out there), or is C++ required?
Test setup
All benchmarks will run the following
main.cpp
#include <cstdlib> // strtoul
#include <mystacktrace.h>
void my_func_2(void) {
print_stacktrace(); // line 6
}
void my_func_1(double f) {
(void)f;
my_func_2();
}
void my_func_1(int i) {
(void)i;
my_func_2(); // line 16
}
int main(int argc, char **argv) {
long long unsigned int n;
if (argc > 1) {
n = std::strtoul(argv[1], NULL, 0);
} else {
n = 1;
}
for (long long unsigned int i = 0; i < n; ++i) {
my_func_1(1); // line 27
}
}
This input is designed to test C++ name demangling since my_func_1(int) and my_func_1(float) are necessarily mangled as a way to implement C++ function overload.
We differentiate between the benchmarks by using different -I includes to point to different implementations of print_stacktrace().
Each benchmark is done with a command of form:
time ./stacktrace.out 100000 &>/dev/null
The number of iterations is adjusted for each implementation to produce a total runtime of the order of 1s for that benchmark.
-O0 is used on all tests below unless noted. Stack traces may be irreparably mutilated by certain optimizations. Tail call optimization is a notable example of that: What is tail call optimization? There's nothing we can do about it.
C++23 <stacktrace>
This method was previously mentioned at: https://stackoverflow.com/a/69384663/895245 please consider upvoting that answer.
This is the best solution... it's portable, fast, shows line numbers and demangles C++ symbols. This option will displace every other alternative as soon as it becomes more widely available, with the exception perhaps only of GDB for one-offs without the need or recompilation.
cpp20_stacktrace/mystacktrace.h
#include <iostream>
#include <stacktrace>
void print_stacktrace() {
std::cout << std::stacktrace::current();
}
GCC 12.1.0 from Ubuntu 22.04 does not have support compiled in, so for now I built it from source as per: How to edit and re-build the GCC libstdc++ C++ standard library source? and set --enable-libstdcxx-backtrace=yes, and it worked!
Compile with:
g++ -O0 -ggdb3 -Wall -Wextra -pedantic -std=c++23 -o cpp20_stacktrace.out main.cpp -lstdc++_libbacktrace
Sample output:
0# print_stacktrace() at cpp20_stacktrace/mystacktrace.h:5
1# my_func_2() at /home/ciro/main.cpp:6
2# my_func_1(int) at /home/ciro/main.cpp:16
3# at /home/ciro/main.cpp:27
4# at :0
5# at :0
6# at :0
7#
If we try to use GCC 12.1.0 from Ubuntu 22.04:
sudo apt install g++-12
g++-12 -ggdb3 -O2 -std=c++23 -Wall -Wextra -pedantic -o stacktrace.out stacktrace.cpp -lstdc++_libbacktrace
It fails with:
stacktrace.cpp: In function ‘void my_func_2()’:
stacktrace.cpp:6:23: error: ‘std::stacktrace’ has not been declared
6 | std::cout << std::stacktrace::current();
| ^~~~~~~~~~
Checking build options with:
g++-12 -v
does not show:
--enable-libstdcxx-backtrace=yes
so it wasn't compiled in. Bibliography:
How to use <stacktrace> in GCC trunk?
How can I generate a C++23 stacktrace with GCC 12.1?
It does not fail on the include because the header file:
/usr/include/c++/12
has a feature check:
#if __cplusplus > 202002L && _GLIBCXX_HAVE_STACKTRACE
Boost stacktrace
The library has changed quite a lot around Ubuntu 22.04, so make sure your version matches: Boost stack-trace not showing function names and line numbers
The library is pretty much superseded by the more portable C++23 implementation, but remains a very good option for those that are not at that standard version yet, but already have a "Boost clearance".
Documented at: https://www.boost.org/doc/libs/1_66_0/doc/html/stacktrace/getting_started.html#stacktrace.getting_started.how_to_print_current_call_stack
Tested on Ubuntu 22.04, boost 1.74.0, you should do:
boost_stacktrace/mystacktrace.h
#include <iostream>
#define BOOST_STACKTRACE_LINK
#include <boost/stacktrace.hpp>
void print_stacktrace(void) {
std::cout << boost::stacktrace::stacktrace();
}
On Ubuntu 19.10 boost 1.67.0 to get the line numbers we had to instead:
#include <iostream>
#define BOOST_STACKTRACE_USE_ADDR2LINE
#include <boost/stacktrace.hpp>
void print_stacktrace(void) {
std::cout << boost::stacktrace::stacktrace();
}
which would call out to the addr2line executable and be 1000x slower than the newer Boost version.
The package libboost-stacktrace-dev did not exist at all on Ubuntu 16.04.
The rest of this section considers only the Ubuntu 22.04, boost 1.74 behaviour.
Compile:
sudo apt-get install libboost-stacktrace-dev
g++ -O0 -ggdb3 -Wall -Wextra -pedantic -std=c++11 -o boost_stacktrace.out main.cpp -lboost_stacktrace_backtrace
Sample output:
0# print_stacktrace() at boost_stacktrace/mystacktrace.h:7
1# my_func_2() at /home/ciro/main.cpp:7
2# my_func_1(int) at /home/ciro/main.cpp:17
3# main at /home/ciro/main.cpp:26
4# __libc_start_call_main at ../sysdeps/nptl/libc_start_call_main.h:58
5# __libc_start_main at ../csu/libc-start.c:379
6# _start in ./boost_stacktrace.out
Note that the lines are off by one line. It was suggested in the comments that this is because the following instruction address is being considered.
Boost stacktrace header only
What the BOOST_STACKTRACE_LINK does is to require -lboost_stacktrace_backtrace at link time, so we imagine without that it will just work. This would be a good option for devs who don't have the "Boost clearance" to just add as one offs to debug.
TODO unfortunately it didn't so well for me:
#include <iostream>
#include <boost/stacktrace.hpp>
void print_stacktrace(void) {
std::cout << boost::stacktrace::stacktrace();
}
then:
g++ -O0 -ggdb3 -Wall -Wextra -pedantic -std=c++11 -o boost_stacktrace_header_only.out main.cpp
contains the overly short output:
0# 0x000055FF74AFB601 in ./boost_stacktrace_header_only.out
1# 0x000055FF74AFB66C in ./boost_stacktrace_header_only.out
2# 0x000055FF74AFB69C in ./boost_stacktrace_header_only.out
3# 0x000055FF74AFB6F7 in ./boost_stacktrace_header_only.out
4# 0x00007F0176E7BD90 in /lib/x86_64-linux-gnu/libc.so.6
5# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
6# 0x000055FF74AFB4E5 in ./boost_stacktrace_header_only.out
which we can't even use with addr2line. Maybe we have to pass some other define from: https://www.boost.org/doc/libs/1_80_0/doc/html/stacktrace/configuration_and_build.html ?
Tested on Ubuntu 22.04. boost 1.74.
Boost boost::stacktrace::safe_dump_to
This is an interesting alternative to boost::stacktrace::stacktrace as it writes the stack trace in a async signal safe manner to a file, which makes it a good option for automatically dumping stack traces on segfaults which is a super common use case: How to automatically generate a stacktrace when my program crashes
Documented at: https://www.boost.org/doc/libs/1_70_0/doc/html/boost/stacktrace/safe_dump_1_3_38_7_6_2_1_6.html
TODO get it to work. All I see each time is a bunch of random bytes. My attempt:
boost_stacktrace_safe/mystacktrace.h
#include <unistd.h>
#define BOOST_STACKTRACE_LINK
#include <boost/stacktrace.hpp>
void print_stacktrace(void) {
boost::stacktrace::safe_dump_to(0, 1024, STDOUT_FILENO);
}
Sample output:
1[FU1[FU"2[FU}2[FUm1#n10[FU
Changes drastically each time, suggesting it is random memory addresses.
Tested on Ubuntu 22.04, boost 1.74.0.
glibc backtrace
This method is quite portable as it comes with glibc itself. Documented at: https://www.gnu.org/software/libc/manual/html_node/Backtraces.html
Tested on Ubuntu 22.04, glibc 2.35.
glibc_backtrace_symbols_fd/mystacktrace.h
#include <execinfo.h> /* backtrace, backtrace_symbols_fd */
#include <unistd.h> /* STDOUT_FILENO */
void print_stacktrace(void) {
size_t size;
enum Constexpr { MAX_SIZE = 1024 };
void *array[MAX_SIZE];
size = backtrace(array, MAX_SIZE);
backtrace_symbols_fd(array, size, STDOUT_FILENO);
}
Compile with:
g++ -O0 -ggdb3 -Wall -Wextra -pedantic -rdynamic -std=c++11 -o glibc_backtrace_symbols_fd.out main.cpp
Sample output with -rdynamic:
./glibc_backtrace_symbols.out(_Z16print_stacktracev+0x47) [0x556e6a131230]
./glibc_backtrace_symbols.out(_Z9my_func_2v+0xd) [0x556e6a1312d6]
./glibc_backtrace_symbols.out(_Z9my_func_1i+0x14) [0x556e6a131306]
./glibc_backtrace_symbols.out(main+0x58) [0x556e6a131361]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f175e7bdd90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f175e7bde40]
./glibc_backtrace_symbols.out(_start+0x25) [0x556e6a131125]
Sample output without -rdynamic:
./glibc_backtrace_symbols_fd_no_rdynamic.out(+0x11f0)[0x556bd40461f0]
./glibc_backtrace_symbols_fd_no_rdynamic.out(+0x123c)[0x556bd404623c]
./glibc_backtrace_symbols_fd_no_rdynamic.out(+0x126c)[0x556bd404626c]
./glibc_backtrace_symbols_fd_no_rdynamic.out(+0x12c7)[0x556bd40462c7]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f0da2b70d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f0da2b70e40]
./glibc_backtrace_symbols_fd_no_rdynamic.out(+0x10e5)[0x556bd40460e5]
To get the line numbers without -rdynamic we can use addr2line:
addr2line -C -e glibc_backtrace_symbols_fd_no_rdynamic.out 0x11f0 0x123c 0x126c 0x12c7
addr2line cannot unfortunately handle the function name + offset in function format of when we are not using -rdynamic, e.g. _Z9my_func_2v+0xd.
GDB can however:
gdb -nh -batch -ex 'info line *(_Z9my_func_2v+0xd)' -ex 'info line *(_Z9my_func_1i+0x14)' glibc_backtrace_symbols.out
Line 7 of "main.cpp" starts at address 0x12d6 <_Z9my_func_2v+13> and ends at 0x12d9 <_Z9my_func_1d>.
Line 17 of "main.cpp" starts at address 0x1306 <_Z9my_func_1i+20> and ends at 0x1309 <main(int, char**)>.
A helper to make it more bearable:
addr2lines() (
perl -ne '$m = s/(.*).*\(([^)]*)\).*/gdb -nh -q -batch -ex "info line *\2" \1/;print $_ if $m' | bash
)
Usage:
xsel -b | addr2lines
glibc backtrace_symbols
A version of backtrace_symbols_fd that returns a string rather than printing to a file handle.
glibc_backtrace_symbols/mystacktrace.h
#include <execinfo.h> /* backtrace, backtrace_symbols */
#include <stdio.h> /* printf */
void print_stacktrace(void) {
char **strings;
size_t i, size;
enum Constexpr { MAX_SIZE = 1024 };
void *array[MAX_SIZE];
size = backtrace(array, MAX_SIZE);
strings = backtrace_symbols(array, size);
for (i = 0; i < size; i++)
printf("%s\n", strings[i]);
free(strings);
}
glibc backtrace with C++ demangling hack 1: -export-dynamic + dladdr
I couldn't find a simple way to automatically demangle C++ symbols with glibc backtrace.
https://panthema.net/2008/0901-stacktrace-demangled/
https://gist.github.com/fmela/591333/c64f4eb86037bb237862a8283df70cdfc25f01d3
Adapted from: https://gist.github.com/fmela/591333/c64f4eb86037bb237862a8283df70cdfc25f01d3
This is a "hack" because it requires changing the ELF with -export-dynamic.
glibc_ldl.cpp
#include <dlfcn.h> // for dladdr
#include <cxxabi.h> // for __cxa_demangle
#include <cstdio>
#include <string>
#include <sstream>
#include <iostream>
// This function produces a stack backtrace with demangled function & method names.
std::string backtrace(int skip = 1)
{
void *callstack[128];
const int nMaxFrames = sizeof(callstack) / sizeof(callstack[0]);
char buf[1024];
int nFrames = backtrace(callstack, nMaxFrames);
char **symbols = backtrace_symbols(callstack, nFrames);
std::ostringstream trace_buf;
for (int i = skip; i < nFrames; i++) {
Dl_info info;
if (dladdr(callstack[i], &info)) {
char *demangled = NULL;
int status;
demangled = abi::__cxa_demangle(info.dli_sname, NULL, 0, &status);
std::snprintf(
buf,
sizeof(buf),
"%-3d %*p %s + %zd\n",
i,
(int)(2 + sizeof(void*) * 2),
callstack[i],
status == 0 ? demangled : info.dli_sname,
(char *)callstack[i] - (char *)info.dli_saddr
);
free(demangled);
} else {
std::snprintf(buf, sizeof(buf), "%-3d %*p\n",
i, (int)(2 + sizeof(void*) * 2), callstack[i]);
}
trace_buf << buf;
std::snprintf(buf, sizeof(buf), "%s\n", symbols[i]);
trace_buf << buf;
}
free(symbols);
if (nFrames == nMaxFrames)
trace_buf << "[truncated]\n";
return trace_buf.str();
}
void my_func_2(void) {
std::cout << backtrace() << std::endl;
}
void my_func_1(double f) {
(void)f;
my_func_2();
}
void my_func_1(int i) {
(void)i;
my_func_2();
}
int main() {
my_func_1(1);
my_func_1(2.0);
}
Compile and run:
g++ -fno-pie -ggdb3 -O0 -no-pie -o glibc_ldl.out -std=c++11 -Wall -Wextra \
-pedantic-errors -fpic glibc_ldl.cpp -export-dynamic -ldl
./glibc_ldl.out
output:
1 0x40130a my_func_2() + 41
./glibc_ldl.out(_Z9my_func_2v+0x29) [0x40130a]
2 0x40139e my_func_1(int) + 16
./glibc_ldl.out(_Z9my_func_1i+0x10) [0x40139e]
3 0x4013b3 main + 18
./glibc_ldl.out(main+0x12) [0x4013b3]
4 0x7f7594552b97 __libc_start_main + 231
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f7594552b97]
5 0x400f3a _start + 42
./glibc_ldl.out(_start+0x2a) [0x400f3a]
1 0x40130a my_func_2() + 41
./glibc_ldl.out(_Z9my_func_2v+0x29) [0x40130a]
2 0x40138b my_func_1(double) + 18
./glibc_ldl.out(_Z9my_func_1d+0x12) [0x40138b]
3 0x4013c8 main + 39
./glibc_ldl.out(main+0x27) [0x4013c8]
4 0x7f7594552b97 __libc_start_main + 231
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f7594552b97]
5 0x400f3a _start + 42
./glibc_ldl.out(_start+0x2a) [0x400f3a]
Tested on Ubuntu 18.04.
glibc backtrace with C++ demangling hack 2: parse backtrace output
Shown at: https://panthema.net/2008/0901-stacktrace-demangled/
This is a hack because it requires parsing.
TODO get it to compile and show it here.
GDB scripting
We can also do this with GDB without recompiling by using: How to do an specific action when a certain breakpoint is hit in GDB?
We setup an empty backtrace function for our testing:
gdb/mystacktrace.h
void print_stacktrace(void) {}
and then with:
main.gdb
start
break print_stacktrace
commands
silent
backtrace
printf "\n"
continue
end
continue
we can run:
gdb -nh -batch -x main.gdb --args gdb.out
Sample output:
Temporary breakpoint 1 at 0x11a7: file main.cpp, line 21.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Temporary breakpoint 1, main (argc=1, argv=0x7fffffffc3e8) at main.cpp:21
warning: Source file is more recent than executable.
21 if (argc > 1) {
Breakpoint 2 at 0x555555555151: file gdb/mystacktrace.h, line 1.
#0 print_stacktrace () at gdb/mystacktrace.h:1
#1 0x0000555555555161 in my_func_2 () at main.cpp:6
#2 0x0000555555555191 in my_func_1 (i=1) at main.cpp:16
#3 0x00005555555551ec in main (argc=1, argv=0x7fffffffc3e8) at main.cpp:27
[Inferior 1 (process 165453) exited normally]
The above can be made more usable with the following Bash function:
gdbbt() (
tmpfile=$(mktemp /tmp/gdbbt.XXXXXX)
fn="$1"
shift
printf '%s' "
start
break $fn
commands
silent
backtrace
printf \"\n\"
continue
end
continue
" > "$tmpfile"
gdb -nh -batch -x "$tmpfile" -args "$#"
rm -f "$tmpfile"
)
Usage:
gdbbt print_stacktrace gdb.out 2
I don't know how to make commands with -ex without the temporary file: Problems adding a breakpoint with commands from command line with ex command
Tested in Ubuntu 22.04, GDB 12.0.90.
GDB code injection
TODO this is the dream! It might allow for both compiled-liked speeds, but without the need to recompile! Either:
with compile code + one of the other options, ideally C++23 <stacktrace>: How to call assembly in gdb? Might already be possible. But compile code is mega-quirky so I'm lazy to even try
a built-in dbt command analogous to dprintf dynamic printf: How to do an specific action when a certain breakpoint is hit in GDB?
libunwind
TODO does this have any advantage over glibc backtrace? Very similar output, also requires modifying the build command, but not part of glibc so requires an extra package installation.
Code adapted from: https://eli.thegreenplace.net/2015/programmatic-access-to-the-call-stack-in-c/
main.c
/* This must be on top. */
#define _XOPEN_SOURCE 700
#include <stdio.h>
#include <stdlib.h>
/* Paste this on the file you want to debug. */
#define UNW_LOCAL_ONLY
#include <libunwind.h>
#include <stdio.h>
void print_trace() {
char sym[256];
unw_context_t context;
unw_cursor_t cursor;
unw_getcontext(&context);
unw_init_local(&cursor, &context);
while (unw_step(&cursor) > 0) {
unw_word_t offset, pc;
unw_get_reg(&cursor, UNW_REG_IP, &pc);
if (pc == 0) {
break;
}
printf("0x%lx:", pc);
if (unw_get_proc_name(&cursor, sym, sizeof(sym), &offset) == 0) {
printf(" (%s+0x%lx)\n", sym, offset);
} else {
printf(" -- error: unable to obtain symbol name for this frame\n");
}
}
puts("");
}
void my_func_3(void) {
print_trace();
}
void my_func_2(void) {
my_func_3();
}
void my_func_1(void) {
my_func_3();
}
int main(void) {
my_func_1(); /* line 46 */
my_func_2(); /* line 47 */
return 0;
}
Compile and run:
sudo apt-get install libunwind-dev
gcc -fno-pie -ggdb3 -O3 -no-pie -o main.out -std=c99 \
-Wall -Wextra -pedantic-errors main.c -lunwind
Either #define _XOPEN_SOURCE 700 must be on top, or we must use -std=gnu99:
Is the type `stack_t` no longer defined on linux?
Glibc - error in ucontext.h, but only with -std=c11
Run:
./main.out
Output:
0x4007db: (main+0xb)
0x7f4ff50aa830: (__libc_start_main+0xf0)
0x400819: (_start+0x29)
0x4007e2: (main+0x12)
0x7f4ff50aa830: (__libc_start_main+0xf0)
0x400819: (_start+0x29)
and:
addr2line -e main.out 0x4007db 0x4007e2
gives:
/home/ciro/main.c:34
/home/ciro/main.c:49
With -O0:
0x4009cf: (my_func_3+0xe)
0x4009e7: (my_func_1+0x9)
0x4009f3: (main+0x9)
0x7f7b84ad7830: (__libc_start_main+0xf0)
0x4007d9: (_start+0x29)
0x4009cf: (my_func_3+0xe)
0x4009db: (my_func_2+0x9)
0x4009f8: (main+0xe)
0x7f7b84ad7830: (__libc_start_main+0xf0)
0x4007d9: (_start+0x29)
and:
addr2line -e main.out 0x4009f3 0x4009f8
gives:
/home/ciro/main.c:47
/home/ciro/main.c:48
Tested on Ubuntu 16.04, GCC 6.4.0, libunwind 1.1.
libunwind with C++ name demangling
Code adapted from: https://eli.thegreenplace.net/2015/programmatic-access-to-the-call-stack-in-c/
unwind.cpp
#define UNW_LOCAL_ONLY
#include <cxxabi.h>
#include <libunwind.h>
#include <cstdio>
#include <cstdlib>
#include <iostream>
void backtrace() {
unw_cursor_t cursor;
unw_context_t context;
// Initialize cursor to current frame for local unwinding.
unw_getcontext(&context);
unw_init_local(&cursor, &context);
// Unwind frames one by one, going up the frame stack.
while (unw_step(&cursor) > 0) {
unw_word_t offset, pc;
unw_get_reg(&cursor, UNW_REG_IP, &pc);
if (pc == 0) {
break;
}
std::printf("0x%lx:", pc);
char sym[256];
if (unw_get_proc_name(&cursor, sym, sizeof(sym), &offset) == 0) {
char* nameptr = sym;
int status;
char* demangled = abi::__cxa_demangle(sym, nullptr, nullptr, &status);
if (status == 0) {
nameptr = demangled;
}
std::printf(" (%s+0x%lx)\n", nameptr, offset);
std::free(demangled);
} else {
std::printf(" -- error: unable to obtain symbol name for this frame\n");
}
}
}
void my_func_2(void) {
backtrace();
std::cout << std::endl; // line 43
}
void my_func_1(double f) {
(void)f;
my_func_2();
}
void my_func_1(int i) {
(void)i;
my_func_2();
} // line 54
int main() {
my_func_1(1);
my_func_1(2.0);
}
Compile and run:
sudo apt-get install libunwind-dev
g++ -fno-pie -ggdb3 -O0 -no-pie -o unwind.out -std=c++11 \
-Wall -Wextra -pedantic-errors unwind.cpp -lunwind -pthread
./unwind.out
Output:
0x400c80: (my_func_2()+0x9)
0x400cb7: (my_func_1(int)+0x10)
0x400ccc: (main+0x12)
0x7f4c68926b97: (__libc_start_main+0xe7)
0x400a3a: (_start+0x2a)
0x400c80: (my_func_2()+0x9)
0x400ca4: (my_func_1(double)+0x12)
0x400ce1: (main+0x27)
0x7f4c68926b97: (__libc_start_main+0xe7)
0x400a3a: (_start+0x2a)
and then we can find the lines of my_func_2 and my_func_1(int) with:
addr2line -e unwind.out 0x400c80 0x400cb7
which gives:
/home/ciro/test/unwind.cpp:43
/home/ciro/test/unwind.cpp:54
TODO: why are the lines off by one?
Tested on Ubuntu 18.04, GCC 7.4.0, libunwind 1.2.1.
Linux kernel
How to print the current thread stack trace inside the Linux kernel?
libdwfl
This was originally mentioned at: https://stackoverflow.com/a/60713161/895245 and it might be the best method, but I have to benchmark a bit more, but please go upvote that answer.
TODO: I tried to minimize the code in that answer, which was working, to a single function, but it is segfaulting, let me know if anyone can find why.
dwfl.cpp: answer reached 30k chars and this was the easiest cut: https://gist.github.com/cirosantilli/f1dd3ee5d324b9d24e40f855723544ac
Compile and run:
sudo apt install libdw-dev libunwind-dev
g++ -fno-pie -ggdb3 -O0 -no-pie -o dwfl.out -std=c++11 -Wall -Wextra -pedantic-errors dwfl.cpp -ldw -lunwind
./dwfl.out
We also need libunwind as that makes results more correct. If you do without it, it runs, but you will see that some of the lines are a bit wrong.
Output:
0: 0x402b72 stacktrace[abi:cxx11]() at /home/ciro/test/dwfl.cpp:65
1: 0x402cda my_func_2() at /home/ciro/test/dwfl.cpp:100
2: 0x402d76 my_func_1(int) at /home/ciro/test/dwfl.cpp:111
3: 0x402dd1 main at /home/ciro/test/dwfl.cpp:122
4: 0x7ff227ea0d8f __libc_start_call_main at ../sysdeps/nptl/libc_start_call_main.h:58
5: 0x7ff227ea0e3f __libc_start_main##GLIBC_2.34 at ../csu/libc-start.c:392
6: 0x402534 _start at ../csu/libc-start.c:-1
0: 0x402b72 stacktrace[abi:cxx11]() at /home/ciro/test/dwfl.cpp:65
1: 0x402cda my_func_2() at /home/ciro/test/dwfl.cpp:100
2: 0x402d5f my_func_1(double) at /home/ciro/test/dwfl.cpp:106
3: 0x402de2 main at /home/ciro/test/dwfl.cpp:123
4: 0x7ff227ea0d8f __libc_start_call_main at ../sysdeps/nptl/libc_start_call_main.h:58
5: 0x7ff227ea0e3f __libc_start_main##GLIBC_2.34 at ../csu/libc-start.c:392
6: 0x402534 _start at ../csu/libc-start.c:-1
Benchmark run:
g++ -fno-pie -ggdb3 -O3 -no-pie -o dwfl.out -std=c++11 -Wall -Wextra -pedantic-errors dwfl.cpp -ldw
time ./dwfl.out 1000 >/dev/null
Output:
real 0m3.751s
user 0m2.822s
sys 0m0.928s
So we see that this method is 10x faster than Boost's stacktrace, and might therefore be applicable to more use cases.
Tested in Ubuntu 22.04 amd64, libdw-dev 0.186, libunwind 1.3.2.
libbacktrace
https://github.com/ianlancetaylor/libbacktrace
Considering the harcore library author, it is worth trying this out, maybe it is The One. TODO check it out.
A C library that may be linked into a C/C++ program to produce symbolic backtraces
As of October 2020, libbacktrace supports ELF, PE/COFF, Mach-O, and XCOFF executables with DWARF debugging information. In other words, it supports GNU/Linux, *BSD, macOS, Windows, and AIX. The library is written to make it straightforward to add support for other object file and debugging formats.
The library relies on the C++ unwind API defined at https://itanium-cxx-abi.github.io/cxx-abi/abi-eh.html This API is provided by GCC and clang.
See also
How can one grab a stack trace in C?
How to make backtrace()/backtrace_symbols() print the function names?
Is there a portable/standard-compliant way to get filenames and linenumbers in a stack trace?
Best way to invoke gdb from inside program to print its stacktrace?
automatic stack trace on failure:
on C++ exception: C++ display stack trace on exception
generic: How to automatically generate a stacktrace when my program crashes
For a linux-only solution you can use backtrace(3) that simply returns an array of void * (in fact each of these point to the return address from the corresponding stack frame). To translate these to something of use, there's backtrace_symbols(3).
Pay attention to the notes section in backtrace(3):
The symbol names may be unavailable
without the use of special linker
options.
For systems using the GNU linker, it is necessary to use the
-rdynamic linker
option. Note that names of "static" functions are not exposed,
and won't be
available in the backtrace.
In C++23, there will be <stacktrace>, and then you can do:
#include <stacktrace>
/* ... */
std::cout << std::stacktrace::current();
Further details:
• https://en.cppreference.com/w/cpp/header/stacktrace
• https://en.cppreference.com/w/cpp/utility/basic_stacktrace/operator_ltlt
Is there any way to dump the call stack in a running process in C or C++ every time a certain function is called?
You can use a macro function instead of return statement in the specific function.
For example, instead of using return,
int foo(...)
{
if (error happened)
return -1;
... do something ...
return 0
}
You can use a macro function.
#include "c-callstack.h"
int foo(...)
{
if (error happened)
NL_RETURN(-1);
... do something ...
NL_RETURN(0);
}
Whenever an error happens in a function, you will see Java-style call stack as shown below.
Error(code:-1) at : so_topless_ranking_server (sample.c:23)
Error(code:-1) at : nanolat_database (sample.c:31)
Error(code:-1) at : nanolat_message_queue (sample.c:39)
Error(code:-1) at : main (sample.c:47)
Full source code is available here.
c-callstack at https://github.com/Nanolat
Linux specific, TLDR:
backtrace in glibc produces accurate stacktraces only when -lunwind is linked (undocumented platform-specific feature).
To output function name, source file and line number use #include <elfutils/libdwfl.h> (this library is documented only in its header file). backtrace_symbols and backtrace_symbolsd_fd are least informative.
On modern Linux your can get the stacktrace addresses using function backtrace. The undocumented way to make backtrace produce more accurate addresses on popular platforms is to link with -lunwind (libunwind-dev on Ubuntu 18.04) (see the example output below). backtrace uses function _Unwind_Backtrace and by default the latter comes from libgcc_s.so.1 and that implementation is most portable. When -lunwind is linked it provides a more accurate version of _Unwind_Backtrace but this library is less portable (see supported architectures in libunwind/src).
Unfortunately, the companion backtrace_symbolsd and backtrace_symbols_fd functions have not been able to resolve the stacktrace addresses to function names with source file name and line number for probably a decade now (see the example output below).
However, there is another method to resolve addresses to symbols and it produces the most useful traces with function name, source file and line number. The method is to #include <elfutils/libdwfl.h>and link with -ldw (libdw-dev on Ubuntu 18.04).
Working C++ example (test.cc):
#include <stdexcept>
#include <iostream>
#include <cassert>
#include <cstdlib>
#include <string>
#include <boost/core/demangle.hpp>
#include <execinfo.h>
#include <elfutils/libdwfl.h>
struct DebugInfoSession {
Dwfl_Callbacks callbacks = {};
char* debuginfo_path = nullptr;
Dwfl* dwfl = nullptr;
DebugInfoSession() {
callbacks.find_elf = dwfl_linux_proc_find_elf;
callbacks.find_debuginfo = dwfl_standard_find_debuginfo;
callbacks.debuginfo_path = &debuginfo_path;
dwfl = dwfl_begin(&callbacks);
assert(dwfl);
int r;
r = dwfl_linux_proc_report(dwfl, getpid());
assert(!r);
r = dwfl_report_end(dwfl, nullptr, nullptr);
assert(!r);
static_cast<void>(r);
}
~DebugInfoSession() {
dwfl_end(dwfl);
}
DebugInfoSession(DebugInfoSession const&) = delete;
DebugInfoSession& operator=(DebugInfoSession const&) = delete;
};
struct DebugInfo {
void* ip;
std::string function;
char const* file;
int line;
DebugInfo(DebugInfoSession const& dis, void* ip)
: ip(ip)
, file()
, line(-1)
{
// Get function name.
uintptr_t ip2 = reinterpret_cast<uintptr_t>(ip);
Dwfl_Module* module = dwfl_addrmodule(dis.dwfl, ip2);
char const* name = dwfl_module_addrname(module, ip2);
function = name ? boost::core::demangle(name) : "<unknown>";
// Get source filename and line number.
if(Dwfl_Line* dwfl_line = dwfl_module_getsrc(module, ip2)) {
Dwarf_Addr addr;
file = dwfl_lineinfo(dwfl_line, &addr, &line, nullptr, nullptr, nullptr);
}
}
};
std::ostream& operator<<(std::ostream& s, DebugInfo const& di) {
s << di.ip << ' ' << di.function;
if(di.file)
s << " at " << di.file << ':' << di.line;
return s;
}
void terminate_with_stacktrace() {
void* stack[512];
int stack_size = ::backtrace(stack, sizeof stack / sizeof *stack);
// Print the exception info, if any.
if(auto ex = std::current_exception()) {
try {
std::rethrow_exception(ex);
}
catch(std::exception& e) {
std::cerr << "Fatal exception " << boost::core::demangle(typeid(e).name()) << ": " << e.what() << ".\n";
}
catch(...) {
std::cerr << "Fatal unknown exception.\n";
}
}
DebugInfoSession dis;
std::cerr << "Stacktrace of " << stack_size << " frames:\n";
for(int i = 0; i < stack_size; ++i) {
std::cerr << i << ": " << DebugInfo(dis, stack[i]) << '\n';
}
std::cerr.flush();
std::_Exit(EXIT_FAILURE);
}
int main() {
std::set_terminate(terminate_with_stacktrace);
throw std::runtime_error("test exception");
}
Compiled on Ubuntu 18.04.4 LTS with gcc-8.3:
g++ -o test.o -c -m{arch,tune}=native -std=gnu++17 -W{all,extra,error} -g -Og -fstack-protector-all test.cc
g++ -o test -g test.o -ldw -lunwind
Outputs:
Fatal exception std::runtime_error: test exception.
Stacktrace of 7 frames:
0: 0x55f3837c1a8c terminate_with_stacktrace() at /home/max/src/test/test.cc:76
1: 0x7fbc1c845ae5 <unknown>
2: 0x7fbc1c845b20 std::terminate()
3: 0x7fbc1c845d53 __cxa_throw
4: 0x55f3837c1a43 main at /home/max/src/test/test.cc:103
5: 0x7fbc1c3e3b96 __libc_start_main at ../csu/libc-start.c:310
6: 0x55f3837c17e9 _start
When no -lunwind is linked, it produces a less accurate stacktrace:
0: 0x5591dd9d1a4d terminate_with_stacktrace() at /home/max/src/test/test.cc:76
1: 0x7f3c18ad6ae6 <unknown>
2: 0x7f3c18ad6b21 <unknown>
3: 0x7f3c18ad6d54 <unknown>
4: 0x5591dd9d1a04 main at /home/max/src/test/test.cc:103
5: 0x7f3c1845cb97 __libc_start_main at ../csu/libc-start.c:344
6: 0x5591dd9d17aa _start
For comparison, backtrace_symbols_fd output for the same stacktrace is least informative:
/home/max/src/test/debug/gcc/test(+0x192f)[0x5601c5a2092f]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92ae5)[0x7f95184f5ae5]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZSt9terminatev+0x10)[0x7f95184f5b20]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(__cxa_throw+0x43)[0x7f95184f5d53]
/home/max/src/test/debug/gcc/test(+0x1ae7)[0x5601c5a20ae7]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe6)[0x7f9518093b96]
/home/max/src/test/debug/gcc/test(+0x1849)[0x5601c5a20849]
In a production version (as well as C language version) you may like to make this code extra robust by replacing boost::core::demangle, std::string and std::cout with their underlying calls.
You can also override __cxa_throw to capture the stacktrace when an exception is thrown and print it when the exception is caught. By the time it enters catch block the stack has been unwound, so it is too late to call backtrace, and this is why the stack must be captured on throw which is implemented by function __cxa_throw. Note that in a multi-threaded program __cxa_throw can be called simultaneously by multiple threads, so that if it captures the stacktrace into a global array that must be thread_local.
You can also make the stack trace printing function async-signal safe, so that you can invoke it directly from your SIGSEGV, SIGBUS signal handlers (which should use their own stacks for robustness). Obtaining function name, source file and line number using libdwfl from a signal handler may fail because it is not async-signal safe or if the address space of the process has been substantially corrupted, but in practice it succeeds 99% of the time (I haven't seen it fail).
To summarize, a complete production-ready library for automatic stacktrace output should:
Capture the stacktrace on throw into thread-specific storage.
Automatically print the stacktrace on unhandled exceptions.
Print the stacktrace in async-signal-safe manner.
Provide a robust signal handler function which uses its own stack that prints the stacktrace in a async-signal-safe manner. The user can install this function as a signal handler for SIGSEGV, SIGBUS, SIGFPE, etc..
The signal handler may as well print the values of all CPU registers at the point of the fault from ucontext_t signal function argument (may be excluding vector registers), a-la Linux kernel oops log messages.
Another answer to an old thread.
When I need to do this, I usually just use system() and pstack
So something like this:
#include <sys/types.h>
#include <unistd.h>
#include <string>
#include <sstream>
#include <cstdlib>
void f()
{
pid_t myPid = getpid();
std::string pstackCommand = "pstack ";
std::stringstream ss;
ss << myPid;
pstackCommand += ss.str();
system(pstackCommand.c_str());
}
void g()
{
f();
}
void h()
{
g();
}
int main()
{
h();
}
This outputs
#0 0x00002aaaab62d61e in waitpid () from /lib64/libc.so.6
#1 0x00002aaaab5bf609 in do_system () from /lib64/libc.so.6
#2 0x0000000000400c3c in f() ()
#3 0x0000000000400cc5 in g() ()
#4 0x0000000000400cd1 in h() ()
#5 0x0000000000400cdd in main ()
This should work on Linux, FreeBSD and Solaris. I don't think that macOS has pstack or a simple equivalent, but this thread seems to have an alternative.
If you are using C, then you will need to use C string functions.
#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
void f()
{
pid_t myPid = getpid();
/*
length of command 7 for 'pstack ', 7 for the PID, 1 for nul
*/
char pstackCommand[7+7+1];
sprintf(pstackCommand, "pstack %d", (int)myPid);
system(pstackCommand);
}
I've used 7 for the max number of digits in the PID, based on this post.
There is no standardized way to do that. For windows the functionality is provided in the DbgHelp library
You can use the Boost libraries to print the current callstack.
#include <boost/stacktrace.hpp>
// ... somewhere inside the `bar(int)` function that is called recursively:
std::cout << boost::stacktrace::stacktrace();
Man here: https://www.boost.org/doc/libs/1_65_1/doc/html/stacktrace.html
I know this thread is old, but I think it can be useful for other people. If you are using gcc, you can use its instrument features (-finstrument-functions option) to log any function call (entry and exit). Have a look at this for more information: http://hacktalks.blogspot.fr/2013/08/gcc-instrument-functions.html
You can thus for instance push and pop every calls into a stack, and when you want to print it, you just look at what you have in your stack.
I've tested it, it works perfectly and is very handy
UPDATE: you can also find information about the -finstrument-functions compile option in the GCC doc concerning the Instrumentation options: https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html
You can implement the functionality yourself:
Use a global (string)stack and at start of each function push the function name and such other values (eg parameters) onto this stack; at exit of function pop it again.
Write a function that will printout the stack content when it is called, and use this in the function where you want to see the callstack.
This may sound like a lot of work but is quite useful.
Of course the next question is: will this be enough ?
The main disadvantage of stack-traces is that why you have the precise function being called you do not have anything else, like the value of its arguments, which is very useful for debugging.
If you have access to gcc and gdb, I would suggest using assert to check for a specific condition, and produce a memory dump if it is not met. Of course this means the process will stop, but you'll have a full fledged report instead of a mere stack-trace.
If you wish for a less obtrusive way, you can always use logging. There are very efficient logging facilities out there, like Pantheios for example. Which once again could give you a much more accurate image of what is going on.
You can use Poppy for this. It is normally used to gather the stack trace during a crash but it can also output it for a running program as well.
Now here's the good part: it can output the actual parameter values for each function on the stack, and even local variables, loop counters, etc.
You can use the GNU profiler. It shows the call-graph as well! the command is gprof and you need to compile your code with some option.
Is there any way to dump the call stack in a running process in C or C++ every time a certain function is called?
No there is not, although platform-dependent solutions might exist.
Introduction
I have been working on writing my own bare metal code for a Raspberry PI as I build up my bare metal skills and learn about kernel mode operations. However, due to the complexity, amount of documentation errors, and missing/scattered info, it has been extremely difficult to finally bring up a custom kernel on the Raspberry PI. However, I finally got that working.
A very broad overview of what is happening in the bootstrap process
My kernel loads into 0x80000, sends all cores except core 0 into an infinite loop, sets up the Stack Pointer, and calls a C function. I can setup the GPIO pins and turn them on and off. Using some additional circuitry, I can drive LEDs and confirm that my code is executing.
The problem
However when it comes to the UART, I have hit a wall. I am using UART0 (PL011). As far as I can tell, the UART is not outputting, although I could be missing it on my scope since I only have an analog oscilloscope. The code gets stuck when outputting the string. I have determined through hours of reflashing my SD card with different YES/NO questions to my LEDs that it is stuck in an infinite loop waiting for the UART Transmit FIFO Full flag to clear. The UART only accepts 1 byte before becoming full. I can not figure out why it is not transmitting the data out. I am also not sure if I have correctly set my baud-rate, but I don't think that would cause the TX FIFO to stay filled.
Getting a foothold in the code
Here is my code. The execution begins at the very beginning of the binary. It is constructed by being linked with symbol "my_entry_pt" from assembly source "entry.s" in the linker script. That is where you will find the entry code. However, you probably only need to look at the last file which is the C code in "base.c". The rest is just bootstrapping up to that. Please disregard some comments/names which don't make sense. This is a port (of primarily the build infrastructure) from an earlier bare-metal project of mine. That project used a RISC-V development board which uses a memory mapped SPI flash to store the binary code of the program.:
[Makefile]
TUPLE := aarch64-unknown-linux-gnu
CC := $(TUPLE)-gcc
OBJCPY := $(TUPLE)-objcopy
STRIP := $(TUPLE)-strip
CFLAGS := -Wall -Wextra -std=c99 -O2 -march=armv8-a -mtune=cortex-a53 -mlittle-endian -ffreestanding -nostdlib -nostartfiles -Wno-unused-parameter -fno-stack-check -fno-stack-protector
LDFLAGS := -static
GFILES :=
KFILES :=
UFILES :=
# Global Library
#GFILES := $(GFILES)
# Kernel
# - Core (Entry/System Setup/Globals)
KFILES := $(KFILES) ./src/kernel/base.o
KFILES := $(KFILES) ./src/kernel/entry.o
# Programs
# - Init
#UFILES := $(UFILES)
export TUPLE
export CC
export OBJCPY
export STRIP
export CFLAGS
export LDFLAGS
export GFILES
export KFILES
export UFILES
.PHONY: all rebuild clean
all: prog-metal.elf prog-metal.elf.strip prog-metal.elf.bin prog-metal.elf.hex prog-metal.elf.strip.bin prog-metal.elf.strip.hex
rebuild: clean
$(MAKE) all
clean:
rm -f *.elf *.strip *.bin *.hex $(GFILES) $(KFILES) $(UFILES)
%.o: %.c
$(CC) $(CFLAGS) $^ -c -o $#
%.o: %.s
$(CC) $(CFLAGS) $^ -c -o $#
prog-metal.elf: $(GFILES) $(KFILES) $(UFILES)
$(CC) $(CFLAGS) $^ -T ./bare_metal.ld $(LDFLAGS) -o $#
prog-%.elf.strip: prog-%.elf
$(STRIP) -s -x -R .comment -R .text.startup -R .riscv.attributes $^ -o $#
%.elf.bin: %.elf
$(OBJCPY) -O binary $^ $#
%.elf.hex: %.elf
$(OBJCPY) -O ihex $^ $#
%.strip.bin: %.strip
$(OBJCPY) -O binary $^ $#
%.strip.hex: %.strip
$(OBJCPY) -O ihex $^ $#
emu: prog-metal.elf.strip.bin
qemu-system-aarch64 -kernel ./prog-metal.elf.strip.bin -m 1G -cpu cortex-a53 -M raspi3 -serial stdio -display none
emu-debug: prog-metal.elf.strip.bin
qemu-system-aarch64 -kernel ./prog-metal.elf.strip.bin -m 1G -cpu cortex-a53 -M raspi3 -serial stdio -display none -gdb tcp::1234 -S
debug:
$(TUPLE)-gdb -ex "target remote localhost:1234" -ex "layout asm" -ex "tui reg general" -ex "break *0x00080000" -ex "break *0x00000000" -ex "set scheduler-locking step"
[bare_metal.ld]
/*
This is not actually needed (At least not on actual hardware.), but
it explicitly sets the entry point in the .elf file to be the same
as the true entry point to the program. The global symbol my_entry_pt
is located at the start of src/kernel/entry.s. More on this below.
*/
ENTRY(my_entry_pt)
MEMORY
{
/*
This is the memory address where this program will reside.
It is the reset vector.
*/
ram (rwx) : ORIGIN = 0x00080000, LENGTH = 0x0000FFFF
}
SECTIONS
{
/*
Force the linker to starting at the start of memory section: ram
*/
. = 0x00080000;
.text : {
/*
Make sure the .text section from src/kernel/entry.o is
linked first. The .text section of src/kernel/entry.s
is the actual entry machine code for the kernel and is
first in the file. This way, at reset, exection starts
by jumping to this machine code.
*/
src/kernel/entry.o (.text);
/*
Link the rest of the kernel's .text sections.
*/
*.o (.text);
} > ram
/*
Put in the .rodata in the flash after the actual machine code.
*/
.rodata : {
*.o (.rodata);
*.o (.rodata.*);
} > ram
/*
END: Read Only Data
START: Writable Data
*/
.sbss : {
*.o (.sbss);
} > ram
.bss : {
*.o (.bss);
} > ram
section_KHEAP_START (NOLOAD) : ALIGN(0x10) {
/*
At the very end of the space reserved for global variables
in the ram, link in this custom section. This is used to
add a symbol called KHEAP_START to the program that will
inform the C code where the heap can start. This allows the
heap to start right after the global variables.
*/
src/kernel/entry.o (section_KHEAP_START);
} > ram
/*
Discard everything that hasn't be explictly linked. I don't
want the linker to guess where to put stuff. If it doesn't know,
don't include it. If this casues a linking error, good. I want
to know that I need to fix something, rather than a silent failure
that could cause hard to debug issues later. For instance,
without explicitly setting the .sbss and .bss sections above,
the linker attempted to put my global variables after the
machine code in the flash. This would mean that ever access to
those variables would mean read a write to the external SPI flash
IC on real hardware. I do not believe that initialized globals
are possible since there is nothing to initialize them. So I don't
want to, for instance, include the .data section.
*/
/DISCARD/ : {
* (.*);
}
}
[src/kernel/entry.s]
.section .text
.globl my_entry_pt
// This is the Arm64 Kernel Header (64 bytes total)
my_entry_pt:
b end_of_header // Executable code (64 bits)
.align 3, 0, 7
.quad my_entry_pt // text_offset (64 bits)
.quad 0x0000000000000000 // image_size (64 bits)
.quad 0x000000000000000A // flags (1010: Anywhere, 4K Pages, LE) (64 bits)
.quad 0x0000000000000000 // reserved 2 (64 bits)
.quad 0x0000000000000000 // reserved 3 (64 bits)
.quad 0x0000000000000000 // reserved 4 (64 bits)
.int 0x644d5241 // magic (32 bits)
.int 0x00000000 // reserved 5 (32 bits)
end_of_header:
// Check What Core This Is
mrs x0, VMPIDR_EL2
and x0, x0, #0x3
cmp x0, #0x0
// If this is not core 0, go into an infinite loop
bne loop
// Setup the Stack Pointer
mov x2, #0x00030000
mov sp, x2
// Get the address of the C main function
ldr x1, =kmain
// Call the C main function
blr x1
loop:
nop
b loop
.section section_KHEAP_START
.globl KHEAP_START
KHEAP_START:
[src/kernel/base.c]
void pstr(char* str) {
volatile unsigned int* AUX_MU_IO_REG = (unsigned int*)(0x3f201000 + 0x00);
volatile unsigned int* AUX_MU_LSR_REG = (unsigned int*)(0x3f201000 + 0x18);
while (*str != 0) {
while (*AUX_MU_LSR_REG & 0x00000020) {
// TX FIFO Full
}
*AUX_MU_IO_REG = (unsigned int)((unsigned char)*str);
str++;
}
return;
}
signed int kmain(unsigned int argc, char* argv[], char* envp[]) {
char* text = "Test Output String\n";
volatile unsigned int* AUXENB = 0;
//AUXENB = (unsigned int*)(0x20200000 + 0x00);
//*AUXENB |= 0x00024000;
//AUXENB = (unsigned int*)(0x20200000 + 0x08);
//*AUXENB |= 0x00000480;
// Set Baud Rate to 115200
AUXENB = (unsigned int*)(0x3f201000 + 0x24);
*AUXENB = 26;
AUXENB = (unsigned int*)(0x3f201000 + 0x28);
*AUXENB = 0;
AUXENB = (unsigned int*)(0x3f200000 + 0x04);
*AUXENB = 0;
// Set GPIO Pin 14 to Mode: ALT0 (UART0)
*AUXENB |= (04u << ((14 - 10) * 3));
// Set GPIO Pin 15 to Mode: ALT0 (UART0)
*AUXENB |= (04u << ((15 - 10) * 3));
AUXENB = (unsigned int*)(0x3f200000 + 0x08);
*AUXENB = 0;
// Set GPIO Pin 23 to Mode: Output
*AUXENB |= (01u << ((23 - 20) * 3));
// Set GPIO Pin 24 to Mode: Output
*AUXENB |= (01u << ((24 - 20) * 3));
// Turn ON Pin 23
AUXENB = (unsigned int*)(0x3f200000 + 0x1C);
*AUXENB = (1u << 23);
// Turn OFF Pin 24
AUXENB = (unsigned int*)(0x3f200000 + 0x28);
*AUXENB = (1u << 24);
// Enable TX on UART0
AUXENB = (unsigned int*)(0x3f201000 + 0x30);
*AUXENB = 0x00000101;
pstr(text);
// Turn ON Pin 24
AUXENB = (unsigned int*)(0x3f200000 + 0x1C);
*AUXENB = (1u << 24);
return 0;
}
Debugging up the this point
So it turns out that all of us were right. My initial trouble shooting in response to #Xiaoyi Chen was wrong. I rebooted back into Raspberry Pi OS to check on a hunch. I was connected to the PI using a 3.3V UART adapter connected to pins 8 (GPIO 14, UART0 TX), 10 (GPIO 15, UART0 RX), and GND(for a common ground of course). I could see the boot messages and a getty login prompt which I could log into. I figured that meant that the PL011 was working, but when I actually checked the process list in htop, I found that getty was actually running on /dev/ttyS0 not /dev/ttyAMA0. /dev/ttyAMA0 was actually being tied to the bluetooth module with the hciattach command in another process listing.
According to the documentation here: https://www.raspberrypi.org/documentation/configuration/uart.md , /dev/ttyS0 is the mini UART while /dev/AMA0 is the PL011, but it also says that UART0 is PL011 and UART1 is the mini UART. Furthermore, the GPIO Pinouts and the BCM2835 documentation say that GPIO Pins 14 and 15 are for the UART0 TX and RX. So something did not add up if I can see the login prompt on pins 14 and 15 when Linux is using the mini UART, but I am supposedly physically connected to the PL011. If I SSHed in and tried to open /dev/ttyAMA0 with minicom, I could see nothing happening. However, if I did the same with /dev/ttyS0, it would conflict with the login terminal. This confirmed to me that /dev/ttyS0 was in fact in use for the boot console.
The Answer
If I set "dtoverlay=disable-bt" in config.txt, the above behavior changed to match expectations. Rebooting the PI made it once again come up with a console on header pins 8 and 10, but checking the process listing showed that this time getty was using /dev/ttyAMA0. If then set "dtoverlay=disable-bt" in config.txt with my custom kernel, the program executed as expected, printing out my string and turned on the second LED. Since the outputs of the PL011 were never actually setup, since it was redirected by some magic, it makes sense that it would not be working as #PMF suggested. This whole deal has just reaffirmed my assertion that the documentation for this so-called "learning computer" is atrocious.
For those who are curious, here are the last few lines from my config.txt:
[all]
dtoverlay=disable-bt
enable_uart=1
core_freq=250
#uart_2ndstage=1
force_eeprom_read=0
disable_splash=1
init_uart_clock=48000000
init_uart_baud=115200
kernel_address=0x80000
kernel=prog-metal.elf.strip.bin
arm_64bit=1
Remaining Questions
A few things still bother me. I could have sworn that I already tried setting "dtoverlay=disable-bt".
Secondly it does seem that this preforms some kind of magic under the hood that is not documented(I know of no documentation for it.) and I do not understand. I can find nothing in the published schematics that redirect the output of GPIO 14 and 15 from the SOC. So ether the schematics are incomplete or there is some proprietary magic taking place inside the SOC which redirects the pins, contradicting the documentation.
I also have questions about precedence when it comes to the config.txt options and setting things up elsewhere.
Anyway, thank you for the help everyone.
My suggestion:
flash your SD card to a rpi distribution to make sure the hardware is still working
if the hardware is good, check the difference of your code with the in-kernel serial driver
I've written a simple kernel that tries to write two characters to the frame buffer.
If I define a string literal in the kernel, I get the following output when it boots:
Booting 'os'
kernel /boot/kernel.elf
Error 13: Invalid or unsupported executable format
Press any key to continue...
Otherwise, if I define two characters I get the following (note 'ab' at the start of the output):
abBooting 'os'
kernel /boot/kernel.elf
[Multiboot-elf, <0x100000:0x201:0x0>, <0x101000:0x0:0x1000>, shtab=0x102168,
entry=0x1001f0]
loader
I wrote the loader in assembly:
global loader ; the entry symbol for ELF
MAGIC_NUMBER equ 0x1BADB002 ; define the magic number constant
FLAGS equ 0x0 ; multiboot flags
CHECKSUM equ -MAGIC_NUMBER ; calculate the checksum
; (magic number + checksum + flags should equal 0)
KERNEL_STACK_SIZE equ 4096 ; size of stack in bytes
section .text: ; start of the text (code) section
align 4 ; the code must be 4 byte aligned
dd MAGIC_NUMBER ; write the magic number to the machine code,
dd FLAGS ; the flags,
dd CHECKSUM ; and the checksum
loader: ; the loader label (defined as entry point in linker script)
mov eax, 0xCAFEBABE ; place the number 0xCAFEBABE in the register eax
mov esp, kernel_stack + KERNEL_STACK_SIZE ; point esp to the start of the
; stack (end of memory area)
extern run
call run
.loop:
jmp .loop ; loop forever
section .bss
align 4 ; align at 4 bytes
kernel_stack: ; label points to beginning of memory
resb KERNEL_STACK_SIZE ; reserve stack for the kernel
The kernel is written in c
#include "io.h"
#include "fb.h"
void run()
{
// try writing message to port
char* c = (char *) 10000;
c[0] = 'a';
c[1] = 'b';
fb_write(c, 2); // this does not cause the error
// fb_write("ab",2); // this line would cause the error
}
External headers
There are two external headers. One for IO ports called io.h and one for writing to the frame buffer called fb.h
Here is io.h and the implementation io.s
io.h:
#ifndef INCLUDE_IO_H
#define INCLUDE_IO_H
/** outb:
* Sends the given data to the given I/O port. Defined in io.s
*
* #param port The I/O port to send the data to
* #param data The data to send to the I/O port
*/
void outb(unsigned short port, unsigned char data);
#endif /* INCLUDE_IO_H */
io.s:
global outb ; make the label outb visible outside this file
; outb - send a byte to an I/O port
; stack: [esp + 8] the data byte
; [esp + 4] the I/O port
; [esp ] return address
outb:
mov al, [esp + 8]
mov dx, [esp + 4]
out dx, al
ret
fb.h
#include "io.h"
// FRAME BUFFER ================================
// Text colors
#define FB_BLACK 0
#define FB_BLUE 1
#define FB_GREEN 2
#define FB_CYAN 3
#define FB_RED 4
#define FB_MAGENTA 5
#define FB_BROWN 6
#define FB_LT_GREY 7
#define FB_DARK_GREY 8
#define FB_LT_BLUE 9
#define FB_LT_GREEN 10
#define FB_LT_CYAN 11
#define FB_LT_RED 12
#define FB_LT_MAGENTA 13
#define FB_LT_BROWN 14
#define FB_WHITE 15
// IO PORTS
#define FB_COMMAND_PORT 0x3D4
#define FB_DATA_PORT 0x3D5
// IO PORT COMMANDS
#define FB_HIGH_BYTE_COMMAND 14 // move cursor command low
#define FB_LOW_BYTE_COMMAND 15 // move cursor command high
/** fb_write_cell:
* used to write a character to a cell in the framebuffer
*
* param i which cell to write to
* param c the ascii char to write
* param fg foreground color
* param bf background color
*/
void fb_write_cell(unsigned int i, char c, unsigned char fg, unsigned char bg);
/** fb_move_cursor:
* used to move the cursor within the frame buffer
*
* param pos position within frame buffer to move cursor to
*/
void fb_move_cursor(unsigned short pos);
/** fb_write:
* write some text to the cursor
*
* param buf pointer to character string
* param len length of string to write
*/
int fb_write(char *buf, unsigned int len);
fb.c
#include "fb.h"
void fb_write_cell(unsigned int i, char c, unsigned char fg, unsigned char bg)
{
char *fb = (char *) 0x000B8000;
fb[i*2] = c;
fb[i*2 + 1] = ((fg & 0x0F) << 4) | (bg & 0x0F);
}
void fb_move_cursor(unsigned short pos) {
outb(FB_COMMAND_PORT, FB_HIGH_BYTE_COMMAND);
outb(FB_DATA_PORT, ((pos>>8) & 0x00FF));
outb(FB_COMMAND_PORT, FB_LOW_BYTE_COMMAND);
outb(FB_DATA_PORT, pos & 0x00FF);
}
int fb_write(char *buf, unsigned int len) {
unsigned int i = 0;
for(i = 0; i < len; i++) {
fb_write_cell(i, buf[i], FB_BLACK, FB_WHITE);
}
return 0;
}
Building it
I have a linker script called link.ld and a Makefile. I'm using gcc cross compiler for i386-elf That I compiled using this guide (http://wiki.osdev.org/GCC_Cross-Compiler).
ENTRY(loader) /* the name of the entry label */
SECTIONS {
. = 0x00100000; /* the code should be loaded at 1 MB */
.text ALIGN (0x1000) : /* align at 4 KB */
{
*(.text) /* all text sections from all files */
}
.rodata ALIGN (0x1000) : /* align at 4 KB */
{
*(.rodata*) /* all read-only data sections from all files */
}
.data ALIGN (0x1000) : /* align at 4 KB */
{
*(.data) /* all data sections from all files */
}
.bss ALIGN (0x1000) : /* align at 4 KB */
{
sbss = .;
*(COMMON) /* all COMMON sections from all files */
*(.bss) /* all bss sections from all files */
ebss = .;
}
}
And here is my makefile
OBJECTS = io.o fb.o loader.o kmain.o
#CC = gcc
CC = /home/albertlockett/opt/cross/bin/i386-elf-gcc
CFLAGS = -m32 -nostdlib -nostdinc -fno-builtin -fno-stack-protector \
-nostartfiles -nodefaultlibs -Wall -Wextra -Werror -c
LDFLAGS = -T link.ld -melf_i386
AS = nasm
ASFLAGS = -f elf
all: kernel.elf
kernel.elf: $(OBJECTS)
ld $(LDFLAGS) $(OBJECTS) -o kernel.elf
os.iso: kernel.elf
cp kernel.elf iso/boot/kernel.elf
genisoimage -R \
-b boot/grub/stage2_eltorito \
-no-emul-boot \
-boot-load-size 4 \
-A os \
-input-charset utf8 \
-quiet \
-boot-info-table \
-o os.iso \
iso
run: os.iso
bochs -f bochsrc.txt -q
%.o: %.c
$(CC) $(CFLAGS) $< -o $#
%.o: %.s
$(AS) $(ASFLAGS) $< -o $#
clean:
rm -rf *.o kernel.elf os.iso
Run it
The makefile builds an iso from the contents of a directory called iso. That folder contains a preconfigured version of grub that I got here (https://github.com/littleosbook/littleosbook/blob/master/files/stage2_eltorito) and a menu.lst file for grub
menu.lst:
default=0
timeout=0
title os
kernel /boot/kernel.elf
contents of iso directory:
iso
`-- boot
|-- grub
| |-- menu.lst
| `-- stage2_eltorito
`-- kernel.elf
The iso image boots in bochs. Here is my bochsrc.txt file
megs: 32
display_library: term
romimage: file=/usr/share/bochs/BIOS-bochs-latest
vgaromimage: file=/usr/share/bochs/VGABIOS-lgpl-latest
ata0-master: type=cdrom, path=os.iso, status=inserted
boot: cdrom
log: bochslog.txt
clock: sync=realtime, time0=local
cpu: count=1, ips=1000000
com1: enabled=1, mode=file, dev=com1.out
Does anyone know why the string literal in the kernel file produces the error when I try to boot the iso?
You have an extra colon at the end of section .text: so that creates a new section named .text:. For some obscure reason that I couldn't find out from a quick glance at the documentation, this section is emitted to the output even though it is not listed in your linker script. When you have no literal data in the C code, you are lucky that it still falls within the first 8kiB of the image, so that the multiboot header is in the required portion. If you do have a string literal, you will get a new section .rodata and that, for yet another obscure reason, gets sorted before your .text: but after the standard .text. Example:
Sections:
Idx Name Size VMA LMA File off Algn
0 .text 00000001 00100000 00100000 00001000 2**4
CONTENTS, ALLOC, LOAD, READONLY, CODE
1 .rodata 00000005 00101000 00101000 00002000 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
2 .text: 00000018 00101008 00101008 00002008 2**2
CONTENTS, ALLOC, LOAD, READONLY, DATA
3 .bss 0000100a 00102000 00102000 00003000 2**2
ALLOC
As you can see it's no longer within the first 8kiB of the image, so grub will be very sad.
TL;DR: remove the extra colon after section .text:.