I am working on a driver code, which is causing stack overrun
issues and memory corruption. Presently running the module gives,
"Exception stack" and the stack trace looks corrupted.
The module had compile warnings. The warnings were resolved
with gcc option "-WFrame-larger-than=len".
The issue is possibly being caused by excessive in-lining and lots of
function arguments and large number of nested functions. I need to continue
testing and continue re-factoring the code, is it possible to
make any modifications kernel to increase the stack size ? Also how would you go about debugging such issues.
Though your module would compile with warnings with "-WFrame-larger-than=len", it would still cause the stack overrun and could corrupt the in-core data structures, leading the system to an inconsistency state.
The Linux kernel stack size was limited to the 8KiB (in kernel versions earlier before 3.18), and now 16KiB (for the versions later than 3.18). There is a recent commit due to lots of issues in virtio and qemu-kvm, kernel stack has been extended to 16KiB.
Now if you want to increase stack size to 32KiB, then you would need to recompile the kernel, after making the following change in the kernel source file:(arch/x86/include/asm/page_64_types.h)
// for 32K stack
- #define THREAD_SIZE_ORDER 2
+ #define THREAD_SIZE_ORDER 3
A recent commit shows on Linux kernel version 3.18, shows the kernel stack size already being increased to 16K, which should be enough in most cases.
"
commit 6538b8ea886e472f4431db8ca1d60478f838d14b
Author: Minchan Kim <minchan#kernel.org>
Date: Wed May 28 15:53:59 2014 +0900
x86_64: expand kernel stack to 16K
"
Refer LWN: [RFC 2/2] x86_64: expand kernel stack to 16K
As for debugging such issues there is no single line answer how to, but here are some tips I can share. Use and dump_stack() within your module to get a stack trace in the syslog which really helps in debugging stack related issues.
Use debugfs, turn on the stack depth checking functions with:
# mount -t debugfs nodev /sys/kernel/debug
# echo 1 > /proc/sys/kernel/stack_tracer_enabled
and regularly capture the output of the following files:
# cat /sys/kernel/debug/tracing/stack_max_size
# cat /sys/kernel/debug/tracing/stack_trace
The above files will report the highest stack usage when the module is loaded and tested.
Leave the below command running:
while true; do date; cat /sys/kernel/debug/tracing/stack_max_size;
cat /sys/kernel/debug/tracing/stack_trace; echo ======; sleep 30; done
If you see the stack_max_size value exceeding maybe ~14000 bytes (for 16KiB stack version of the kernel) then the stack trace would be worth capturing looking into further. Also you may want to set-up crash tool to capture vmcore core file in cases of panics.
Related
I am creating the application on STM32 microcontroller. I am using some library stack. Above that stack, I am using my application. I have two questions:
How can I detect and handle the stack over flow in runtime. Because I don't know how much memory that library is using.
How can I detect and handle the stack over flow in runtime if I am developing the code from the scratch. I read some where we have to maintain some count for each declaration. Is this correct way or any standard way to find it.
if you are limited to your device and no "high sophisticated" tools available, you could at least try "the old way". A simple stack guard may help. Somewhere in your code (depends on the tools you use), there must be the definition of the stack area. Something similar to:
.equ stacksize, 1024
stack: .space stacksize,0
(gnu as syntax, your's might be different)
with your device's startup code somewhere initializing the stack register to the top address of the stack area.
A stack guard would then just add a "magic number" to the stack top and bottom:
.equ stackmagic,0xaffeaffe
.equ stacksize, 1024
stacktop: .int stackmagic
stack: .space stacksize,0
stackbottom: .int stackmagic
with some code at least periodically checking (e.g. in a timer interrupt routine or - if available - in your debugger) if the stackmagic values are still there.
If your software is not tiny, I would first try to debug most of it on your laptop or desktop or tablet (perhaps running Linux, because it has good tools and very standard compliant compilers, similar to the cross-compiler you are using). Then you can profit from tools like valgrind or GCC compilation options like -Wall -Wextra -g -fsanitize=address etc....
You might store an approximation of the top of stack at start of your main function (e.g. by doing extern int* start_top_of_stack; then int i=0; start_top_of_stack= &i; near beginning of your main function. You could then have some local int j=0; in several functions and check at start of them that &j - start_top_of_stack is not too big.
But there is no silver bullet. I am just suggesting a trick.
If your application is critical to the point of accepting costly development efforts, you could use some formal method & source static program analysis tools (e.g. Frama-C, or make your own using MELT). If you are cross-compiling with a recent GCC you might want to use -Wstack-usage=some length and/or -fstack-usage to check that every call frame is not too big, or to compute manually the required stack depth.
I am using the Hope functional program on Ubuntu 14.04 with gcc 4.8.2, and doing a highly recursive function to find a large number of prime numbers. However, I get a segmentation fault: 0x000000000040e03f in reach
cell=<error reading variable: Cannot access memory at address 0x7fffff7feff8> at runtime.c:250
The segmentation fault occurs when accessing address 0x7fffff7feff8.
What the reach routine is doing is unmarking heap items that can be reached by the current expression (using a garbage collection mark-sweep). The stack is very deep (100000+ calls), but there is not a stack overflow:
base_memory = 0x7ffff35a4010
top_string = 0x7ffff35a5260
BaseHeap = 0x7ffff35a5260
heap = 0x7ffff603a450
stack = 0x7ffff72da498
TopStack = 0x7ffff7584d60
The area from base_memory to TopStack was allocated with malloc.
Whenever I get a segment violation, the address is always 0x7fffff7feff8, even with very different functions.
If you google 0x7fffff7feff8 there are quite a few entries with segment violations with this address, with no solution to the problem.
I put code into check that the heap address was in the heap range, but it never failed.
I did a gdb
find 0x7ffff35a4010,0x7ffff7584d60,0x7fffff7feff8
and nothing was found.
Why does the address 0x7fffff7feff8 show up in so many problems? Is there something amiss with the stack mechanism, or do I need to change the code in some way for the platform?
This rather looks like a stack overflow on an x86-64 system without adress space layout randomization. If the stack begins at 0x7ffffffff000, as they do on such systems, 0x7fffff7feff8 is suspiciously close to 8 MB below the beginning of the stack, which is a common default thread stack size on Linux systems.
Dump the contents of /proc/self/maps and check if the start of the stack matches this (it is listed at the bottom), and check ulimit -s to see the stack size new processes get. If /proc/self/maps lists 0x7ffffffff000 as end of the stack address range andulimit -s prints 8192, what you have is simply a stack overflow. In this event, a quick fix would be to increase the stack size of new processes (subprocesses of the active shell) like so:
ulimit -Ss size_in_kilobytes
This will work up to a hard limit root may or may not impose. In the long run, it would probably be a good idea to rewrite the code in a less excessively recursive manner.
Also, if all this hits home, you should probably enable ASLR (sysctl kernel.randomize_va_sapce=1. If that complains, your system is in dire need of an upgrade. Write kernel.randomize_va_space = 1 to /etc/sysctl.conf to make it permanent).
I think I am experiencing a stack overflow issue when running my unit-tests through gdb within emacs on Windows.
I run exactly the same unit-tests on linux with no problems
I am also aware that I am using a horrendously memory inefficient stack-based .ini file parser within these unit-tests so it would seem to be a reasonable possibility that stack overflow is occuring.
I have noticed a few unit tests faling on Windows that passed on Linux. Further investigation reveals a (stack-based) counter in a for loop resetting itself to zero at random points in the for loops execution and the (stack-based) values in the arrays that the for loop is inspecting change for the same index value
I noticed that gdb seems to be allocated its own thread of execution under Windows - is there any way of finding out how much stack space the thread is allocated?
One of the differences between Linux and Windows is that on Windows the stack-size has to be set at compile-time (there are two sizes, initial and a predefined reserved limit). Not sure what the default is with the compiler you are using, but you might try to increase it using --stack parameter (gcc).
On Linux the stack-size is dynamic up to the limits, generally set by the sysadmin.
Well then, perhaps Windows has tighter restrictions on the maximum amount of stack per process than Linux does?
This page details how to debug a stack overflow in Windows. It's not based on gdb, but perhaps you can still infer something.
I have a quite big driver module that I am trying to compile for a recent Linux kernel (3.4.4). I can successfully compile and insmod the same module with a 2.6.27.25 kernel.
GCC version are also different, 4.7.0 vs 4.3.0. Note that this module is quite complicated and I cannot simply go through all the code and all the makefiles.
When "inserting" the module I get a Cannot allocate memory with the following traces:
vmap allocation for size 30248960 failed: use vmalloc=<size> to increase size.
vmalloc: allocation failure: 30243566 bytes
insmod: page allocation failure: order:0, mode:0xd2
Pid: 5840, comm: insmod Tainted: G O 3.4.4-5.fc17.i686 #1
Call Trace:
[<c092702a>] ? printk+0x2d/0x2f
[<c04eff8d>] warn_alloc_failed+0xad/0xf0
[<c05178d9>] __vmalloc_node_range+0x169/0x1d0
[<c0517994>] __vmalloc_node+0x54/0x60
[<c0490825>] ? sys_init_module+0x65/0x1d80
[<c0517a60>] vmalloc+0x30/0x40
[<c0490825>] ? sys_init_module+0x65/0x1d80
[<c0490825>] sys_init_module+0x65/0x1d80
[<c050cda6>] ? handle_mm_fault+0xf6/0x1d0
[<c0932b30>] ? spurious_fault+0xae/0xae
[<c0932ce7>] ? do_page_fault+0x1b7/0x450
[<c093665f>] sysenter_do_call+0x12/0x28
-- clip --
The obvious answer seems to be that the module is allocating too much memory, however:
I have no problem with the old kernel version, what ever size this module is
if I prune some part of this module to get a much lower memory consumption, I will get always the same error message with the new kernel
I can unload a lot of other modules, but it has no impact (and is it anyway relevant? is there a global limit with Linux regarding the total memory usage by modules)
I am therefore suspecting a problem with the new kernel not directly related to limited memory.
The new kernel is complaining about a vmalloc() of 30,000 KB, but with the old kernel, an lsmod gives me a size of 4,800 KB. Should these figures be directly related? Is it possible that something went wrong during the build and that it is just too much RAM being requested? When I compile the sections size of both .ko, I do not see big differences.
So I am trying to understand where the problem is from. When I check the dumped stack, I am unable to find the matching piece of code. It seems that the faulty vmalloc() is done by sys_init_module(), which is init_module() from kernel/module.c. But the code does not match. When I check the object code from my .ko, the init_module() code also does not match.
I am more or less blocked as I do not know the kernel well enough, and all the build system and the module loading is quite tough to understand. The error occurs before the module is loaded, as I suspect that some functions are missing and insmod does not report these errors at this point.
I believe the allocation is done in layout_and_allocate, which is called by load_module. Both are static function, so they may be inlined, and therefore not on the stack.
So it's not an allocation done by your code, but an allocation done by Linux in order to load your code.
If your old kernel is 4.8MB and the new one is 30MB, it can explain why it fails.
So the question is why is it so large.
The size may be due to the amount of code (not likely that it has grown so much) or statically allocated data.
A likely explanation is that you have a large statically allocated array, whose size is defined in Linux. If the size has grown significantly, your array would grow.
A guess - an array whose size is NR_CPUS.
You should be able to use commands such as nm or objdump to find such an array. I'm not sure how exactly to do it however.
The problem was actually due to the debug sections in the module. The old kernel was able to ignore these sections, but the new one was counting them in the total size to allocate. However, when enabling the pr_debug() traces from module.c at loading time, these sections were not dumped with the others.
How to get rid of them and solve the problem:
objcopy -R .debug_aranges \
-R .debug_info \
-R .debug_abbrev \
-R .debug_line \
-R .debug_frame \
-R .debug_str \
-R .debug_loc \
-R .debug_ranges \
orignal.ko new.ko
It is also possible that the specific build files for this project were adding debug information "tailored" for the old kernel version, but when trying with a dummy module, I find exactly the same kind of debug sections appended, so I would rather suspect some policy change regarding module management in the kernel or in Fedora.
Any information regarding these changes are welcome.
I'm having a blast tracking down some heap corruption. I've enabled standard page heap verification with
gflags /p /enable myprogram.exe
and this succeeds in confirming the corruption:
===========================================================
VERIFIER STOP 00000008: pid 0x1040: corrupted suffix pattern
10C61000 : Heap handle
19BE0CF8 : Heap block
00000010 : Block size
00000000 :
===========================================================
When I turn on full page heap verification (gflags /p /enable myprogram.exe /full) in anticipation that this will cause an error to occur at the time the corruption is introduced, I get nothing more.
I started to get my hopes up while reading Advanced Windows Debugging: Memory Corruption Part II—Heaps, which is a chapter from Advanced Windows Debugging. I installed WinDbg, and downloaded debug symbols for user32.dll, kernel32.dll, ntdll.dll according to http://support.microsoft.com/kb/311503. Now when the program halts in the debugger I can issue this command to see information about the heap page:
0:000> dt _DPH_BLOCK_INFORMATION 19BE0CF8-0x20
ntdll!_DPH_BLOCK_INFORMATION
+0x000 StartStamp : 0xabcdaaaa
+0x004 Heap : 0x90c61000
+0x008 RequestedSize : 0x10
+0x00c ActualSize : 0x38
+0x010 FreeQueue : _LIST_ENTRY [ 0x0 - 0x0 ]
+0x010 TraceIndex : 0
+0x018 StackTrace : (null)
+0x01c EndStamp : 0xdcbaaaaa
I am dismayed by the (null) stack trace. Now, http://msdn.microsoft.com/en-us/library/ms220938%28VS.80%29.aspx says:
The StackTrace field will not always contain a non-null value for various reasons. First of all stack trace detection is supported only on x86 platforms and second, even on x86 machines the stack trace detection algorithms are not completely reliable. If the block is an allocated block the stack trace is for the allocation moment. If the block was freed, the stack trace is for the free moment.
But I wonder if anyone has any thoughts on increasing the chances of seeing the stack trace from the allocation moment.
Thanks for reading!
Ah ha! Turns out I needed to enable more gflags options:
gflags /i myprogram.exe +ust
Which has this effect:
ust - Create user mode stack trace database
Seems straightforward when I see parameter description. Silly me. But I also seem to need to set the size of the trace database before it will take effect:
gflags /i myprogram.exe /tracedb 512
...or whatever (in MB).
According to Microsoft, the malloc function in the C run-time (CRT) module uses the frame pointer omission (FPO) in some Windows versions. You may not see the complete stack information of the malloc function. (http://support.microsoft.com/kb/268343)
If possible, try to link the debug version CRT, e.g. link with /MDd option, to solve this issue.