Does the main function of a C program ever reclaim the stack? - c

Question

I am working through the OverTheWire wargames and one of my exploits overwrites the return address of main with the address of system. I have then used the fact that at the point main returns, esp is still pointing at one of my local variables and hence I can fill it with the command I want system to run (e.g. sh;#).
My confusion comes from that I thought functions in C reclaim the stack before returning and hence at the point the return address is called the stack pointer would be pointing at the return address rather than at the local variables. However, my exploit works so it seems that my stack pointer is pointing at the local variables when the return address is called.
The main thing I have noticed about this particular challenge compared to others is that it calls exit(0) at the end, instead of just ending, so the assembly doesn't end with leave, which may be the reason for this behaviour.
I haven't included the actual code since it's quite long and I was hoping there was a general explanation for what I am seeing, but please let me know if the assembly would be useful.

Answer 1

#include <stdio.h>
int main ( void )
{
printf("hello\n");
return(0);
}
the interesting relevant parts.
0000000000400430 <main>:
400430: 48 83 ec 08 sub $0x8,%rsp
400434: bf d4 05 40 00 mov $0x4005d4,%edi
400439: e8 c2 ff ff ff callq 400400 <puts#plt>
40043e: 31 c0 xor %eax,%eax
400440: 48 83 c4 08 add $0x8,%rsp
400444: c3 retq
400445: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
40044c: 00 00 00
40044f: 90 nop
0000000000400450 <_start>:
400450: 31 ed xor %ebp,%ebp
400452: 49 89 d1 mov %rdx,%r9
400455: 5e pop %rsi
400456: 48 89 e2 mov %rsp,%rdx
400459: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp
40045d: 50 push %rax
40045e: 54 push %rsp
40045f: 49 c7 c0 c0 05 40 00 mov $0x4005c0,%r8
400466: 48 c7 c1 50 05 40 00 mov $0x400550,%rcx
40046d: 48 c7 c7 30 04 40 00 mov $0x400430,%rdi
400474: e8 97 ff ff ff callq 400410 <__libc_start_main#plt>
400479: f4 hlt
40047a: 66 0f 1f 44 00 00 nopw 0x0(%rax,%rax,1)
For the most part there is nothing special about main nor printf, etc these are just functions that conform to the calling convention. As re-asked SO questions will show sometimes the compiler will add extra stack or other calls when it sees a main() that it doesnt otherwise. but still it is a function that needs to conform to the calling convention. As seen in this case where the stack pointer is put back where it was found.
Before an operating system (Linux, Windows, MacOS, etc) can even think about running a program it needs to allocate some space for that program and tag that memory for that program in some way depending on the features of the processor and the OS, etc. Then you load the program from whatever media, and launch it at the binary file specified and/or well known entry point. A clean exit of the program will cause the operating system to free that memory, which the .text, .data, .bss and stack are the trivial/obvious ones that just go away as their memory just goes away. Other items that may have been allocated and associated with this program, open files, runtime allocated (not stack) memory, etc can/should also be freed, depends on the design of the os and/or the C library as to how that happens.
In the above case we see the bootstrap calls main and main returns then hlt is hit which this is an application not kernel code so that should cause a trap that causes the OS to clean up. An explicit exit() should be no different than a printf() or puts() or fopen() or any other function that ultimately makes one or more syscalls to the operating system. All that you can possibly find for these types of operating systems (Linux, Windows, MacOS) is the syscall. The release of memory happens outside the program as the program does not have control over it, would be a chicken and egg problem, program frees mmu tables that is using to free mmu tables...
compile and disassemble the object for main rather than the whole program
0000000000000000 <main>:
0: 48 83 ec 08 sub $0x8,%rsp
4: bf 00 00 00 00 mov $0x0,%edi
9: e8 00 00 00 00 callq e <main+0xe>
e: 31 c0 xor %eax,%eax
10: 48 83 c4 08 add $0x8,%rsp
14: c3 retq
no surprise there same as before, all we needed to see to understand that the stack was cleaned up before return. and that main is not special:
#include <stdio.h>
int notmain ( void )
{
printf("hello\n");
return(0);
}
0000000000000000 <notmain>:
0: 48 83 ec 08 sub $0x8,%rsp
4: bf 00 00 00 00 mov $0x0,%edi
9: e8 00 00 00 00 callq e <notmain+0xe>
e: 31 c0 xor %eax,%eax
10: 48 83 c4 08 add $0x8,%rsp
14: c3 retq
Now if you are asking if there is an exit() within main then sure it wont hit the return point in main so the stack pointer is offset by whatever amount. but if main calls some function and that function calls some function then that function calls exit() then the stack pointer is left at the stack frame point of function number two plus whatever the call (this is an x86) plus the exit() stack frame adds to it. You cannot simply assume that when exit() is called, if it is called, what the stack pointer is pointing at. You would have to examine the disassembly around that call to exit() plus the exit() code and anything it calls, to figure this out.