Smashing the stack not working - c

Question

I have gone through the walkthrough about smashing the stack. Both the one http://insecure.org/stf/smashstack.html here and one I found on here Trying to smash the stack. I understand what is suppose to be happening, but I can't get it to work properly.
This is just like the other scenarios. I need to skip x=1 and print 0 as the value of x.
I compile with:
gcc file.c
The original code :
void function(){
char buffer[8];
}
void main(){
int x;
x = 0;
function();
x = 1;
printf("%d\n", x);
}
When I run
objdump -dS a.out
I get
0000000000400530 <function>:
400530: 55 push %rbp
400531: 48 89 e5 mov %rsp,%rbp
400534: 5d pop %rbp
400535: c3 retq
0000000000400536 <main>:
400536: 55 push %rbp
400537: 48 89 e5 mov %rsp,%rbp
40053a: 48 83 ec 20 sub $0x20,%rsp
40053e: 89 7d ec mov %edi,-0x14(%rbp)
400541: 48 89 75 e0 mov %rsi,-0x20(%rbp)
400545: c7 45 fc 00 00 00 00 movl $0x0,-0x4(%rbp)
40054c: b8 00 00 00 00 mov $0x0,%eax
400551: e8 da ff ff ff callq 400530 <function>
400556: c7 45 fc 01 00 00 00 movl $0x1,-0x4(%rbp)
40055d: 8b 45 fc mov -0x4(%rbp),%eax
400560: 89 c6 mov %eax,%esi
400562: bf 10 06 40 00 mov $0x400610,%edi
400567: b8 00 00 00 00 mov $0x0,%eax
40056c: e8 9f fe ff ff callq 400410 <printf#plt>
400571: c9 leaveq
400572: c3 retq
400573: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1)
40057a: 00 00 00
40057d: 0f 1f 00 nopl (%rax)
In the function I need to figure out how many bytes the return address is beyond the start of the buffer. I am not sure about this value. But since there are 6 bytes from the beginnig of the function to the return; would I add 7 bytes to the buffer?
Then I need to skip the instruction
x=1;
And since that instruction is 7 bytes long. Would I add 7 to return pointer?
Something like this?
void function(){
char buffer[8];
int *ret = buffer + 7;
(*ret) += 7;
}
void main(){
int x;
x = 0;
function();
x = 1;
printf("%d\n", x);
}
This throws the warning:
warning: initialization from incompatible pointer type [enabled by default]
int *ret = buffer1 + 5;
^
And the output is 1. What am I doing wrong? And can you explain how to do it right and why it is the correct way?
Thank you.

Answer 1

Try the function below, I wrote it for 32-bit compiler try using (-m32 gcc flag) or with a little effort you can make it work with your 64-bit compiler (Note that in your objdump listing you got 7 bytes offset between call to function and the next instruction so use 7 instead of 8.
void function(void)
{
unsigned long *x;
/* &x will more likely be at -4(ebp) */
/* Adding 1 (+4) gets us to stored ebp */
/* Adding 2 (+8) gets us to stored return address */
x = (unsigned long *)(&x + 2);
/* This is the tricky part */
/* TODO: On my 32-bit compiler gap between call to function
and the next instruction is 8 */
*x += 8;
}

Answer 2

We know that automatic variables are created on the stack - so taking the address of an automatic variable yields a pointer into the stack. When you call a void function, its return address is pushed onto the stack and the size of that address depends on your platform (4 or 8 bytes normally). So if you pass the address of an automatic variable to a function and then write over the memory before that address, you will damage the return address and smash the stack. Here is an example:
#include <stdlib.h>
#include <stdio.h>
static void f(int *p)
{
p[0] = 0x30303030;
p[1] = 0x31313131;
*(p - 1) = 0x35353535;
*(p - 2) = 0x36363636;
}
int main()
{
int a = 0x41424344;
int b = 0x45464748;
int c = 0x494a4b5c;
f(&b);
printf("%08x %08x %08x\n", a, b, c);
return 0;
}
I compiled this on linux with 'gcc -g' and ran under gdb and got this:
Program received signal SIGSEGV, Segmentation fault.
0x000000000040056a in f (p=0x7fffffffde74) at smash.c:10
10 }
(gdb) bt
#0 0x000000000040056a in f (p=0x7fffffffde74) at smash.c:10
#1 0x3636363600400594 in ?? ()
#2 0x3030303035353535 in ?? ()
#3 0x494a4b5c31313131 in ?? ()
#4 0x0000000000000000 in ?? ()
(gdb)
As you can see, the parent function addresses now contain some of my magic numbers. I ran this on 64 bit linux, so really I should have used 64 bit ints to fully overwrite the return address - as it is I left the lower word untouched.