I've written a simple assembly program:
section .data
str_out db "%d ",10,0
section .text
extern printf
extern exit
global main
main:
MOV EDX, ESP
MOV EAX, EDX
PUSH EAX
PUSH str_out
CALL printf
SUB ESP, 8 ; cleanup stack
MOV EAX, EDX
PUSH EAX
PUSH str_out
CALL printf
SUB ESP, 8 ; cleanup stack
CALL exit
I am the NASM assembler and the GCC to link the object file to an executable on linux.
Essentially, this program is first putting the value of the stack pointer into register EDX, it is then printing the contents of this register twice. However, after the second printf call, the value printed to the stdout does not match the first.
This behaviour seems strange. When I replace every usage of EDX in this program with EBX, the outputted integers are identical as expected. I can only infer that EDX is overwritten at some point during the printf function call.
Why is this the case? And how can I make sure that the registers I use in future don't conflict with C lib functions?
According to the x86 ABI, EBX, ESI, EDI, and EBP are callee-save registers and EAX, ECX and EDX are caller-save registers.
It means that functions can freely use and destroy previous values EAX, ECX, and EDX.
For that reason, save values of EAX, ECX, EDX before calling functions if you don't want their values to change. It is what "caller-save" mean.
Or better, use other registers for values that you're still going to need after a function call. push/pop of EBX at the start/end of a function is much better than push/pop of EDX inside a loop that makes a function call. When possible, use call-clobbered registers for temporaries that aren't needed after the call. Values that are already in memory, so they don't need to written before being re-read, are also cheaper to spill.
Since EBX, ESI, EDI, and EBP are callee-save registers, functions have to restore the values to the original for any of those they modify, before returning.
ESP is also callee-saved, but you can't mess this up unless you copy the return address somewhere.
The ABI for the target platform (e.g. 32bit x86 Linux) defines which registers can be used by functions without saving. (i.e., if you want them preserved across a call, you have to do it yourself).
Links to ABI docs for Windows and non-Window, 32 and 64bit, at https://stackoverflow.com/tags/x86/info
Having some registers that aren't preserved across calls (available as scratch registers) means functions can be smaller. Simple functions can often avoid doing any push/pop save/restores. This cuts down on the number of instructions, leading to faster code.
It's important to have some of each: having to spill all state to memory across calls would bloat the code of non-leaf functions, and slow things down esp. in cases where the called function didn't touch all the registers.
See also What are callee and caller saved registers? for more about call-preserved vs. call-clobbered registers in general.
Related
I belive push/pop instructions will result in a more compact code, maybe will even run slightly faster. This requires disabling stack frames as well though.
To check this, I will need to either rewrite a large enough program in assembly by hand (to compare them), or to install and study a few other compilers (to see if they have an option for this, and to compare the results).
Here is the forum topic about this and simular problems.
In short, I want to understand which code is better. Code like this:
sub esp, c
mov [esp+8],eax
mov [esp+4],ecx
mov [esp],edx
...
add esp, c
or code like this:
push eax
push ecx
push edx
...
add esp, c
What compiler can produce the second kind of code? They usually produce some variation of the first one.
You're right, push is a minor missed-optimization with all 4 major x86 compilers. There's some code-size, and thus indirectly performance to be had. Or maybe more directly a small amount of performance in some cases, e.g. saving a sub rsp instruction.
But if you're not careful, you can make things slower with extra stack-sync uops by mixing push with [rsp+x] addressing modes. pop doesn't sound useful, just push. As the forum thread you linked suggests, you only use this for the initial store of locals; later reloads and stores should use normal addressing modes like [rsp+8]. We're not talking about trying to avoid mov loads/stores entirely, and we still want random access to the stack slots where we spilled local variables from registers!
Modern code generators avoid using PUSH. It is inefficient on today's processors because it modifies the stack pointer, that gums-up a super-scalar core. (Hans Passant)
This was true 15 years ago, but compilers are once again using push when optimizing for speed, not just code-size. Compilers already use push/pop for saving/restoring call-preserved registers they want to use, like rbx, and for pushing stack args (mostly in 32-bit mode; in 64-bit mode most args fit in registers). Both of these things could be done with mov, but compilers use push because it's more efficient than sub rsp,8 / mov [rsp], rbx. gcc has tuning options to avoid push/pop for these cases, enabled for -mtune=pentium3 and -mtune=pentium, and similar old CPUs, but not for modern CPUs.
Intel since Pentium-M and AMD since Bulldozer(?) have a "stack engine" that tracks the changes to RSP with zero latency and no ALU uops, for PUSH/POP/CALL/RET. Lots of real code was still using push/pop, so CPU designers added hardware to make it efficient. Now we can use them (carefully!) when tuning for performance. See Agner Fog's microarchitecture guide and instruction tables, and his asm optimization manual. They're excellent. (And other links in the x86 tag wiki.)
It's not perfect; reading RSP directly (when the offset from the value in the out-of-order core is nonzero) does cause a stack-sync uop to be inserted on Intel CPUs. e.g. push rax / mov [rsp-8], rdi is 3 total fused-domain uops: 2 stores and one stack-sync.
On function entry, the "stack engine" is already in a non-zero-offset state (from the call in the parent), so using some push instructions before the first direct reference to RSP costs no extra uops at all. (Unless we were tailcalled from another function with jmp, and that function didn't pop anything right before jmp.)
It's kind of funny that compilers have been using dummy push/pop instructions just to adjust the stack by 8 bytes for a while now, because it's so cheap and compact (if you're doing it once, not 10 times to allocate 80 bytes), but aren't taking advantage of it to store useful data. The stack is almost always hot in cache, and modern CPUs have very excellent store / load bandwidth to L1d.
int extfunc(int *,int *);
void foo() {
int a=1, b=2;
extfunc(&a, &b);
}
compiles with clang6.0 -O3 -march=haswell on the Godbolt compiler explorer See that link for all the rest of the code, and many different missed-optimizations and silly code-gen (see my comments in the C source pointing out some of them):
# compiled for the x86-64 System V calling convention:
# integer args in rdi, rsi (,rdx, rcx, r8, r9)
push rax # clang / ICC ALREADY use push instead of sub rsp,8
lea rdi, [rsp + 4]
mov dword ptr [rdi], 1 # 6 bytes: opcode + modrm + imm32
mov rsi, rsp # special case for lea rsi, [rsp + 0]
mov dword ptr [rsi], 2
call extfunc(int*, int*)
pop rax # and POP instead of add rsp,8
ret
And very similar code with gcc, ICC, and MSVC, sometimes with the instructions in a different order, or gcc reserving an extra 16B of stack space for no reason. (MSVC reserves more space because it's targeting the Windows x64 calling convention which reserves shadow space instead of having a red-zone).
clang saves code-size by using the LEA results for store addresses instead of repeating RSP-relative addresses (SIB+disp8). ICC and clang put the variables at the bottom of the space it reserved, so one of the addressing modes avoids a disp8. (With 3 variables, reserving 24 bytes instead of 8 was necessary, and clang didn't take advantage then.) gcc and MSVC miss this optimization.
But anyway, more optimal would be:
push 2 # only 2 bytes
lea rdi, [rsp + 4]
mov dword ptr [rdi], 1
mov rsi, rsp # special case for lea rsi, [rsp + 0]
call extfunc(int*, int*)
# ... later accesses would use [rsp] and [rsp+] if needed, not pop
pop rax # alternative to add rsp,8
ret
The push is an 8-byte store, and we overlap half of it. This is not a problem, CPUs can store-forward the unmodified low half efficiently even after storing the high half. Overlapping stores in general are not a problem, and in fact glibc's well-commented memcpy implementation uses two (potentially) overlapping loads + stores for small copies (up to the size of 2x xmm registers at least), to load everything then store everything without caring about whether or not there's overlap.
Note that in 64-bit mode, 32-bit push is not available. So we still have to reference rsp directly for the upper half of of the qword. But if our variables were uint64_t, or we didn't care about making them contiguous, we could just use push.
We have to reference RSP explicitly in this case to get pointers to the locals for passing to another function, so there's no getting around the extra stack-sync uop on Intel CPUs. In other cases maybe you just need to spill some function args for use after a call. (Although normally compilers will push rbx and mov rbx,rdi to save an arg in a call-preserved register, instead of spilling/reloading the arg itself, to shorten the critical path.)
I chose 2x 4-byte args so we could reach a 16-byte alignment boundary with 1 push, so we can optimize away the sub rsp, ## (or dummy push) entirely.
I could have used mov rax, 0x0000000200000001 / push rax, but 10-byte mov r64, imm64 takes 2 entries in the uop cache, and a lot of code-size.
gcc7 does know how to merge two adjacent stores, but chooses not to do that for mov in this case. If both constants had needed 32-bit immediates, it would have made sense. But if the values weren't actually constant at all, and came from registers, this wouldn't work while push / mov [rsp+4] would. (It wouldn't be worth merging values in a register with SHL + SHLD or whatever other instructions to turn 2 stores into 1.)
If you need to reserve space for more than one 8-byte chunk, and don't have anything useful to store there yet, definitely use sub instead of multiple dummy PUSHes after the last useful PUSH. But if you have useful stuff to store, push imm8 or push imm32, or push reg are good.
We can see more evidence of compilers using "canned" sequences with ICC output: it uses lea rdi, [rsp] in the arg setup for the call. It seems they didn't think to look for the special case of the address of a local being pointed to directly by a register, with no offset, allowing mov instead of lea. (mov is definitely not worse, and better on some CPUs.)
An interesting example of not making locals contiguous is a version of the above with 3 args, int a=1, b=2, c=3;. To maintain 16B alignment, we now need to offset 8 + 16*1 = 24 bytes, so we could do
bar3:
push 3
push 2 # don't interleave mov in here; extra stack-sync uops
push 1
mov rdi, rsp
lea rsi, [rsp+8]
lea rdx, [rdi+16] # relative to RDI to save a byte with probably no extra latency even if MOV isn't zero latency, at least not on the critical path
call extfunc3(int*,int*,int*)
add rsp, 24
ret
This is significantly smaller code-size than compiler-generated code, because mov [rsp+16], 2 has to use the mov r/m32, imm32 encoding, using a 4-byte immediate because there's no sign_extended_imm8 form of mov.
push imm8 is extremely compact, 2 bytes. mov dword ptr [rsp+8], 1 is 8 bytes: opcode + modrm + SIB + disp8 + imm32. (RSP as a base register always needs a SIB byte; the ModRM encoding with base=RSP is the escape code for a SIB byte existing. Using RBP as a frame pointer allows more compact addressing of locals (by 1 byte per insn), but takes an 3 extra instructions to set up / tear down, and ties up a register. But it avoids further access to RSP, avoiding stack-sync uops. It could actually be a win sometimes.)
One downside to leaving gaps between your locals is that it may defeat load or store merging opportunities later. If you (the compiler) need to copy 2 locals somewhere, you may be able to do it with a single qword load/store if they're adjacent. Compilers don't consider all the future tradeoffs for the function when deciding how to arrange locals on the stack, as far as I know. We want compilers to run quickly, and that means not always back-tracking to consider every possibility for rearranging locals, or various other things. If looking for an optimization would take quadratic time, or multiply the time taken for other steps by a significant constant, it had better be an important optimization. (IDK how hard it might be to implement a search for opportunities to use push, especially if you keep it simple and don't spend time optimizing the stack layout for it.)
However, assuming there are other locals which will be used later, we can allocate them in the gaps between any we spill early. So the space doesn't have to be wasted, we can simply come along later and use mov [rsp+12], eax to store between two 32-bit values we pushed.
A tiny array of long, with non-constant contents
int ext_longarr(long *);
void longarr_arg(long a, long b, long c) {
long arr[] = {a,b,c};
ext_longarr(arr);
}
gcc/clang/ICC/MSVC follow their normal pattern, and use mov stores:
longarr_arg(long, long, long): # #longarr_arg(long, long, long)
sub rsp, 24
mov rax, rsp # this is clang being silly
mov qword ptr [rax], rdi # it could have used [rsp] for the first store at least,
mov qword ptr [rax + 8], rsi # so it didn't need 2 reg,reg MOVs to avoid clobbering RDI before storing it.
mov qword ptr [rax + 16], rdx
mov rdi, rax
call ext_longarr(long*)
add rsp, 24
ret
But it could have stored an array of the args like this:
longarr_arg_handtuned:
push rdx
push rsi
push rdi # leave stack 16B-aligned
mov rsp, rdi
call ext_longarr(long*)
add rsp, 24
ret
With more args, we start to get more noticeable benefits especially in code-size when more of the total function is spent storing to the stack. This is a very synthetic example that does nearly nothing else. I could have used volatile int a = 1;, but some compilers treat that extra-specially.
Reasons for not building stack frames gradually
(probably wrong) Stack unwinding for exceptions, and debug formats, I think don't support arbitrary playing around with the stack pointer. So at least before making any call instructions, a function is supposed to have offset RSP as much as its going to for all future function calls in this function.
But that can't be right, because alloca and C99 variable-length arrays would violate that. There may be some kind of toolchain reason outside the compiler itself for not looking for this kind of optimization.
This gcc mailing list post about disabling -maccumulate-outgoing-args for tune=default (in 2014) was interesting. It pointed out that more push/pop led to larger unwind info (.eh_frame section), but that's metadata that's normally never read (if no exceptions), so larger total binary but smaller / faster code. Related: this shows what -maccumulate-outgoing-args does for gcc code-gen.
Obviously the examples I chose were trivial, where we're pushing the input parameters unmodified. More interesting would be when we calculate some things in registers from the args (and data they point to, and globals, etc.) before having a value we want to spill.
If you have to spill/reload anything between function entry and later pushes, you're creating extra stack-sync uops on Intel. On AMD, it could still be a win to do push rbx / blah blah / mov [rsp-32], eax (spill to the red zone) / blah blah / push rcx / imul ecx, [rsp-24], 12345 (reload the earlier spill from what's still the red-zone, with a different offset)
Mixing push and [rsp] addressing modes is less efficient (on Intel CPUs because of stack-sync uops), so compilers would have to carefully weight the tradeoffs to make sure they're not making things slower. sub / mov is well-known to work well on all CPUs, even though it can be costly in code-size, especially for small constants.
"It's hard to keep track of the offsets" is a totally bogus argument. It's a computer; re-calculating offsets from a changing reference is something it has to do anyway when using push to put function args on the stack. I think compilers could run into problems (i.e. need more special-case checks and code, making them compile slower) if they had more than 128B of locals, so you couldn't always mov store below RSP (into what's still the red-zone) before moving RSP down with future push instructions.
Compilers already consider multiple tradeoffs, but currently growing the stack frame gradually isn't one of the things they consider. push wasn't as efficient before Pentium-M introduce the stack engine, so efficient push even being available is a somewhat recent change as far as redesigning how compilers think about stack layout choices.
Having a mostly-fixed recipe for prologues and for accessing locals is certainly simpler.
This requires disabling stack frames as well though.
It doesn't, actually. Simple stack frame initialisation can use either enter or push ebp \ mov ebp, esp \ sub esp, x (or instead of the sub, a lea esp, [ebp - x] can be used). Instead of or additionally to these, values can be pushed onto the stack to initialise the variables, or just pushing any random register to move the stack pointer without initialising to any certain value.
Here's an example (for 16-bit 8086 real/V 86 Mode) from one of my projects: https://bitbucket.org/ecm/symsnip/src/ce8591f72993fa6040296f168c15f3ad42193c14/binsrch.asm#lines-1465
save_slice_farpointer:
[...]
.main:
[...]
lframe near
lpar word, segment
lpar word, offset
lpar word, index
lenter
lvar word, orig_cx
push cx
mov cx, SYMMAIN_index_size
lvar word, index_size
push cx
lvar dword, start_pointer
push word [sym_storage.main.start + 2]
push word [sym_storage.main.start]
The lenter macro sets up (in this case) only push bp \ mov bp, sp and then lvar sets up numeric defs for offsets (from bp) to variables in the stack frame. Instead of subtracting from sp, I initialise the variables by pushing into their respective stack slots (which also reserves the stack space needed).
I use rsp and rbp registers for memory indexing in x64 inline assembly with VS 2015 C++. Is it possible without save and restore push rsp, push rbp,... pop rbp, pop rsp?
__asm{
mov rsi, memory
mov rsp, rsi
mov rbp, rsi
...
movntps [rsp+rdx], ymm0
...
}
In debugging mode, after mov rbp, rsi, the base address of memory is changed. Is this no problem?
When writing inline assembly, you must conform to the compiler's rules for what registers you are allowed to use. VS2015's rules are documented here and I will quote the key bit:
When using __asm to write assembly language in C/C++ functions, you don't need to preserve the EAX, EBX, ECX, EDX, ESI, or EDI registers...You should preserve other registers you use (such as DS, SS, SP, BP, and flags registers) for the scope of the __asm block.
Read "should" as "must". So the narrow answer to your question is "No, you must save and restore ESP and EBP if you want to use them for anything."
Reading between the lines, I think what you actually want is to write your entire function in assembly language. With Visual Studio, that means you will put it in a separate .ASM file. You still have to conform to the calling convention, but not the compiler's stricter rules for __asm blocks. Note that the calling convention still requires you to save and restore ESP and EBP (and, in fact, all registers except EAX, ECX, and EDX).
If you are actually coding for x64/Windows, then, if I'm reading this documentation correctly, you can't use __asm; you must use a separate .ASM file. And the calling convention is different.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
Let's consider a situation where we are writing in C code. When the compiler encounters a function call, my understanding is that it does the following:
Push all registers onto the stack
Jump to new function, do stuff in there
Pop old context off the stack back into the registers.
Now, some processors have 1 working register, some 32, some more than that. I'm mostly concerned with the larger number of registers. If my processor has 32 registers, the compiler will need to emit 32 push and pop instructions, just as base overhead for a function call. It would be nice if I could trade some compilation flexibility[1] in the function for less push and pop instructions. That is to say, I would like a way that I could tell the compiler "For function foo(), only use 4 registers. This would imply that the compiler would only need to push/pop 4 registers before jumping to foo().
I realize this is pretty silly to worry about on a modern PC, but I am thinking more for a low speed embedded system where you might be servicing an interrupt very quickly, or calling a simple function over and over. I also realize this could very quickly become an architecture dependant feature. Processors that use a "Source Source -> Dest" instruction set (Like ARM), as opposed to an accumulator (Like Freescale/NXP HC08) might have some lower limit on the number of registers we allow functions to use.
I do know the compiler uses tricks like inlining small functions to increase speed, and I realize I could inform most compilers to not generate the push/pop code and just hand code it myself in assembly, but my question focuses on instructing the compiler to do this from "C-Land".
My question is, are there compilers that allow this? Is this even necessary with optimizing compilers (do they already do this)?
[1] Compilation flexibility: By reducing the number of registers available to the compiler to use in a function body, you are restricting it's flexibility, and it might need to utilize the stack more since it can't just use another register.
When it comes to compilers, registers and function calls you can generally think of the registers falling into one of three categories: "hands off", volatile and non-volatile.
The "hands off" category are those that the compiler will not generally be futzing around with unless you explicitly tell it to (such as with inline assembly). These may include debugging registers and other special purpose registers. The list will vary from platform to platform.
The volatile (or scratch / call-clobbered / caller-saved) set of registers are those that a function can futz around with without the need for saving. That is, the caller understands that the contents of those registers might not be the same after the function call. Thus, if the caller has any data in those registers that it wants to keep, it must save that data before making the call and then restore it after. On a 32-bit x86 platform, these volatile registers (sometimes called scratch registers) are usually EAX, ECX and EDX.
The non-volatile (or call-preserved or callee-saved) set of registers are those that a function must save before using them and restore to their original values before returning. They only need to be saved/restored by the called function if it uses them. On a 32-bit x86 platform, these are usually the remaining general purpose registers: EBX, ESI, EDI, ESP, EBP.
Hope this helps.
(I meant to just add a small example, but quickly got carried away. I would add my own answer if this question wasn't closed, but I'm going to leave this long section here because I think it's interesting. Condense it or edit it out entirely if you don't want it in your answer -- Peter)
For a more concrete example, the SysV x86-64 ABI is well-designed (with args passed in registers, and a good balance of call-preserved vs. scratch/arg regs). There are some other links in the x86 tag wiki explaining what ABIs / calling conventions are all about.
Consider a simple example of with function calls that can't be inlined (because the definition isn't available):
int foo(int);
int bar(int a) {
return 5 * foo(a+2) + foo (a) ;
}
It compiles (on godbolt with gcc 5.3 for x86-64 with -O3 to the following:
## gcc output
# AMD64 SysV ABI: first arg in e/rdi, return value in e/rax
# the call-preserved regs used are: rbp and rbx
# the scratch regs used are: rdx. (arg-passing / return regs are not call-preserved)
push rbp # save a call-preserved reg
mov ebp, edi # stash `a` in a call-preserved reg
push rbx # save another call-preserved reg
lea edi, [rdi+2] # edi=a+2 as an arg for foo. `add edi, 2` would also work, but they're both 3 bytes and little perf difference
sub rsp, 8 # align the stack to a 16B boundary (the two pushes are 8B each, and call pushes an 8B return address, so another 8B is needed)
call foo # eax=foo(a+2)
mov edi, ebp # edi=a as an arg for foo
mov ebx, eax # stash foo(a+2) in ebx
call foo # eax=foo(a)
lea edx, [rbx+rbx*4] # edx = 5*foo(a+2), using the call-preserved register
add rsp, 8 # undo the stack offset
add eax, edx # the add between the to function-call results
pop rbx # restore the call-preserved regs we saved earlier
pop rbp
ret # return value in eax
As usual, compilers could do better: instead of stashing foo(a+2) in ebx to survive the 2nd call to foo, it could have stashed 5*foo(a+2) with a single instruction (lea ebx, [rax+rax*4]). Also, only one call-preserved register is needed, since we don't need a after the 2nd call. This removes a push/pop pair, and also the sub rsp,8 / add rsp,8 pair. (gcc bug report already filed for this missed optimization)
## Hand-optimized implementation (still ABI-compliant):
push rbx # save a call-preserved reg; also aligns the stack
lea ebx, [rdi+2] # stash ebx=a+2
call foo # eax=foo(a)
mov edi, ebx # edi=a+2 as an arg for foo
mov ebx, eax # stash foo(a) in ebx, replacing `a+2` which we don't need anymore
call foo # eax=foo(a+2)
lea eax, [rax+rax*4] #eax=5*foo(a+2)
add eax, ebx # eax=5*foo(a+2) + foo(a)
pop rbx # restore the call-preserved regs we saved earlier
ret # return value in eax
Note that the call to foo(a) happens before foo(a+2) in this version. It saved an instruction at the start (since we can pass on our arg unchanged to the first call to foo), but removed a potential saving later (since the multiply-by-5 now has to happen after the second call, and can't be combined with moving into the call-preserved register).
I could get rid of an extra mov if it was 5*foo(a) + foo(a+2). With the expression as I wrote it, I can't combine arithmetic with data movement (using lea) in every case. Or I'd need to both save a and do a separate add edi,2 before the first call.
Push all registers onto the stack
No. In the vast majority of function calls in optimized code, only a small fraction of all registers are pushed on the stack.
I'm mostly concerned with the larger number of registers.
Do you have any experimental evidence to support this concern? Is this a performance bottleneck?
I could trade some compilation flexibility[1] in the function for less
push and pop instructions.
Modern compilers use sophisticated inter-procedural register allocation. By limiting the number of registers, you will most likely degrade performance.
I realize this is pretty silly to worry about on a modern PC, but I am
thinking more for a low speed embedded system where you might be
servicing an interrupt very quickly, or calling a simple function over
and over.
This is very vague. You have to show the "simple" function, all call sites and specify the compiler and the target embedded system. You need to measure performance (compared to hand-written assembly code) to determine whether this is a problem in the first place.
I have an application which creates .text segment dumps of win32 processes. Then it divides the code on basic blocks. Basic block is a set of instructions which are executed always one after another (jumps are always the last instructions of such basic blocks). Here is an example:
Basic block 1
mov ecx, dword ptr [ecx]
test ecx, ecx
je 00401013h
Basic block 2
mov eax, dword ptr [ecx]
call dword ptr [eax+08h]
Basic block 3
test eax, eax
je 0040100Ah
Basic block 4
mov edx, dword ptr [eax]
push 00000001h
mov ecx, eax
call dword ptr [edx]
Basic block 5
ret 000008h
Now I would like to group such basic blocks in functions - say which basic blocks form a function. What's the algorithm? I have to remember that there might be many ret instructions inside one function. How to detect fast_call functions?
The simplest algorithm for grouping blocks into functions would be:
note all addresses to which calls are made with call some_address instructions
if the first block after such an address ends with ret, you're done with the function, else
follow the jump in the block to another block and so on until you've followed all possible execution paths (remember about conditional jumps, each of which splits a path into two) and all the paths have finished with ret. You'll need to recognize jumps that organize loops so your program itself does not hang by entering an infinite loop
Problems:
a number of calls can be made indirectly by reading function pointers from memory, e.g. you'd have call [some_address] instead of call some_address
some indirect calls can be made to calculated addresses
functions that call other functions before returning may have jump some_address instead of call some_address immediately followed by ret
call some_address can be simulated with a combination of push some_address + ret OR push some_address + jmp some_other_address
some functions may share code at their end (e.g. they have different entry points, but one or more exit points are the same)
You may use some heuristic to determine where functions start by looking for the most common prolog instruction sequence:
push ebp
mov ebp, esp
Again, this may not work if functions are compiled with the frame pointer suppressed (i.e. they'd use esp instead of ebp to access their parameters on the stack, it's possible).
The compiler (e.g. MSVC++) may also pad the inter-function space with the int 3 instruction and that too can serve as a hint for an upcoming function beginning.
As for differentiating between the various calling conventions, it's perhaps the easiest to look at the symbols (of course, if you have them). MSVC++ generates different name prefixes and suffixes, e.g.:
_function - cdecl
_function#number - stdcall
#function#number - fastcall
If you cannot extract this information from the symbols, you must analyze code to see how parameters are passed to functions and whether functions or their callers remove them from the stack.
You could use the presence of enter to denote the beginning of a function, or certain code which sets up a frame.
push ebp
mov ebp, esp
sub esp, (bytes for "local" stack space)
Later you'll find the opposite code (or leave) before a call to ret:
mov esp, ebp
pop ebp
You can also use the number of bytes for local stack space to identify local variables.
Identifying thiscall, fastcall, etc, will take some analysis of the code just prior to calls which use the initial location and an evaluation of the registers used/cleaned up.
Have a look at software like windasm or ollydbg. The call and ret operations denote function calls. However code does not run sequentially and jumps can be made all over the place. call dword ptr [edx] depends on the edx register and thus you won't be able to know where it goes unless you do runtime debugging.
To recognize fastcall functions you have to look at how parameters are passed on. Fastcall will put the first two pointer sized parameters in edx and ecx registers, where stdcall will push them on the stack. See this article for an explanation.
I'm trying to put some C hooks into some code that someone else wrote in asm. I don't know much about x86 asm and I want to make sure I'm doing this safely.
So far I've got this much:
EXTERN _C_func
CALL _C_func
And that seems to work, but I'm sure that's dangerous as-is. At the very least it could be destroying EAX, and possibly more.
So I think all I need to know is: Which registers does C destroy and how do I ensure the registers are being saved properly? Is there anything else I can do to make sure I can safely insert hooks into arbitrary locations?
I'm using Visual Studio for the C functions, if that helps.
Thank you.
UPDATE:
I looked up the "paranoid" method several people suggested and it looks like this:
pushfd
pushad
call _C_func
popad
popfd
AD = (A)ll general registers
FD = (F)lags
(I'm not sure what the 'd' stands for, but it means 32-bit registers instead of 16-bit.)
But the most efficient method is illustrated in Kragen's answer below. (Assuming you don't need the preserve the flags. If you do, add the FD instructions.)
Also, be careful about allocating lots of stack variables in your C function as it could overrun the asm subroutine's stack space.
I think that about does it, thanks for you help everyone!
If the calling convention is cdecl and the signature of C_funct is simply:
void C_func(void);
Then that is perfectly "safe", however the registers EAX, ECX and EDX are "available for use inside the function", and so may be overwritten.
To protect against this you can save the registers that you care about and restore them afterwards:
push eax
push ecx
push edx
call _C_func
pop edx
pop ecx
pop eax
By convention the registers EBX, ESI, EDI, and EBP shouldn't be modified by the callee.
I believe that the flags may be modified by the callee - again if you care about preserving the value of flags then you should save them.
See the Wikipedia page on x86 calling conventions.
You need to be aware of calling conventions. See this article for a discussion on calling conventions.
If you are paranoid about registers, you can always push all registers onto the stack before calling the function (functions shouldn't return without cleaning up, except for those registers that carry return information).
This will depend on whether you are using an x86 or x86_64 platform ... the calling conventions for each, as well as the caller-save and callee-save registers (and even the available register set) are a bit different.
That being said, caller-save registers need to be pushed on the stack before you call your C-function. Callee-save registers you don't need to worry about if you are calling a C-function, but you will need to pay attention to if you call an assembly routine from a C-function.
For a x86 32-bit platform, the caller-save registers are:
EAX
EDX
ECX
Also keep in mind that the stack-pointer registers, ESP and EBP will be changed as well with cdecl calling convention, but a C-function will restore those values after the call assuming you have not done something that a C-function would not expect you to-do with those registers.
The callee-save registers are:
EBX
ESI
EDI
Finally, the return value from your C-function is in the EAX register, and the C-function arguments are pushed onto the stack before calling the function. The order in which that is done will depend on the calling convention, but for cdecl, that will be in right-to-left order, meaning the left-most argument to the C-function is pushed last on the stack before calling the function.
Should you decide to simply save all the general purpose registers on the stack before you call your C-function, and then pop them back off the stack after the C-function, you can do that on x86 using the PUSHAD and POPAD instructions. These instructions are not useable on x86_64 though. And also keep in mind that if you have a return value in EAX, you will need to-do something with that (like saving it in memory), before you call POPAD.