How exactly does RunDll32 call a function, without knowing the number/types of arguments that the function can take?
Does it have a built-in compiler or something of the sort?
RunDll32 is pretty much a thin wrapper that calls LoadLibrary to load the given DLL, calls GetProcAddress to get the function address of the desired function, and then calls the function.
It can't call just any exported function in the DLL, though—it assumes that the function has a very specific function signature of the following:
void CALLBACK
EntryPoint(HWND hwnd, HINSTANCE hinst, LPSTR lpszCmdLine, int nCmdShow);
where CALLBACK is a macro that expands to the __stdcall calling convention. See this knowledge base article for a more detailed description.
If your DLL's function does not have the correct signature or calling convention, lots of badness will ensue. See What can go wrong when you mismatch the calling convention? for lots of gory details. Fortunately (or perhaps unfortunately), RunDll32 is written in such a way to ameliorate those types of errors, but that still doesn't mean it's a good idea. Do not use RunDll32 to call functions that do not have the correct signature. It's just a ticking time bomb waiting to go off in the next version of Windows.
It can't call just any function, it can only call function specifically written to be called. Hence, there is no magic.
Related
From what I understand, the caller and the callee both need to have the same calling convention. Otherwise, the stack might be corrupted.
WinMain is declared with __stdcall and calls all the functions I've defined. Does this mean all the functions I define should use the stdcall calling convention?
I've tried not using __stdcall and nothing bad happened. I have also seen well-known GUI libraries supporting Windows don't use stdcall. Why is the stack not corrupting?
WinMain is declared with __stdcall and calls all the functions I've defined. Does this mean all the functions I define should use the stdcall calling convention?
No. Calling conventions are handled on a per-function-call basis, right at the call site. The convention dictates how the caller and callee manage the call stack - how parameters are passed, in what order, who cleans up the stack, etc. As long as the caller and callee agree to use the same calling convention on each individual function call, it is perfectly safe for a stdcall function to call a function that uses a different convention, like cdecl, and vice versa. A function's calling convention applies only when:
the function is being entered by a new caller.
the function is returning back to that caller.
the function is accessing its own parameters.
Outside of that, what a function does internally has nothing to with its own calling convention.
For example, lets say that WinMain(), a stdcall function, wants to call a cdecl function.
It does not matter at all that WinMain() is itself a stdcall function. While code execution is inside of WinMain(), it can do whatever it wants. WinMain()'s stdcall convention is applied only upon entry and exit of WinMain() itself. That is the contract WinMain() has with ITS caller.
What matters is that WinMain() must follow the rules of cdecl when setting up the call stack for a cdecl function that it is about to call into, and cleaning up the call stack when that function returns back to WinMain().
The same goes for any function call of any calling convention.
I've tried not using __stdcall and nothing bad happened. I have also seen well-known GUI libraries supporting Windows don't use stdcall. Why is the stack not corrupting?
Because the call stack is being managed correctly at every function call and return, so there is no unbalanced cleanup to corrupt the stack.
Using Pin, I would like to call some instrumentation function before and after each application function call. I've read that RTN_InsertCall to add some entry/exit analysis functions with IPOINT_BEFORE and IPOINT_AFTER is unreliable as the exit may never be called.
My understanding is that the 'correct' way to do this is to replace the routine via RTN_ReplaceSignature, then in my replacement function add the entry and exit calls around a call to the original routine, where the original routine is called using PIN_CallApplicationFunction.
However, as far as I can tell PIN_CallApplicationFunction requires that I state in advance all of the arguments for the routine that I am wrapping, e.g., for malloc I would need to explicitly pass in some size_t argument, whereas for free I would pass in a pointer, and so on.
As I just want to wrap all function calls, I don't know the arguments! Is there some way to simply jump into the original function that I replaced, passing along the arguments for the original signature? Or perhaps some better way to do this?
Thanks for any help!
The problem with IPOINT_BEFORE, IPOINT_AFTER is that IPOINT_AFTER may miss some ret instructions. RTN_Replace functions will require a function pointer having the same signature as original (as you do not want to modify the default code).
A simple solution could be instrument all 'call' and 'ret' instructions. use INS_Rtn function to find out the routine name. This way you can instrument all functions without bothering for each function signature.
I'm trying to make a call to a DLL function (via GetProcAddress etc) from C, using lcc compiler. The function gets called and everything goes well, but it looks like the top of the stack gets corrupted. I've tried to play with calling conventions (__stdcall / __cdecl), but that didn't help.
Unfortunately I don't have access to the dll code, and have to use the lcc compiler.
I found that this simple hack avoids stack corruption:
void foo(params)
{
int dummy;
dll_foo(params);
}
Here dll_foo is the pointer returned by GetProcAddress, and the stack is kind of protected by the dummy variable. So it's not the stack pointer that gets corrupted, but the data at the top of the stack. It works like this, but I'd like to know the reason of the corruption.
Any ideas?
UPD:
As asked in the comments, here are the actual function types:
typedef unsigned char (CALLBACK Tfr)(unsigned char);
typedef void (CALLBACK Tfw)(unsigned char,unsigned char);
typedef int (CALLBACK Tfs)(int);
typedef void (CALLBACK Tfwf)(int*,int);
All they show a similar behavior.
Unfortunately, it is not so straightforward to attach a debugger, as the code is compiled and launched by Matlab, using the LCC compiler, and there is no debugging support. Probably I will have to reproduce this problem in a standalone configuration, but it is not that easy to make it.
Sounds like you use MSVC, Debug + Windows + Registers. Look at the value of ESP before and after the call. If it doesn't match then first change the calling convention in the function pointer declaration (did you do that right?) If it still doesn't match then it is __stdcall and you haven't guessed the arguments you need to pass correctly.
Or the function could just clobbers the stack frame, it isn't impossible.
Posting your function pointer declaration that shows the real arguments would probably help diagnose this better.
It sounds to me like you were on the right track with looking at the calling convention. The main thing you need to do is ensure that the caller and callee are both using the same convention. Typically for a DLL, you want to use __stdcall for both, but if (as you say) you have no control over the DLL, then you need to modify your code to match what it's doing. Unfortunately, it's almost impossible to guess what that is -- I'm pretty sure lcc (like most C and C++ compilers) can produce code to use a variety of conventions.
Based on your hack working by putting an extra dword on the stack, it sounds like you currently have a mismatch where both the caller and the callee are trying to clear arguments off the stack (e.g., the caller using __cdecl and the callee using __stdcall.
You could try to "follow" the call to dll_foo() i assembler using a debugger, at check out exactly what the routine does stack-wise.
I was looking through some code from the SDL library and came across a function declared like this:
LRESULT CALLBACK WndProc(HWND hwnd, UINT msg, WPARAM wParam, LPARAM lParam)
Now, I'm a Delphi coder. No hablo C muy bien, senor. But I remember enough syntax from my college courses to read it like this:
Function name is WndProc. Argument list is pretty self-explanatory. Function return type is LRESULT. But what in the world is that "CALLBACK" doing there? In Delphi, any function can be used as a callback; you just need to pass the right type of function pointer. Is there any particular reason why C doesn't work that way? Or does it mean something different?
The "CALLBACK" is a calling convention. There are other kinds of calling conventions. CALLBACK is the same as __stdcall.
http://www.codeguru.com/cpp/cpp/cpp_mfc/callbacks/article.php/c10557
Some more information at Raymond Chen's blog:
https://devblogs.microsoft.com/oldnewthing/20040108-00/?p=41163
Short roundup from Raymond Chen's Blog:
The great thing about calling
conventions on the x86 platform is
that there are so many to choose from!
C calling convention (__cdecl)
The C calling convention is constrained because it allows the use of functions with a variable number of parameters. It pretty much requires that the stack be caller-cleaned and that the parameters be pushed right to left, so that the first parameter is at a fixed position relative to the top of the stack. In summary: Caller cleans the stack, parameters pushed right to left.
Pascal calling convention (__pascal)
Pascal does not support functions with a variable number of parameters, so it can use the callee-clean convention. Parameters are pushed from left to right. Nearly all Win16 functions are exported as Pascal calling convention. The callee-clean convention saves three bytes at each call point, with a fixed overhead of two bytes per function. It was also fractionally faster. On Win16, saving a few hundred bytes and a few cycles was a big deal. Note: The Fortran calling convention (__fortran) is the same as the Pascal calling convention
It's the calling convention. It's required when you pass a pointer to this function to a Windows API which later calls that function. The Windows calling convention is different from the C calling convention, therefore you need to specify to the compiler that WndProc() is special, and that it needs different startup and cleanup code.
It's a calling convention, and Delphi has them too. Try looking up 'cdecl' in the Delphi Help. In Delphi (or Object Pascal as we oldies like to call it) calling conventions come after the function declaration, like this;
function MyFunction(X, Y: Real): Real; cdecl;
I'm learning about Win32 programming, and the WinMain prototype looks like:
int WINAPI WinMain ( HINSTANCE instance, HINSTANCE prev_instance, PSTR cmd_line, int cmd_show )
I was confused as to what this WINAPI identifier was for and found:
#define WINAPI __stdcall
What does this do? I'm confused by this having something at all after a return type. What is __stdcall for? What does it mean when there is something between the return type and function name?
__stdcall is the calling convention used for the function. This tells the compiler the rules that apply for setting up the stack, pushing arguments and getting a return value.
There are a number of other calling conventions, __cdecl, __thiscall, __fastcall and the wonderfully named __declspec(naked). __stdcall is the standard calling convention for Win32 system calls.
Wikipedia covers the details.
It primarily matters when you are calling a function outside of your code (e.g. an OS API) or the OS is calling you (as is the case here with WinMain). If the compiler doesn't know the correct calling convention then you will likely get very strange crashes as the stack will not be managed correctly.
C or C++ itself do not define those identifiers. They are compiler extensions and stand for certain calling conventions. That determines where to put arguments, in what order, where the called function will find the return address, and so on. For example, __fastcall means that arguments of functions are passed over registers.
The Wikipedia Article provides an overview of the different calling conventions found out there.
The answers so far have covered the details, but if you don't intend to drop down to assembly, then all you have to know is that both the caller and the callee must use the same calling convention, otherwise you'll get bugs that are hard to find.
I agree that all the answers so far are correct, but here is the reason. Microsoft's C and C++ compilers provide various calling conventions for (intended) speed of function calls within an application's C and C++ functions. In each case, the caller and callee must agree on which calling convention to use. Now, Windows itself provides functions (APIs), and those have already been compiled, so when you call them you must conform to them. Any calls to Windows APIs, and callbacks from Windows APIs, must use the __stdcall convention.
Have a look at:
http://www.codeproject.com/KB/cpp/calling_conventions_demystified.aspx
It has to do with how the function is called- basically the order in which things are put on the the stack and who is responsible for cleanup.
Here's the documentation, but it doesn't mean much unless you understand the first part:
http://msdn.microsoft.com/en-us/library/zxk0tw93.aspx
__stdcall is used to put the function arguments in the stack.
After the completion of the function it automatically deallocates the memory.
This is used for fixed arguments.
void __stdcall fnname ( int, int* )
{
...
}
int main()
{
CreateThread ( NULL, 0, fnname, int, int*...... )
}
Here the fnname has args it directly push into the stack.
I never had to use this before until today. Its because in my code I am using multi-threadding and the multi-threading API I am using is the windows one (_beginthreadex).
To start the thread:
_beginthreadex(NULL, 0, ExecuteCommand, currCommand, 0, 0);
The ExecuteCommand function MUST use the __stdcall keyword in the method signature in order for beginthreadex to call it:
unsigned int __stdcall Scene::ExecuteCommand(void* command)
{
return system(static_cast<char*>(command));
}