Jumping between object code with different (and unknown) calling convention in C - c

Question

I have (mapped in memory) two object files, "A.o" and "B.o", with the same CPU Instruction Set (not necessarily Intel --it can be x86, x86_64, MIPS(32/64), ARM(32/64), PowerPC(32/64),..., but always the same in both object files).
Also, both object files are compiled with the same endianness (both little endian, or both big endian).
However (you knew there was a however, otherwise there wouldn't be any question), "A.o" and "B.o" can have a different function calling convention and, to make things worse, unknown to each other ("A.o" has not even the slightest idea about the calling convention for functions in "B.o", and vice versa).
"A.o" and "B.o" are obviously designed to call functions within their same object file, but there must be a (very) limited interface for communicating between them (otherwise, if execution starts at some function in "A.o", no function from "B.o" would ever be executed if there was no such interface).
The file where execution started (let's suppose it's "A.o") knows the addresses of all static symbols from "B.o" (the addresses of all functions and all global variables). But the opposite is not true (well, the limited interface I'm trying to write would overcome that, but "B.o" doesn't know any address from "A.o" before such interface is established).
Finally the question: How can execution jump from a function in "A.o" to a function in "B.o", and back, while also communicating some data?
I need it to:
Be done in standard C (no assembly).
Be portable C (not compiler-dependent, nor CPU-dependent).
Be thread safe.
Don't make any assumption about the calling conventions involved.
Be able to communicate data between the two object files.
My best idea, for the moment, seems that can meet all these requirements, except thread safety. For example, if I define an struct like this:
struct data_interface {
int value_in;
int value_out; };
I could write a pointer to an struct like this from "A.o" into a global variable of "B.o" (knowing in advance that such global variable in "B.o" has space enough for storing a pointer).
Then, the interface function would be a void interface(void) (I'm assuming that calling void(void) functions is safe across different calling conventions... if this is not true, then my idea wouldn't work). Calling such a function from "A.o" to "B.o" would communicate the data to the code in "B.o". And, fingers crossed, when the called function in "B.o" returns, it would travel back nicely (supposing the different calling convention doesn't change the behaviour when returning from void(void) functions).
However, this is not thread safe, of course.
For it to be thread safe, I guess my only option is to access the stack.
But... can the stack be accessed in a portable way in standard C?

Answer 1

Here are two suggestions.
Data interface
This elaborates on the struct you defined yourself. From what I've seen in the past, compilers typically use a single register (e.g. eax) for their return value (provided the return type fits in a register). My guess is, the following function prototype is likely to be unaffected by differing calling conventions.
struct data_interface *get_empty_data_interface(void);
If so, then you could use that in a way that is similar to the idea you already had about using arrays. Define the following struct and functions in B:
struct data_interface {
int ready;
int the_real_data;
};
struct data_interface *get_empty_data_interface(void)
{
struct data_interface *ptr = malloc(sizeof(struct data_interface));
add_to_list_of_data_block_pointers(ptr);
ptr->ready = 0;
return ptr;
}
void the_function(void)
{
execute_functionality_for_every_data_block_in_my_list_that_is_flagged_ready_and_remove_from_list();
}
To call the function, do this in A:
struct data_interface *ptr = get_empty_data_interface();
ptr->the_real_data = 12345;
ptr->ready = 1;
the_function();
For thread-safety, make sure the list of data blocks maintained by B is thread-safe.
Simultaneous calls to get_empty_data_interface should not overwrite each other's slot in the list.
Simultaneous calls to the_function should not both pick up the same list element.
Wrapper functions
You could try to expose wrapper functions with a well-known calling convention (e.g. cdecl); if necessary defined in a separate object file that is aware of the calling convention of the functions it wraps.
Unfortunately you will probably need non-portable function attributes for this.
You may be able to cheat your way out of it by declaring variadic wrapper functions (with an ellipsis parameter, like printf has); compilers are likely to fall back on cdecl for those. This eliminates non-portable function attributes, but it may be unreliable; you would have to verify my assumption for every compiler you'd like to support. When testing this, keep in mind that compiler options (in particular optimizations) may well play a role. All in all, quite a dirty approach.

Answer 2

the question implies that both object files are compiled differently except for the endianness and that they are linked together into one executable.
it says that A.o knows all static symbols from B.o, but the opposite is not true.
Don't make any assumption about the calling conventions involved.
so we'll be using only void f(void) type of functions.
you'll declare int X, Y; in B.o and extern int X, Y; in A.o so before you call the functions in B.o you check the Y flag, if raised wait until it falls. when a B's function is called it raises the Y flag, read the input from X, do some calculations, write the result back in X and return.
then the calling function in A.o copies the value from X into it's own compilation unit and clears the Y flag.
...if calling a void f(void) function just makes a wild jump from one point in the code to another.
another way to do it would be to declare static int Y = 0; in B.o and omit it entirely in A.o
then when a B.o function gets called it checks if Y == 0 and if so increase Y, read X, do calculations, write X, decrease Y and return. if not so then wait to become 0 and block the calling function.
or maybe even have a static flag in every B.o function, but i don't see the point in this waste since the communication data is global in B.o

Answer 3

Remember that there are both caller saves and callee saves conventions out there, together with variations on use of registers to pass values, use or not of a frame pointer, and even (in some architectures, in some optimisation levels) the use of the delay slot in a branch to hold the first instruction of the subroutine. You are not going to be able to do this without some knowledge of the calling conventions in play, but fortunately the linker will need that anyway. Presumably there is some higher level entity that is responsible for loading those DLLs and that knows the calling conventions for both of them?
Anything you do here is going to be at best deep into implementation defined territory, if not technically undefined behaviour, and you will want to make a deep study of the linker and loader (In particular the linker must know how to resolve dynamic linkage in your unknown calling convention or you will not be able to load that shared object in a meaningful way, so you may be able to leaverage it using libbfd or such but that is outside the scope of C).
The place this sort of thing can go very wrong is if shared resources are allocated in A and freed in B (Memory springs to mind) as memory management is a usually a library based wrapper over the operating systems SBRK or similar, and these implementations of memory management are not inherently compatible in memory layout, other places you may be bitten by this include IO (see shennanigans you sometimes get when mixing printf and cout in c++ for a benign example), and locking.