Related
I have read that converting a function pointer to a data pointer and vice versa works on most platforms but is not guaranteed to work. Why is this the case? Shouldn't both be simply addresses into main memory and therefore be compatible?
An architecture doesn't have to store code and data in the same memory. With a Harvard architecture, code and data are stored in completely different memory. Most architectures are Von Neumann architectures with code and data in the same memory but C doesn't limit itself to only certain types of architectures if at all possible.
Some computers have (had) separate address spaces for code and data. On such hardware it just doesn't work.
The language is designed not only for current desktop applications, but to allow it to be implemented on a large set of hardware.
It seems like the C language committee never intended void* to be a pointer to function, they just wanted a generic pointer to objects.
The C99 Rationale says:
6.3.2.3 Pointers
C has now been implemented on a wide range of architectures. While some of these
architectures feature uniform pointers which are the size of some integer type, maximally
portable code cannot assume any necessary correspondence between different pointer types and the integer types. On some implementations, pointers can even be wider than any integer type.
The use of void* (“pointer to void”) as a generic object pointer type is an invention of the C89 Committee. Adoption of this type was stimulated by the desire to specify function prototype arguments that either quietly convert arbitrary pointers (as in fread) or complain if the argument type does not exactly match (as in strcmp). Nothing is said about pointers to functions, which may be incommensurate with object pointers and/or integers.
Note Nothing is said about pointers to functions in the last paragraph. They might be different from other pointers, and the committee is aware of that.
For those who remember MS-DOS, Windows 3.1 and older the answer is quite easy. All of these used to support several different memory models, with varying combinations of characteristics for code and data pointers.
So for instance for the Compact model (small code, large data):
sizeof(void *) > sizeof(void(*)())
and conversely in the Medium model (large code, small data):
sizeof(void *) < sizeof(void(*)())
In this case you didn't have separate storage for code and date but still couldn't convert between the two pointers (short of using non-standard __near and __far modifiers).
Additionally there's no guarantee that even if the pointers are the same size, that they point to the same thing - in the DOS Small memory model, both code and data used near pointers, but they pointed to different segments. So converting a function pointer to a data pointer wouldn't give you a pointer that had any relationship to the function at all, and hence there was no use for such a conversion.
Pointers to void are supposed to be able to accommodate a pointer to any kind of data -- but not necessarily a pointer to a function. Some systems have different requirements for pointers to functions than pointers to data (e.g, there are DSPs with different addressing for data vs. code, medium model on MS-DOS used 32-bit pointers for code but only 16-bit pointers for data).
In addition to what is already said here, it is interesting to look at POSIX dlsym():
The ISO C standard does not require that pointers to functions can be cast back and forth to pointers to data. Indeed, the ISO C standard does not require that an object of type void * can hold a pointer to a function. Implementations supporting the XSI extension, however, do require that an object of type void * can hold a pointer to a function. The result of converting a pointer to a function into a pointer to another data type (except void *) is still undefined, however. Note that compilers conforming to the ISO C standard are required to generate a warning if a conversion from a void * pointer to a function pointer is attempted as in:
fptr = (int (*)(int))dlsym(handle, "my_function");
Due to the problem noted here, a future version may either add a new function to return function pointers, or the current interface may be deprecated in favor of two new functions: one that returns data pointers and the other that returns function pointers.
C++11 has a solution to the long-standing mismatch between C/C++ and POSIX with regard to dlsym(). One can use reinterpret_cast to convert a function pointer to/from a data pointer so long as the implementation supports this feature.
From the standard, 5.2.10 para. 8, "converting a function pointer to an object pointer type or vice versa is conditionally-supported." 1.3.5 defines "conditionally-supported" as a "program construct that an implementation is not required to support".
Depending on the target architecture, code and data may be stored in fundamentally incompatible, physically distinct areas of memory.
undefined doesn't necessarily mean not allowed, it can mean that the compiler implementor has more freedom to do it how they want.
For instance it may not be possible on some architectures - undefined allows them to still have a conforming 'C' library even if you can't do this.
Another solution:
Assuming POSIX guarantees function and data pointers to have the same size and representation (I can't find the text for this, but the example OP cited suggests they at least intended to make this requirement), the following should work:
double (*cosine)(double);
void *tmp;
handle = dlopen("libm.so", RTLD_LAZY);
tmp = dlsym(handle, "cos");
memcpy(&cosine, &tmp, sizeof cosine);
This avoids violating the aliasing rules by going through the char [] representation, which is allowed to alias all types.
Yet another approach:
union {
double (*fptr)(double);
void *dptr;
} u;
u.dptr = dlsym(handle, "cos");
cosine = u.fptr;
But I would recommend the memcpy approach if you want absolutely 100% correct C.
They can be different types with different space requirements. Assigning to one can irreversibly slice the value of the pointer so that assigning back results in something different.
I believe they can be different types because the standard doesn't want to limit possible implementations that save space when it's not needed or when the size could cause the CPU to have to do extra crap to use it, etc...
The only truly portable solution is not to use dlsym for functions, and instead use dlsym to obtain a pointer to data that contains function pointers. For example, in your library:
struct module foo_module = {
.create = create_func,
.destroy = destroy_func,
.write = write_func,
/* ... */
};
and then in your application:
struct module *foo = dlsym(handle, "foo_module");
foo->create(/*...*/);
/* ... */
Incidentally, this is good design practice anyway, and makes it easy to support both dynamic loading via dlopen and static linking all modules on systems that don't support dynamic linking, or where the user/system integrator does not want to use dynamic linking.
A modern example of where function pointers can differ in size from data pointers: C++ class member function pointers
Directly quoted from https://blogs.msdn.microsoft.com/oldnewthing/20040209-00/?p=40713/
class Base1 { int b1; void Base1Method(); };
class Base2 { int b2; void Base2Method(); };
class Derived : public Base1, Base2 { int d; void DerivedMethod(); };
There are now two possible this pointers.
A pointer to a member function of Base1 can be used as a pointer to a
member function of Derived, since they both use the same this
pointer. But a pointer to a member function of Base2 cannot be used
as-is as a pointer to a member function of Derived, since the this
pointer needs to be adjusted.
There are many ways of solving this. Here's how the Visual Studio
compiler decides to handle it:
A pointer to a member function of a multiply-inherited class is really
a structure.
[Address of function]
[Adjustor]
The size of a pointer-to-member-function of a class that uses multiple inheritance is the size of a pointer plus the size of a size_t.
tl;dr: When using multiple inheritance, a pointer to a member function may (depending on compiler, version, architecture, etc) actually be stored as
struct {
void * func;
size_t offset;
}
which is obviously larger than a void *.
On most architectures, pointers to all normal data types have the same representation, so casting between data pointer types is a no-op.
However, it's conceivable that function pointers might require a different representation, perhaps they're larger than other pointers. If void* could hold function pointers, this would mean that void*'s representation would have to be the larger size. And all casts of data pointers to/from void* would have to perform this extra copy.
As someone mentioned, if you need this you can achieve it using a union. But most uses of void* are just for data, so it would be onerous to increase all their memory use just in case a function pointer needs to be stored.
I know that this hasn't been commented on since 2012, but I thought it would be useful to add that I do know an architecture that has very incompatible pointers for data and functions since a call on that architecture checks privilege and carries extra information. No amount of casting will help. It's The Mill.
With the struct:
struct stree {
union {
void *ob;
struct stree *strees[256];
};
};
Do the C standards (particularly C11) guarantee that ob is an alias of stree[0]?
Maybe. The C standard only guarantees the following, 6.3.2.3:
A pointer to void may be converted to or from a pointer to any object type. A pointer to
any object type may be converted to a pointer to void and back again; the result shall
compare equal to the original pointer.
In theory/formally, a void* doesn't need to have the same memory representation as a pointer to object. Whenever it encounters such a conversion, it could in theory do some tricks like keeping track of the original value, to restore it later.
In practice, void* and object pointers have the same size and format on any sane system, because that's by far the easiest way to sate the above requirement. Systems with extended addressing modes for objects exist, but they don't work as the C standard intended. Instead they use non-standard extensions by inventing near and far pointer qualifiers, which could then be applied either to void* or an object pointer. (Some examples of this are low-end microcontrollers and old MS DOS.)
Bottom line: designing for portability to insane or fictional systems is a huge waste of time. Toss in a static_assert(sizeof(void*) == sizeof(int*), "...") and that should be enough. Because in practice you're not going to find a system with magical or mysterious void* format. If you do, deal with it then.
What you have to keep in mind though, is that void* in itself is a different type, so you can't generally access the memory where the void* is stored through another pointer type (strict pointer aliasing). But there are a few exceptions to the strict aliasing rule, type punning through unions is one of them.
So regarding your union, the void* and the first object pointer are guaranteed to be allocated starting at the same address, but again their respective sizes an internal formats could in theory be different. In practice you can assume that they are the same, and because you use a union you aren't breaking strict aliasing.
As for best practice, it is to avoid void* in the first place, as there's very few places where you actually need them. For the few cases where you need to pass some generic pointer around, it is often better to use uintptr_t and store the result as an integer.
In an old program I serialized a data structure to bytes, by allocating an array of unsigned char, and then converted ints by:
*((*int)p) = value;
(where p is the unsigned char*, and value is the value to be stored).
This worked fine, except when compiled on Sparc where it triggered exceptions due to accessing memory with improper alignment. Which made perfect sense because the data elements had varying sizes so p quickly became unaligned, and triggered the error when used to store an int value, where the underlying Sparc instructions require alignment.
This was quickly fixed (by writing out the value to the char-array byte-by-byte). But I'm a bit concerned about this because I've used this construction in many programs over the years without issue. But clearly I'm violating some C rule (strict aliasing?) and whereas this case was easily discovered, maybe the violations can cause other types of undefined behavior that is more subtle due to optimizing compilers etc. I'm also a bit puzzled because I believe I've seen constructions like this in lot of C code over the years. I'm thinking of hardware drivers that describe the data-structure exchanged by the hardware as structs (using pack(1) of course), and writing those to h/w registers etc. So it seems to be a common technique.
So my question is, is exactly what rule was violated by the above, and what would be the proper C way to realize the use-case (i.e. serializing data to an array of unsigned char). Of course custom serialization functions can be written for all functions to write it out byte-by-byte but it sounds cumbersome and not very efficient.
Finally, can ill effects (outside of alignment problems etc.) in general be expected through violation of this aliasing rule?
Yes, your code violates strict aliasing rule. In C, only char* and its signed and unsigned counterparts are assumed to alias other types.
So, the proper way to do such raw serialization is to create an array on ints, and then treat it as unsigned char buffer.
int arr[] = { 1, 2, 3, 4, 5 };
unsigned char* rawData = (unsigned char*)arr;
You can memcpy, fwrite, or do other serialization of rawData, and it is absolutely valid.
Deserialization code may look like this:
int* arr = (int*)calloc(5, sizeof(int));
memcpy(arr, rawData, 5 * sizeof(int));
Sure, you should care of endianness, padding and other issues to implement reliable serialization.
It is compiler and platform specific, on how a struct is represented (layed out) in memory and whether or not the start address of a struct is aligned to a 1,2,4,8,... byte boundary. Therefore, you should not take any assumptions on the layout of your structs members.
On platforms, where your member types require specific alignment, padding bytes are added to the struct (which equals the statement I made above, that sizeof(struct Foo) >= the sum of its data member sizes). The padding...
Now, if you fwrite() or memcpy() a struct from one instance to another, on the same machine with the same compiler and settings (e.g. in the same program of yours), you will write both the data content and the padding bytes, added by the compiler. As long as you handle the whole struct, you can successfully round trip (as long as there are no pointer members inside the struct, at least).
What you cannot assume is, that you can cast smaller types (e.g. unsigned char ) to "larger types" (e.g. unsigned int) and memcpy between those in that direction, because unsigned int might require proper alignment on that target platform. Usually if you do that wrong, you see bus errors or alike.
malloc() in the most general case is the generic way to get heap-memory for any type of data. Be it a byte array or some struct, independent of its alignment requirements. There is no system existing, where you cannot struct Foo *ps = malloc(sizeof(struct Foo)). On platforms, where alignment is vital, malloc will not return unaligned addresses as it would break any code, trying to allocate memory for a struct. As malloc() is not psychic, it will also return "struct compatible aligned" pointers if you use it to allocate byte arrays.
Any form of "ad hoc" serialization like writing the whole struct is only a promising approach as long as you need not exchange the serialized data with other machines or other applications (or future versions of the same application where someone might have tinkered with compiler settings, related to alignment).
If you look for a portable and more reliable and robust solution, you should consider using one of the main stream serialization packages, one of which being the aforementioned Google protocol buffers.
I have read that converting a function pointer to a data pointer and vice versa works on most platforms but is not guaranteed to work. Why is this the case? Shouldn't both be simply addresses into main memory and therefore be compatible?
An architecture doesn't have to store code and data in the same memory. With a Harvard architecture, code and data are stored in completely different memory. Most architectures are Von Neumann architectures with code and data in the same memory but C doesn't limit itself to only certain types of architectures if at all possible.
Some computers have (had) separate address spaces for code and data. On such hardware it just doesn't work.
The language is designed not only for current desktop applications, but to allow it to be implemented on a large set of hardware.
It seems like the C language committee never intended void* to be a pointer to function, they just wanted a generic pointer to objects.
The C99 Rationale says:
6.3.2.3 Pointers
C has now been implemented on a wide range of architectures. While some of these
architectures feature uniform pointers which are the size of some integer type, maximally
portable code cannot assume any necessary correspondence between different pointer types and the integer types. On some implementations, pointers can even be wider than any integer type.
The use of void* (“pointer to void”) as a generic object pointer type is an invention of the C89 Committee. Adoption of this type was stimulated by the desire to specify function prototype arguments that either quietly convert arbitrary pointers (as in fread) or complain if the argument type does not exactly match (as in strcmp). Nothing is said about pointers to functions, which may be incommensurate with object pointers and/or integers.
Note Nothing is said about pointers to functions in the last paragraph. They might be different from other pointers, and the committee is aware of that.
For those who remember MS-DOS, Windows 3.1 and older the answer is quite easy. All of these used to support several different memory models, with varying combinations of characteristics for code and data pointers.
So for instance for the Compact model (small code, large data):
sizeof(void *) > sizeof(void(*)())
and conversely in the Medium model (large code, small data):
sizeof(void *) < sizeof(void(*)())
In this case you didn't have separate storage for code and date but still couldn't convert between the two pointers (short of using non-standard __near and __far modifiers).
Additionally there's no guarantee that even if the pointers are the same size, that they point to the same thing - in the DOS Small memory model, both code and data used near pointers, but they pointed to different segments. So converting a function pointer to a data pointer wouldn't give you a pointer that had any relationship to the function at all, and hence there was no use for such a conversion.
Pointers to void are supposed to be able to accommodate a pointer to any kind of data -- but not necessarily a pointer to a function. Some systems have different requirements for pointers to functions than pointers to data (e.g, there are DSPs with different addressing for data vs. code, medium model on MS-DOS used 32-bit pointers for code but only 16-bit pointers for data).
In addition to what is already said here, it is interesting to look at POSIX dlsym():
The ISO C standard does not require that pointers to functions can be cast back and forth to pointers to data. Indeed, the ISO C standard does not require that an object of type void * can hold a pointer to a function. Implementations supporting the XSI extension, however, do require that an object of type void * can hold a pointer to a function. The result of converting a pointer to a function into a pointer to another data type (except void *) is still undefined, however. Note that compilers conforming to the ISO C standard are required to generate a warning if a conversion from a void * pointer to a function pointer is attempted as in:
fptr = (int (*)(int))dlsym(handle, "my_function");
Due to the problem noted here, a future version may either add a new function to return function pointers, or the current interface may be deprecated in favor of two new functions: one that returns data pointers and the other that returns function pointers.
C++11 has a solution to the long-standing mismatch between C/C++ and POSIX with regard to dlsym(). One can use reinterpret_cast to convert a function pointer to/from a data pointer so long as the implementation supports this feature.
From the standard, 5.2.10 para. 8, "converting a function pointer to an object pointer type or vice versa is conditionally-supported." 1.3.5 defines "conditionally-supported" as a "program construct that an implementation is not required to support".
Depending on the target architecture, code and data may be stored in fundamentally incompatible, physically distinct areas of memory.
undefined doesn't necessarily mean not allowed, it can mean that the compiler implementor has more freedom to do it how they want.
For instance it may not be possible on some architectures - undefined allows them to still have a conforming 'C' library even if you can't do this.
Another solution:
Assuming POSIX guarantees function and data pointers to have the same size and representation (I can't find the text for this, but the example OP cited suggests they at least intended to make this requirement), the following should work:
double (*cosine)(double);
void *tmp;
handle = dlopen("libm.so", RTLD_LAZY);
tmp = dlsym(handle, "cos");
memcpy(&cosine, &tmp, sizeof cosine);
This avoids violating the aliasing rules by going through the char [] representation, which is allowed to alias all types.
Yet another approach:
union {
double (*fptr)(double);
void *dptr;
} u;
u.dptr = dlsym(handle, "cos");
cosine = u.fptr;
But I would recommend the memcpy approach if you want absolutely 100% correct C.
They can be different types with different space requirements. Assigning to one can irreversibly slice the value of the pointer so that assigning back results in something different.
I believe they can be different types because the standard doesn't want to limit possible implementations that save space when it's not needed or when the size could cause the CPU to have to do extra crap to use it, etc...
The only truly portable solution is not to use dlsym for functions, and instead use dlsym to obtain a pointer to data that contains function pointers. For example, in your library:
struct module foo_module = {
.create = create_func,
.destroy = destroy_func,
.write = write_func,
/* ... */
};
and then in your application:
struct module *foo = dlsym(handle, "foo_module");
foo->create(/*...*/);
/* ... */
Incidentally, this is good design practice anyway, and makes it easy to support both dynamic loading via dlopen and static linking all modules on systems that don't support dynamic linking, or where the user/system integrator does not want to use dynamic linking.
A modern example of where function pointers can differ in size from data pointers: C++ class member function pointers
Directly quoted from https://blogs.msdn.microsoft.com/oldnewthing/20040209-00/?p=40713/
class Base1 { int b1; void Base1Method(); };
class Base2 { int b2; void Base2Method(); };
class Derived : public Base1, Base2 { int d; void DerivedMethod(); };
There are now two possible this pointers.
A pointer to a member function of Base1 can be used as a pointer to a
member function of Derived, since they both use the same this
pointer. But a pointer to a member function of Base2 cannot be used
as-is as a pointer to a member function of Derived, since the this
pointer needs to be adjusted.
There are many ways of solving this. Here's how the Visual Studio
compiler decides to handle it:
A pointer to a member function of a multiply-inherited class is really
a structure.
[Address of function]
[Adjustor]
The size of a pointer-to-member-function of a class that uses multiple inheritance is the size of a pointer plus the size of a size_t.
tl;dr: When using multiple inheritance, a pointer to a member function may (depending on compiler, version, architecture, etc) actually be stored as
struct {
void * func;
size_t offset;
}
which is obviously larger than a void *.
On most architectures, pointers to all normal data types have the same representation, so casting between data pointer types is a no-op.
However, it's conceivable that function pointers might require a different representation, perhaps they're larger than other pointers. If void* could hold function pointers, this would mean that void*'s representation would have to be the larger size. And all casts of data pointers to/from void* would have to perform this extra copy.
As someone mentioned, if you need this you can achieve it using a union. But most uses of void* are just for data, so it would be onerous to increase all their memory use just in case a function pointer needs to be stored.
I know that this hasn't been commented on since 2012, but I thought it would be useful to add that I do know an architecture that has very incompatible pointers for data and functions since a call on that architecture checks privilege and carries extra information. No amount of casting will help. It's The Mill.
Rather than sending an actual pointer to a value, the value is cast to a pointer. I found these examples in the GUI interface code of a GTK program.
g_signal_connect (pastebutton[pane],
"clicked",
G_CALLBACK(on_paste_button_pressed),
(void*)((long)pane<<4));
In the above example, I am referring to the last parameter of g_signal_connect. When on_paste_button_pressed is called by GTK2, on_paste_button_pressed casts the user_data void pointer back like so:
int pane = ((long)user_data) >> 4;
Actually, I added this particular example to the code, but I based it upon what was already there. I added the bit-shifting so as to avoid warnings about casting. The program itself has four panes containing quite a number of widgets, the copy and paste buttons allow you to copy all the values from one pane to another.
Is this way of casting a value to a pointer address often used, and are there reasons why this should not be used?
edit:
The cast from an integer to a void pointer can also be achieved like so:
void* void_ptr = some_int - NULL;
It is used. It is used quite commonly when it does what's required.
One reason not to use it is that theoretically pointer size might be smaller than the source integer size.
Another reason not to use it is that it allows you to pass only one piece of integer data to the callback. If in the future you'll need to add another piece of data (or switch to a non-integer), you'll have to locate and rewrite every single place where the callback makes access to the passed data. So, if there's a chance that you'd have to extend the data in the future, it is better to create a struct (even if it holds just a single int at this time) and pass a pointer to that struct.
But if you are sure that you'll never have to pass anything other than that single integer and that your integer fits into a void *, then this technique is not in any way broken.
P.S. Pedantically speaking, neither C nor C++ appear to have the roundtrip guarantee for integer-to-void *-to-integer conversion, i.e. they don't guarantee that it will work and restore the original integral value.
You should use macros GINT_TO_POINTER() and GPOINTER_TO_INT() to cast between pointers and integers.
glib: Type Conversion Macros
Casting an integer to a pointer is used to pass a value by-value. This is the preferred way if you do need a by-reference parameter because the compiler then does not need to dereference the pointer.
The bit-shifting is a bad idea because it can cause overflows.
For really portable code, you should use intptr_t as your integer type cause it will nicely fit into a pointer.
It does see use in these kinds of cases, yes. It works on many platforms, but might fail because an arbitrary integer is not always a valid pointer value, though the shifting you do should get around that. It is also possible that a pointer cannot hold all the values that an integer can; this would be the case if you're using a 64-bit long on a platform where pointers are 32 bits, and of course since you are shifting the value, it could also fail even if pointers and integers are the same size. If you want to use this trick, you should probably check sizeof (void*) against the size of the integer type you use, and at runtime check against the actual value if the pointer isn't big enough. It's probably not worth it to make this fully portable so that you would use an actual pointer on platforms where that's needed, so either limit yourself to platforms where this trick works or abandon the trick altogether.
I find it just fine. Don't have statistics about use frequency, but do you really care?
I'm not sure I understand how bit-shifting helps, though.
It is used but it's by no means portable so you have to be careful.
The C standard does not mandate that pointer types have at least as many bits as integer types, so you may not always be able to do it.
But I can't recall any platform in which pointers have actually been smaller than integers so you're probably safe (despite not being technically safe).
The only reason I can think that the casting may be there for is to remove the possibility of alignment warnings. Section 6.3.2.3 of the C1x draft states:
An integer may be converted to any pointer type. Except as previously specified, the
result is implementation-defined, might not be correctly aligned, might not point to an
entity of the referenced type, and might be a trap representation.
This is from the C-FAQ:
Q: How are integers converted to and from pointers? Can I temporarily stuff an integer into a pointer, or vice versa?
A: Once upon a time, it was guaranteed that a pointer could be converted to an integer (though one never knew whether an int or a long might be required), and that an integer could be converted to a pointer, and that a pointer remained unchanged when converted to a (large enough) integer and back again, and that the conversions (and any mapping) were intended to be ``unsurprising to those who know the addressing structure of the machine.'' In other words, there is some precedent and support for integer/pointer conversions, but they have always been machine dependent, and hence nonportable. Explicit casts have always been required (though early compilers rarely complained if you left them out).
The ANSI/ISO C Standard, in order to ensure that C is widely implementable, has weakened those earlier guarantees. Pointer-to-integer and integer-to-pointer conversions are implementation-defined (see question 11.33), and there is no longer any guarantee that pointers can be converted to integers and back, without change.
Forcing pointers into integers, or integers into pointers, has never been good practice. When you need a generic slot that can hold either kind of data, a union is a much better idea.
See also questions 4.15, 5.18, and 19.25.
References: K&R1 Sec. A14.4 p. 210
K&R2 Sec. A6.6 p. 199
ISO Sec. 6.3.4
Rationale Sec. 3.3.4
H&S Sec. 6.2.3 p. 170, Sec. 6.2.7 pp. 171-2