Prerequisites:
As per C standard, pointer arithmetics that would yield an invalid pointer, cause undefined behavior.
Linux source code seems to conform with C standard in a desire to be compatible with most architectures.
Linux's list implementation contains the following code(formatting preserved, probably the idea for another question is how to set proper tabulation width using Stackoverflow syntax):
#define list_entry(ptr, type, member) \
container_of(ptr, type, member)
#define list_next_entry(pos, member) \
list_entry((pos)->member.next, typeof(*(pos)), member)
#define list_first_entry(ptr, type, member) \
list_entry((ptr)->next, type, member)
#define list_entry_is_head(pos, head, member) \
(&pos->member == (head))
#define list_for_each_entry(pos, head, member) \
for (pos = list_first_entry(head, typeof(*pos), member); \
!list_entry_is_head(pos, head, member); \
pos = list_next_entry(pos, member))
Typical usecase of the aforementioned list implementation is having structure of say type struct A, containing a head for the list of stuctures of type struct B.
Q: Let's assume offsetof(struct B, entry_in_list) > offsetof(struct A, list_head) and the following loop is implemented:
struct A* A_ptr = something_meaningful;
struct B* pos = NULL;
list_for_each_entry(pos, &A_ptr->list_head, entry_in_list) {
do_something();
}
Then last (before loop exit) evaluation of list_next_entry(pos, member) would extend to:
container_of(A_ptr->list_head, struct B, entry_in_list) =
= (char*)A_ptr->list_head - offsetof(struct B, entry_in_list) =
= (char*)A_ptr + offsetof(struct A, list_head) - offsetof(struct B, entry_in_list)
, which, according to our assumption, would point to area before A struct. Assuming this area does not contain allocated memory, the result of the container_of() macro would be an invalid pointer, thus causing UB(in general case OFC) in Linux. Is this reasoning plausible or am I mistaken somehow?
Or are there some parts of the standard universally considered to not be worth to follow?
As suspected by OP, the implementation of the list_for_each_entry(pos, head, member) macro depends on undefined behavior in the C language in order for the loop termination condition !list_entry_is_head(pos, head, member) to become false.
Assuming the list is non-empty, then after the final iteration, the third "advancing" expression of the for loop produces a pointer to an invalid typeof(*pos) at an address offsetof(typeof(*pos), member) bytes before the struct list_head pointed to by head. It relies on &pos->member nevertheless comparing equal to head.
Although it depends on undefined behavior, it is hard for the compiler to determine that pos is technically an invalid pointer. As long as both pos and head point within the same flat address space, the Linux kernel manages to get away with this bending of the rules.
The alternative would be for #include <linux/list.h> to not provide the list_for_each_entry(pos, head, member) macro at all, and for code to use the list_for_each(pos, head) and list_entry(ptr, type, member) macros instead (where pos is a struct list_head * and ptr is a type *), but that would typically require extra variables in the code.
Additional assertions made when compiling the kernel. These are actually used all over the place.
A pointer may be loaded with an address that isn't allocated. You see this on every system call entry. Special care must be taken handling such pointers as dereferencing them can do worse than crash.
Dereferencing a NULL pointer is not guaranteed to crash; and the compiler is not allowed to assume that a path that dereferences NULL is unreachable. (This one was added late after a NULL pointer optimization removed a security check.) On some architectures there's actually something there; on other architectures it's just another usermode pointer.
The compiler is told these are true by compiler options. (In fact the first one is generally assumed to be true in flat model, which the kernel is.)
The flag passed to gcc is -fno-delete-null-pointer-checks. Reference for null pointer optimization change: https://lwn.net/Articles/342420/
You are right that this is undefined behavior. According to Richard Biener, this kind of undefined behavior is not supported/made defined by -fno-strict-aliasing. (Clang treats this as undefined as well.)
This particular miscompilation was observed with Open vSwitch, but its list macros are clearly modeled after/copied from the kernel. Why does the kernel get away with it?
It is built with -fno-strict-aliasing (although this does not help with this particular case).
The kernel is less often built with LTO.
The kernel does not use list heads on the stack.
Compilers do not recognize the kernel allocation functions.
As a result, compilers do not observe the invalid/impossible object references and do not optimize based on that.
Related
While looking at Linux kernel's implementation of doubly linked circular lists, I've found following macro:
#define container_of(ptr, type, member) ({ \
const typeof( ((type *)0)->member ) *__mptr = (ptr); \
(type *)( (char *)__mptr - offsetof(type,member) );})
The way this works is that it returns pointer to structure given only address of one of its members:
struct blabla
{
int value;
struct list_head *list;
}
Thus you can get pointer to blabla (and get to "value") given only pointer to list.
To my question, how would I make this as portable as possible (best case conforming to C89/C99?). Due to usage of typeof(), this is gcc only.
This is what I've got so far:
#define container_of(ptr, type, member) ( \
(type *) (char *)(ptr)-offsetof(type,member)\
)
Is this snippet conforming to ISO standards (and thus should be able to be compiled on any conforming compiler)?
As Ouah commented, the ({ ... }) statement expression is a GNU extension; you won't be able to use that. Your core expression is close to what's required, but doesn't have enough parentheses:
#define container_of(ptr, type, member) \
((type *) ((char *)(ptr) - offsetof(type, member)))
That looks clean to me. It's only spread across two lines for SO.
The macro is written the way it is to perfom a type check on ptr. It's possible to use a compound literal instead of the statement expression and fall back to a simple check for pointers instead of using __typeof__ if the compiler is not gcc-compatible:
#ifdef __GNUC__
#define member_type(type, member) __typeof__ (((type *)0)->member)
#else
#define member_type(type, member) const void
#endif
#define container_of(ptr, type, member) ((type *)( \
(char *)(member_type(type, member) *){ ptr } - offsetof(type, member)))
ISO C90 compatible version with type check. (However, caveat: two evaluations of ptr!)
#define container_of(ptr, type, member) \
((type *) ((char *) (ptr) - offsetof(type, member) + \
(&((type *) 0)->member == (ptr)) * 0))
struct container {
int dummy;
int memb;
};
#include <stddef.h>
#include <stdio.h>
int main()
{
struct container c;
int *p = &c.memb;
double *q = (double *) p;
struct container *pc = container_of(p, struct container, memb);
struct container *qc = container_of(q, struct container, memb);
return 0;
}
Test:
$ gcc -Wall containerof.c
containerof.c: In function ‘main’:
containerof.c:20:26: warning: comparison of distinct pointer types lacks a cast
containerof.c:20:21: warning: unused variable ‘qc’
containerof.c:19:21: warning: unused variable ‘pc’
We get the distinct pointer types warning for 26, but not 25. That is our diagnostic about pointers being misused.
I first tried placing the type check into the left hand side of a comma operator, gcc complains about that having no effect, which is a nuisance. But by making it an operand, we ensure that it is used.
The &((type *) 0)->member trick isn't well defined by ISO C, but it's widely used for defining offsetof. If your compiler uses this null pointer trick for offsetof, it will almost certainly behave itself in your own macro.
Yes, you can make "container_of" macros to be strictly ISO C conforming. To do this you need two things:
take rid of GNU exensions;
find a way to check types compatibility.
Basically, types checking is not run time operation, but compile time rather. And I not see any reasons, why original "container_of" implementation creates new variable just to assign it and perform type checking. This can be done without creation of new variable in some expression which is only computed (and types checked) in compile time. Fortunately, we have no much options in C and only choice is to use "sizeof(expression)" to check the type. See an example:
#define container_of(ptr, type, member) \
( (void)sizeof(0 ? (ptr) : &((type *)0)->member), \
(type *)((char*)(ptr) - offsetof(type, member)) )
In first line types compatibility is checked (for ternary operator compiler must insure, that types might be converted to common type, or that both types are compatible). Second line is the same, as in original "container_of" macros.
You can play with test program on GodBolt (https://godbolt.org/z/MncvzWfYn) and make sure, that this ISO conforming variant works even in Microsoft's Visual Studio compiler.
PS: After some time, I found that the following variant can be better:
#define CONTAINER_OF(ptr, type, member) \
( (void)sizeof(0 ? (ptr) : &((type*)0)->member), \
(typeof(_Generic((typeof(ptr))0, const typeof(*(typeof(ptr))0)*: (const type*)0, default: (type*)0))) \
((uintptr_t)(const void*)(ptr) - offsetof(type, member)) )
The difference, is that it preserves const qualifier from ptr and assigns it to the result, for example:
if ptr argument is const struct * pointer, the result then will have a type of const type *, despite if the type is const or not;
if ptr argument is non-const pointer (struct*), the result then will have type type*, which may be const or non-const, depending on the type of type argument.
As a result, preserving const qualifier reduces the possibility of the errors, when const pointer to some structure translated into non-const pointer via container_of macro.
Unfortunately, this version requires C23 or non-standard typeof() operator for earlier versions of C standard.
Another reason for ISO-compiant container_of macro, as opposed to implementation from Linux kernel, is that latter uses "statement expression" GCC-extension which works badly in a case, when argument ptr is a temporary variable. The latter might happen, when container_of macro is applied to the result of a function invocation (container_of(func().x, struct y, m), here is assumed, that func() returns a structure in which x is an array of structures), or to the compound statement (container_of((&(struct S){...}), struct B, m)). In both of these cases, a call to container_of macro borrowed from linux will result in a dangling pointer! This happens because a temporary object passed as ptr argument will be destroyed after the first semicolon (at the first line of Linux implementation of container_of macro), and because a variable created by a compound statement expression will be destroyed at the end of the nearest block, which is "statement expression" itself. ISO-compliant implementation of container_of macro have no such issues.
How to pass struct type in arguments? I guess it is not possible, still wanted to check if it is feasible.
My requirement is something like this
Below is a macro list_entry used in linux kernel
#define list_entry(ptr, type, member) \
((type *)((char *)(ptr)-(unsigned long)(&((type *)0)->member)))
I want to rewrite it using function. How do I write?
list_entry(void *ptr, ???, ???)
Is there a way we can do this?
I am trying to write a wrapper to free a given list. So to use list_entry in my function, my function needs to be passed with type and the member. Is it possible?
It's fundamentally not possible to write list_entry, with its same behavior and signature, as a function. This is because its arguments include a type and a member name, neither of which is a value in C.
On the other hand, you could to some extent abstract out the "code" content of the macro into a function:
#define list_entry(ptr, type, member) \
((type *)f_list_entry(ptr, offsetof(type, member)))
static void *f_list_entry(void *ptr, size_t offset)
{
return (char *)(ptr)-offset;
}
But as you can see, the only actual "code" is a single subtraction.
Since I used the standard offsetof macro rather than the kernel's invocation of undefined behavior via a poor hack, I also took the liberty of fixing the type-correctness (size_t instead of unsigned long).
I have a question about some code in Eric Roberts' Programming Abstractions in C. He use several libraries of his own both to simplify things for readers and to teach how to write libraries. (All of the library code for the book can be found on this site.)
One library, genlib provides a macro for generic allocation of a pointer to a struct type. I don't understand part of the macro. I'll copy the code below, plus an example of how it is meant to be used, then I'll explain my question in more detail.
/*
* Macro: New
* Usage: p = New(pointer-type);
* -----------------------------
* The New pseudofunction allocates enough space to hold an
* object of the type to which pointer-type points and returns
* a pointer to the newly allocated pointer. Note that
* "New" is different from the "new" operator used in C++;
* the former takes a pointer type and the latter takes the
* target type.
*/
#define New(type) ((type) GetBlock(sizeof *((type) NULL)))
/* GetBlock is a wrapper for malloc. It encasulates the
* common sequence of malloc, check for NULL, return or
* error out, depending on the NULL check. I'm not going
* to copy that code since I'm pretty sure it isn't
* relevant to my question. It can be found here though:
* ftp://ftp.awl.com/cseng/authors/roberts/cs1-c/standard/genlib.c
*/
Roberts intends for the code to be used as follows:
typedef struct {
string name;
/* etc. */
} *employeeT;
employeeT emp;
emp = New(employeeT);
He prefers to use a pointer to the record as the type name, rather than the record itself. So New provides a generic way to allocate such struct records.
In the macro New, what I don't understand is this: sizeof *((type)) NULL). If I'm reading that correctly, it says "take the size of the dereferenced cast of NULL to whatever struct type type represents in a given call". I think I understand the dereferencing: we want to allocate enough space for the struct; the size of the pointer is not what we need, so we dereference to get at the size of the underlying record-type. But I don't understand the idea of casting NULL to a type.
My questions:
You can cast NULL? What does that even mean?
Why is the cast necessary? When I tried removing it, the compiler says error: expected expression. So, sizeof *(type) is not an expression? That confused me since I can do the following to get the sizes of arbitrary pointers-to-structs:
#define struct_size(s_ptr) do { \
printf("sizeof dereferenced pointer to struct %s: %lu\n", \
#s_ptr, sizeof *(s_ptr)); \
} while(0)
Edit: As many people point out below, the two examples aren't the same:
/* How genlib uses the macro. */
New(struct MyStruct*)
/* How I was using my macro. */
struct MyStruct *ptr; New(ptr)
For the record, this isn't homework. I'm an amateur trying to improve at C. Also, there's no problem with the code, as far as I can tell. That is, I'm not asking how I can do something different with it. I'm just trying to better understand (1) how it works and (2) why it must be written the way it is. Thanks.
The issue is that the macro needs to get the size of the type pointed at by the pointer type.
As an example, suppose that you have the the pointer type struct MyStruct*. Without removing the star from this expression, how would you get the size of struct MyStruct? You couldn't write
sizeof(*(struct MyStruct*))
since that's not legal C code.
On the other hand, if you had a variable of type struct MyStruct*, you could do something like this:
struct MyStruct* uselessPointer;
sizeof(*uselessPointer);
Since sizeof doesn't actually evaluate its argument (it just determines the static size of the type of the expression), this is safe.
Of course, in a macro, you can't define a new variable. However, you could make up a random pointer to a struct MyStruct* by casting an existing pointer. Here, NULL is a good candidate - it's an existing pointer that you can legally cast to a struct MyStruct*. Therefore, if you were to write
sizeof(* ((struct MyStruct*)NULL))
the code would
Cast NULL to a struct MyStruct*, yielding a pointer of static type struct MyStruct*.
Determine the size of the object that would be formed by dereferencing the pointer. Since the pointer has type struct MyStruct*, it points at an object of type struct MyStruct, so this yields the type of struct MyStruct.
In other words, it's a simple way to get an object of the pointer type so that you can dereference it and obtain an object of the underlying type.
I've worked with Eric on some other macros and he is a real pro with the preprocessor. I'm not surprised that this works, and I'm not surprised that it's tricky, but it certainly is clever!
As a note - in C++, this sort of trick used to be common until the introduction of the declval utility type, which is a less-hacky version of this operation.
Hope this helps!
It's a hack. It relies on the fact that the argument to the sizeof operator isn't actually evaluated.
To answer your specific questions:
Yes, NULL is just a pointer literal. Like any other pointer, it may be cast.
sizeof operates on either a type or an expression. *(type) would be neither (after macro substitution has occurred), it would be a syntax error.
The following code and its output:
#include <stdio.h>
int x = 0;
#define offsetof(TYPE, MEMBER) ((size_t) &((TYPE *)x)->MEMBER)
#define offsetof_1(TYPE, MEMBER) ((size_t) &((TYPE *)1)->MEMBER)
#define offsetof_2(TYPE, MEMBER) ((size_t) &((TYPE *)2)->MEMBER)
#define offsetof_3(TYPE, MEMBER) ((size_t) &((TYPE *)0)->MEMBER)
struct test{};
struct m {
int b;
char w;
struct test t;
int c;
};
int main(void) {
printf("Checking:%x\n",offsetof(struct m,b));
printf("Checking:%x\n",offsetof_1(struct m,b));
printf("Checking:%x\n",offsetof_2(struct m,b));
printf("Checking:%x\n",offsetof_3(struct m,b));
return 0;
}
Output:
Checking:0
Checking:1
Checking:2
Checking:0
I am trying to understanding the typecasting used here. I think, how compiler does is it treats (type *)(value) as that type starting at address (value). Hence for the given structure the expected value is 0,i.e. offsetof is b, but as we have used different values for typecasting we get different offset. Please let me know if my understanding is correct. I am just trying to understand how literals are typecasted and its implications. This is used in Linux kernel. Hence tagging them as well.
offsetof is a standard facility. The C standard defines it to behave a certain way, but does not define how the compiler implements it. Most compilers rely on undefined behavior, defining a special case for the sake of offsetof implementation.
What you have done is nonsense in terms of the language semantics. Treating the value 2 as a pointer is a classic example of undefined behavior.
It is not the design intent of the language that the user be able to define their own offsetof, so it is better not to try.
"Under the hood", yes pointers are typically scalars held in machine registers, and scalar arithmetic is used to obtain the address of a struct member. Performing the same arithmetic to get the address of an object at "location zero" yields its generic offset within any object. Using integers besides zero as locations may yield an arithmetic result or it may not. (It usually will but the idea is nonsense.)
Experimenting with programs that are conceptual nonsense is a risky way to discover how the system works, because you run the risk that it will flag an error or take a shortcut having detected that something is wrong.
While looking at Linux kernel's implementation of doubly linked circular lists, I've found following macro:
#define container_of(ptr, type, member) ({ \
const typeof( ((type *)0)->member ) *__mptr = (ptr); \
(type *)( (char *)__mptr - offsetof(type,member) );})
The way this works is that it returns pointer to structure given only address of one of its members:
struct blabla
{
int value;
struct list_head *list;
}
Thus you can get pointer to blabla (and get to "value") given only pointer to list.
To my question, how would I make this as portable as possible (best case conforming to C89/C99?). Due to usage of typeof(), this is gcc only.
This is what I've got so far:
#define container_of(ptr, type, member) ( \
(type *) (char *)(ptr)-offsetof(type,member)\
)
Is this snippet conforming to ISO standards (and thus should be able to be compiled on any conforming compiler)?
As Ouah commented, the ({ ... }) statement expression is a GNU extension; you won't be able to use that. Your core expression is close to what's required, but doesn't have enough parentheses:
#define container_of(ptr, type, member) \
((type *) ((char *)(ptr) - offsetof(type, member)))
That looks clean to me. It's only spread across two lines for SO.
The macro is written the way it is to perfom a type check on ptr. It's possible to use a compound literal instead of the statement expression and fall back to a simple check for pointers instead of using __typeof__ if the compiler is not gcc-compatible:
#ifdef __GNUC__
#define member_type(type, member) __typeof__ (((type *)0)->member)
#else
#define member_type(type, member) const void
#endif
#define container_of(ptr, type, member) ((type *)( \
(char *)(member_type(type, member) *){ ptr } - offsetof(type, member)))
ISO C90 compatible version with type check. (However, caveat: two evaluations of ptr!)
#define container_of(ptr, type, member) \
((type *) ((char *) (ptr) - offsetof(type, member) + \
(&((type *) 0)->member == (ptr)) * 0))
struct container {
int dummy;
int memb;
};
#include <stddef.h>
#include <stdio.h>
int main()
{
struct container c;
int *p = &c.memb;
double *q = (double *) p;
struct container *pc = container_of(p, struct container, memb);
struct container *qc = container_of(q, struct container, memb);
return 0;
}
Test:
$ gcc -Wall containerof.c
containerof.c: In function ‘main’:
containerof.c:20:26: warning: comparison of distinct pointer types lacks a cast
containerof.c:20:21: warning: unused variable ‘qc’
containerof.c:19:21: warning: unused variable ‘pc’
We get the distinct pointer types warning for 26, but not 25. That is our diagnostic about pointers being misused.
I first tried placing the type check into the left hand side of a comma operator, gcc complains about that having no effect, which is a nuisance. But by making it an operand, we ensure that it is used.
The &((type *) 0)->member trick isn't well defined by ISO C, but it's widely used for defining offsetof. If your compiler uses this null pointer trick for offsetof, it will almost certainly behave itself in your own macro.
Yes, you can make "container_of" macros to be strictly ISO C conforming. To do this you need two things:
take rid of GNU exensions;
find a way to check types compatibility.
Basically, types checking is not run time operation, but compile time rather. And I not see any reasons, why original "container_of" implementation creates new variable just to assign it and perform type checking. This can be done without creation of new variable in some expression which is only computed (and types checked) in compile time. Fortunately, we have no much options in C and only choice is to use "sizeof(expression)" to check the type. See an example:
#define container_of(ptr, type, member) \
( (void)sizeof(0 ? (ptr) : &((type *)0)->member), \
(type *)((char*)(ptr) - offsetof(type, member)) )
In first line types compatibility is checked (for ternary operator compiler must insure, that types might be converted to common type, or that both types are compatible). Second line is the same, as in original "container_of" macros.
You can play with test program on GodBolt (https://godbolt.org/z/MncvzWfYn) and make sure, that this ISO conforming variant works even in Microsoft's Visual Studio compiler.
PS: After some time, I found that the following variant can be better:
#define CONTAINER_OF(ptr, type, member) \
( (void)sizeof(0 ? (ptr) : &((type*)0)->member), \
(typeof(_Generic((typeof(ptr))0, const typeof(*(typeof(ptr))0)*: (const type*)0, default: (type*)0))) \
((uintptr_t)(const void*)(ptr) - offsetof(type, member)) )
The difference, is that it preserves const qualifier from ptr and assigns it to the result, for example:
if ptr argument is const struct * pointer, the result then will have a type of const type *, despite if the type is const or not;
if ptr argument is non-const pointer (struct*), the result then will have type type*, which may be const or non-const, depending on the type of type argument.
As a result, preserving const qualifier reduces the possibility of the errors, when const pointer to some structure translated into non-const pointer via container_of macro.
Unfortunately, this version requires C23 or non-standard typeof() operator for earlier versions of C standard.
Another reason for ISO-compiant container_of macro, as opposed to implementation from Linux kernel, is that latter uses "statement expression" GCC-extension which works badly in a case, when argument ptr is a temporary variable. The latter might happen, when container_of macro is applied to the result of a function invocation (container_of(func().x, struct y, m), here is assumed, that func() returns a structure in which x is an array of structures), or to the compound statement (container_of((&(struct S){...}), struct B, m)). In both of these cases, a call to container_of macro borrowed from linux will result in a dangling pointer! This happens because a temporary object passed as ptr argument will be destroyed after the first semicolon (at the first line of Linux implementation of container_of macro), and because a variable created by a compound statement expression will be destroyed at the end of the nearest block, which is "statement expression" itself. ISO-compliant implementation of container_of macro have no such issues.