While looking at Linux kernel's implementation of doubly linked circular lists, I've found following macro:
#define container_of(ptr, type, member) ({ \
const typeof( ((type *)0)->member ) *__mptr = (ptr); \
(type *)( (char *)__mptr - offsetof(type,member) );})
The way this works is that it returns pointer to structure given only address of one of its members:
struct blabla
{
int value;
struct list_head *list;
}
Thus you can get pointer to blabla (and get to "value") given only pointer to list.
To my question, how would I make this as portable as possible (best case conforming to C89/C99?). Due to usage of typeof(), this is gcc only.
This is what I've got so far:
#define container_of(ptr, type, member) ( \
(type *) (char *)(ptr)-offsetof(type,member)\
)
Is this snippet conforming to ISO standards (and thus should be able to be compiled on any conforming compiler)?
As Ouah commented, the ({ ... }) statement expression is a GNU extension; you won't be able to use that. Your core expression is close to what's required, but doesn't have enough parentheses:
#define container_of(ptr, type, member) \
((type *) ((char *)(ptr) - offsetof(type, member)))
That looks clean to me. It's only spread across two lines for SO.
The macro is written the way it is to perfom a type check on ptr. It's possible to use a compound literal instead of the statement expression and fall back to a simple check for pointers instead of using __typeof__ if the compiler is not gcc-compatible:
#ifdef __GNUC__
#define member_type(type, member) __typeof__ (((type *)0)->member)
#else
#define member_type(type, member) const void
#endif
#define container_of(ptr, type, member) ((type *)( \
(char *)(member_type(type, member) *){ ptr } - offsetof(type, member)))
ISO C90 compatible version with type check. (However, caveat: two evaluations of ptr!)
#define container_of(ptr, type, member) \
((type *) ((char *) (ptr) - offsetof(type, member) + \
(&((type *) 0)->member == (ptr)) * 0))
struct container {
int dummy;
int memb;
};
#include <stddef.h>
#include <stdio.h>
int main()
{
struct container c;
int *p = &c.memb;
double *q = (double *) p;
struct container *pc = container_of(p, struct container, memb);
struct container *qc = container_of(q, struct container, memb);
return 0;
}
Test:
$ gcc -Wall containerof.c
containerof.c: In function ‘main’:
containerof.c:20:26: warning: comparison of distinct pointer types lacks a cast
containerof.c:20:21: warning: unused variable ‘qc’
containerof.c:19:21: warning: unused variable ‘pc’
We get the distinct pointer types warning for 26, but not 25. That is our diagnostic about pointers being misused.
I first tried placing the type check into the left hand side of a comma operator, gcc complains about that having no effect, which is a nuisance. But by making it an operand, we ensure that it is used.
The &((type *) 0)->member trick isn't well defined by ISO C, but it's widely used for defining offsetof. If your compiler uses this null pointer trick for offsetof, it will almost certainly behave itself in your own macro.
Yes, you can make "container_of" macros to be strictly ISO C conforming. To do this you need two things:
take rid of GNU exensions;
find a way to check types compatibility.
Basically, types checking is not run time operation, but compile time rather. And I not see any reasons, why original "container_of" implementation creates new variable just to assign it and perform type checking. This can be done without creation of new variable in some expression which is only computed (and types checked) in compile time. Fortunately, we have no much options in C and only choice is to use "sizeof(expression)" to check the type. See an example:
#define container_of(ptr, type, member) \
( (void)sizeof(0 ? (ptr) : &((type *)0)->member), \
(type *)((char*)(ptr) - offsetof(type, member)) )
In first line types compatibility is checked (for ternary operator compiler must insure, that types might be converted to common type, or that both types are compatible). Second line is the same, as in original "container_of" macros.
You can play with test program on GodBolt (https://godbolt.org/z/MncvzWfYn) and make sure, that this ISO conforming variant works even in Microsoft's Visual Studio compiler.
PS: After some time, I found that the following variant can be better:
#define CONTAINER_OF(ptr, type, member) \
( (void)sizeof(0 ? (ptr) : &((type*)0)->member), \
(typeof(_Generic((typeof(ptr))0, const typeof(*(typeof(ptr))0)*: (const type*)0, default: (type*)0))) \
((uintptr_t)(const void*)(ptr) - offsetof(type, member)) )
The difference, is that it preserves const qualifier from ptr and assigns it to the result, for example:
if ptr argument is const struct * pointer, the result then will have a type of const type *, despite if the type is const or not;
if ptr argument is non-const pointer (struct*), the result then will have type type*, which may be const or non-const, depending on the type of type argument.
As a result, preserving const qualifier reduces the possibility of the errors, when const pointer to some structure translated into non-const pointer via container_of macro.
Unfortunately, this version requires C23 or non-standard typeof() operator for earlier versions of C standard.
Another reason for ISO-compiant container_of macro, as opposed to implementation from Linux kernel, is that latter uses "statement expression" GCC-extension which works badly in a case, when argument ptr is a temporary variable. The latter might happen, when container_of macro is applied to the result of a function invocation (container_of(func().x, struct y, m), here is assumed, that func() returns a structure in which x is an array of structures), or to the compound statement (container_of((&(struct S){...}), struct B, m)). In both of these cases, a call to container_of macro borrowed from linux will result in a dangling pointer! This happens because a temporary object passed as ptr argument will be destroyed after the first semicolon (at the first line of Linux implementation of container_of macro), and because a variable created by a compound statement expression will be destroyed at the end of the nearest block, which is "statement expression" itself. ISO-compliant implementation of container_of macro have no such issues.
Related
Prerequisites:
As per C standard, pointer arithmetics that would yield an invalid pointer, cause undefined behavior.
Linux source code seems to conform with C standard in a desire to be compatible with most architectures.
Linux's list implementation contains the following code(formatting preserved, probably the idea for another question is how to set proper tabulation width using Stackoverflow syntax):
#define list_entry(ptr, type, member) \
container_of(ptr, type, member)
#define list_next_entry(pos, member) \
list_entry((pos)->member.next, typeof(*(pos)), member)
#define list_first_entry(ptr, type, member) \
list_entry((ptr)->next, type, member)
#define list_entry_is_head(pos, head, member) \
(&pos->member == (head))
#define list_for_each_entry(pos, head, member) \
for (pos = list_first_entry(head, typeof(*pos), member); \
!list_entry_is_head(pos, head, member); \
pos = list_next_entry(pos, member))
Typical usecase of the aforementioned list implementation is having structure of say type struct A, containing a head for the list of stuctures of type struct B.
Q: Let's assume offsetof(struct B, entry_in_list) > offsetof(struct A, list_head) and the following loop is implemented:
struct A* A_ptr = something_meaningful;
struct B* pos = NULL;
list_for_each_entry(pos, &A_ptr->list_head, entry_in_list) {
do_something();
}
Then last (before loop exit) evaluation of list_next_entry(pos, member) would extend to:
container_of(A_ptr->list_head, struct B, entry_in_list) =
= (char*)A_ptr->list_head - offsetof(struct B, entry_in_list) =
= (char*)A_ptr + offsetof(struct A, list_head) - offsetof(struct B, entry_in_list)
, which, according to our assumption, would point to area before A struct. Assuming this area does not contain allocated memory, the result of the container_of() macro would be an invalid pointer, thus causing UB(in general case OFC) in Linux. Is this reasoning plausible or am I mistaken somehow?
Or are there some parts of the standard universally considered to not be worth to follow?
As suspected by OP, the implementation of the list_for_each_entry(pos, head, member) macro depends on undefined behavior in the C language in order for the loop termination condition !list_entry_is_head(pos, head, member) to become false.
Assuming the list is non-empty, then after the final iteration, the third "advancing" expression of the for loop produces a pointer to an invalid typeof(*pos) at an address offsetof(typeof(*pos), member) bytes before the struct list_head pointed to by head. It relies on &pos->member nevertheless comparing equal to head.
Although it depends on undefined behavior, it is hard for the compiler to determine that pos is technically an invalid pointer. As long as both pos and head point within the same flat address space, the Linux kernel manages to get away with this bending of the rules.
The alternative would be for #include <linux/list.h> to not provide the list_for_each_entry(pos, head, member) macro at all, and for code to use the list_for_each(pos, head) and list_entry(ptr, type, member) macros instead (where pos is a struct list_head * and ptr is a type *), but that would typically require extra variables in the code.
Additional assertions made when compiling the kernel. These are actually used all over the place.
A pointer may be loaded with an address that isn't allocated. You see this on every system call entry. Special care must be taken handling such pointers as dereferencing them can do worse than crash.
Dereferencing a NULL pointer is not guaranteed to crash; and the compiler is not allowed to assume that a path that dereferences NULL is unreachable. (This one was added late after a NULL pointer optimization removed a security check.) On some architectures there's actually something there; on other architectures it's just another usermode pointer.
The compiler is told these are true by compiler options. (In fact the first one is generally assumed to be true in flat model, which the kernel is.)
The flag passed to gcc is -fno-delete-null-pointer-checks. Reference for null pointer optimization change: https://lwn.net/Articles/342420/
You are right that this is undefined behavior. According to Richard Biener, this kind of undefined behavior is not supported/made defined by -fno-strict-aliasing. (Clang treats this as undefined as well.)
This particular miscompilation was observed with Open vSwitch, but its list macros are clearly modeled after/copied from the kernel. Why does the kernel get away with it?
It is built with -fno-strict-aliasing (although this does not help with this particular case).
The kernel is less often built with LTO.
The kernel does not use list heads on the stack.
Compilers do not recognize the kernel allocation functions.
As a result, compilers do not observe the invalid/impossible object references and do not optimize based on that.
The following is the quick and easy way to define the containerof
macro, using only standard C features:
#include <stddef.h>
#define containerof(ptr, type, member) \
((type *)((char *)(ptr)-offsetof(type, member)))
Another commonly found definition of the macro uses GNU C statement
expressions to
provide additional type-checking:
#include <stddef.h>
#if __GNUC__+0 >= 3
#define containerof(ptr, type, member) ({ \
const __typeof__(((type *)0)->member) *containerof_memberptr_ = (ptr); \
(type *)((char *)(containerof_memberptr_)-offsetof(type, member)); })
#else
/* fallback */
#endif
My question is if there's a definition of the macro using only standard
C features that still checks the referenced type of the ptr argument
against the type of the member argument?
I've come up with the following definition:
#include <stddef.h>
#define containerof(ptr, type, member) \
((type *)((char *)(0 ? &((type *)0)->member : (ptr))-offsetof(type, member)))
Any incompatible pointer passed as the ptr argument violates the
following constraint of the conditional operator (C90 6.3.15
Conditional operator):
— both operands are pointers to qualified or unqualified versions of
compatible types
I was hoping for such a violation to generate an error by default,
however GCC and clang only issue a warning and there doesn't seem to be
a specific -Werror=... option in GCC to turn the emitted warning into
an error.
warning: pointer type mismatch in conditional expression
clang conveniently provides such an option called
-Werror=pointer-type-mismatch. And both GCC and clang support the
broad -pedantic-errors option that promotes the warning to an error as
well.
The standard requires a diagnostic message to be issued on constraint
violation (C90 5.1.1.3 Diagnostics), but that doesn't mean the
compiler has to error out on non-compliant code.
In conclusion, it is possible to define a containerof macro that
doesn't rely on extensions to the language, yet still yields some
type-checking behavior, whether that is by issuing a mandatory
diagnostic message or a compilation error, given compiler support.
One other advantage of this definition (shared with the one lacking
type-checking) over the one using a GNU statement expression is that it
results in a constant expression usable in initializer contexts.
How to pass struct type in arguments? I guess it is not possible, still wanted to check if it is feasible.
My requirement is something like this
Below is a macro list_entry used in linux kernel
#define list_entry(ptr, type, member) \
((type *)((char *)(ptr)-(unsigned long)(&((type *)0)->member)))
I want to rewrite it using function. How do I write?
list_entry(void *ptr, ???, ???)
Is there a way we can do this?
I am trying to write a wrapper to free a given list. So to use list_entry in my function, my function needs to be passed with type and the member. Is it possible?
It's fundamentally not possible to write list_entry, with its same behavior and signature, as a function. This is because its arguments include a type and a member name, neither of which is a value in C.
On the other hand, you could to some extent abstract out the "code" content of the macro into a function:
#define list_entry(ptr, type, member) \
((type *)f_list_entry(ptr, offsetof(type, member)))
static void *f_list_entry(void *ptr, size_t offset)
{
return (char *)(ptr)-offset;
}
But as you can see, the only actual "code" is a single subtraction.
Since I used the standard offsetof macro rather than the kernel's invocation of undefined behavior via a poor hack, I also took the liberty of fixing the type-correctness (size_t instead of unsigned long).
While looking at Linux kernel's implementation of doubly linked circular lists, I've found following macro:
#define container_of(ptr, type, member) ({ \
const typeof( ((type *)0)->member ) *__mptr = (ptr); \
(type *)( (char *)__mptr - offsetof(type,member) );})
The way this works is that it returns pointer to structure given only address of one of its members:
struct blabla
{
int value;
struct list_head *list;
}
Thus you can get pointer to blabla (and get to "value") given only pointer to list.
To my question, how would I make this as portable as possible (best case conforming to C89/C99?). Due to usage of typeof(), this is gcc only.
This is what I've got so far:
#define container_of(ptr, type, member) ( \
(type *) (char *)(ptr)-offsetof(type,member)\
)
Is this snippet conforming to ISO standards (and thus should be able to be compiled on any conforming compiler)?
As Ouah commented, the ({ ... }) statement expression is a GNU extension; you won't be able to use that. Your core expression is close to what's required, but doesn't have enough parentheses:
#define container_of(ptr, type, member) \
((type *) ((char *)(ptr) - offsetof(type, member)))
That looks clean to me. It's only spread across two lines for SO.
The macro is written the way it is to perfom a type check on ptr. It's possible to use a compound literal instead of the statement expression and fall back to a simple check for pointers instead of using __typeof__ if the compiler is not gcc-compatible:
#ifdef __GNUC__
#define member_type(type, member) __typeof__ (((type *)0)->member)
#else
#define member_type(type, member) const void
#endif
#define container_of(ptr, type, member) ((type *)( \
(char *)(member_type(type, member) *){ ptr } - offsetof(type, member)))
ISO C90 compatible version with type check. (However, caveat: two evaluations of ptr!)
#define container_of(ptr, type, member) \
((type *) ((char *) (ptr) - offsetof(type, member) + \
(&((type *) 0)->member == (ptr)) * 0))
struct container {
int dummy;
int memb;
};
#include <stddef.h>
#include <stdio.h>
int main()
{
struct container c;
int *p = &c.memb;
double *q = (double *) p;
struct container *pc = container_of(p, struct container, memb);
struct container *qc = container_of(q, struct container, memb);
return 0;
}
Test:
$ gcc -Wall containerof.c
containerof.c: In function ‘main’:
containerof.c:20:26: warning: comparison of distinct pointer types lacks a cast
containerof.c:20:21: warning: unused variable ‘qc’
containerof.c:19:21: warning: unused variable ‘pc’
We get the distinct pointer types warning for 26, but not 25. That is our diagnostic about pointers being misused.
I first tried placing the type check into the left hand side of a comma operator, gcc complains about that having no effect, which is a nuisance. But by making it an operand, we ensure that it is used.
The &((type *) 0)->member trick isn't well defined by ISO C, but it's widely used for defining offsetof. If your compiler uses this null pointer trick for offsetof, it will almost certainly behave itself in your own macro.
Yes, you can make "container_of" macros to be strictly ISO C conforming. To do this you need two things:
take rid of GNU exensions;
find a way to check types compatibility.
Basically, types checking is not run time operation, but compile time rather. And I not see any reasons, why original "container_of" implementation creates new variable just to assign it and perform type checking. This can be done without creation of new variable in some expression which is only computed (and types checked) in compile time. Fortunately, we have no much options in C and only choice is to use "sizeof(expression)" to check the type. See an example:
#define container_of(ptr, type, member) \
( (void)sizeof(0 ? (ptr) : &((type *)0)->member), \
(type *)((char*)(ptr) - offsetof(type, member)) )
In first line types compatibility is checked (for ternary operator compiler must insure, that types might be converted to common type, or that both types are compatible). Second line is the same, as in original "container_of" macros.
You can play with test program on GodBolt (https://godbolt.org/z/MncvzWfYn) and make sure, that this ISO conforming variant works even in Microsoft's Visual Studio compiler.
PS: After some time, I found that the following variant can be better:
#define CONTAINER_OF(ptr, type, member) \
( (void)sizeof(0 ? (ptr) : &((type*)0)->member), \
(typeof(_Generic((typeof(ptr))0, const typeof(*(typeof(ptr))0)*: (const type*)0, default: (type*)0))) \
((uintptr_t)(const void*)(ptr) - offsetof(type, member)) )
The difference, is that it preserves const qualifier from ptr and assigns it to the result, for example:
if ptr argument is const struct * pointer, the result then will have a type of const type *, despite if the type is const or not;
if ptr argument is non-const pointer (struct*), the result then will have type type*, which may be const or non-const, depending on the type of type argument.
As a result, preserving const qualifier reduces the possibility of the errors, when const pointer to some structure translated into non-const pointer via container_of macro.
Unfortunately, this version requires C23 or non-standard typeof() operator for earlier versions of C standard.
Another reason for ISO-compiant container_of macro, as opposed to implementation from Linux kernel, is that latter uses "statement expression" GCC-extension which works badly in a case, when argument ptr is a temporary variable. The latter might happen, when container_of macro is applied to the result of a function invocation (container_of(func().x, struct y, m), here is assumed, that func() returns a structure in which x is an array of structures), or to the compound statement (container_of((&(struct S){...}), struct B, m)). In both of these cases, a call to container_of macro borrowed from linux will result in a dangling pointer! This happens because a temporary object passed as ptr argument will be destroyed after the first semicolon (at the first line of Linux implementation of container_of macro), and because a variable created by a compound statement expression will be destroyed at the end of the nearest block, which is "statement expression" itself. ISO-compliant implementation of container_of macro have no such issues.
Is it possible to implement the container_of macro in pure C90? I'm not sure how to do it as the Kernel implementation depends on GCC Hacks such as the typeof operator.
I'm asking because I would like to implement a generic container in C90 similar to the Kernel's linked list. The actual container I'm thinking of is a sequenced set similar to what you might get from Boost MultiIndex.
The use of typeof in the kernel definition of container_of() is just for compile-time type-checking - it ensures that the passed ptr is really a pointer to the same type as member. It can be modified to be entirely ANSI C at the cost of this type-checking:
#define container_of(ptr, type, member) ((type *)((char *)ptr - offsetof(type, member)))
(offsetof() is in <stddef.h>)
Yes.
The trick is replacing compound expression and typeof used for type checking. It can be done by replacing evaluation of (ptr) with:
(1 ? (ptr) : &((type*)0)->member)
A conditional operator always returns the first operand because its selection operand is 1. The other operand is not evaluated, therefore there is no UB due to NULL dereferencing. Moreover, the types of operands of ?: have to be compatible forcing type checking from a compiler.
The final macro is:
#define container_of(ptr, type, member) \
((type*)((char*)(1 ? (ptr) : &((type*)0)->member) - offsetof(type, member)))
This macro is even better than the original container_of.
The C89 variant can be used at file scope or for initialization of static variables, while the Linux one cannot.