compile-time struct size check, error out if odd - c

Is there any possible way to make the compiler bail out if the sizeof (struct Astruct) is uneven?
Background information:
We have a 16-bit microprocessor which will give processor alignment errors if a 16-bit value is mis-aligned. That might happen in the following scenario:
typedef struct
{
U8BIT u8BitValue1;
U8BIT u8BitValue2;
U8BIT u8BitValue3;
} unevenAmountOf8BitValues;
typedef struct
{
U16BIT u16BitValue1;
U16BIT u16BitValue2;
} my16BitValues;
#define U8BIT_COUNT 3
#define U16BIT_COUNT 2
typedef struct
{
unevenAmountOf8BitValues u8BitValues;
my16BitValues u16BitValues;
} valuesCombined;
typedef union
{
valuesCombined myValues;
U8BIT buffer[sizeof(valuesCombined)];
struct
{
U8BIT bufferU8[U8BIT_COUNT];
U16BIT bufferU16[U16BIT_COUNT]; /* <<-- missalignment */
} valuesPerType;
} myValuesInRamAndRom
What we do now is counting the amount of U8BIT/U16BIT/U32BIT values (well, keeping track of the amount using excel) manually and putting that in the U(8/16/32)BIT_COUNT define and then the following:
#if U8BIT_COUNT % 2 == 1
#error The number of U8BIT parameters need to be even, add a dummy
#endif
Keeping track of the amount of U8-/U16-/U32BIT values is pretty error prone and we've had quite some moments that we were thinking "hey, it ain't working", an hour or what later, oh! Darn, forgot to adjust the amount of values define.
A preferred method would be to use the sizeof operator, however that can't be used in the error checking, which I would really like to keep.
So is there anyway to use the sizeof operator and to keep some form of error checking that the amount of U8BIT values must be even?
Combined solution by Lundin and Aaron McDaid:
#define COMPILE_TIME_ASSERT(expr) {typedef U8BIT COMP_TIME_ASSERT[((!!(expr))*2-1)];}

With a C11 compiler, use:
static_assert (sizeof(the struct) % 2 == 0,
"Misaligned");
With older compilers, you can use dirty tricks like
#define COMPILE_TIME_ASSERT(expr) typedef char COMP_TIME_ASSERT[(expr) ? 1 : 0];
...
COMPILE_TIME_ASSERT(sizeof(the_struct) % 2 == 0);
The real solution to your specific problem might however be to ensure that struct padding is enabled. You shouldn't get any misalignments then.

It's possible, using a trick that's also being used in the Linux kernel:
#define BUILD_BUG_OR_ZERO(e) (sizeof(struct{ int:-!!(e);}))
#define ENSURE_EVEN_SIZE(e) BUILD_BUG_OR_ZERO(sizeof(e) % 2 == 1)
struct uneven{
char a,b,c;
};
struct even{
char a,b,c,d;
};
int main(){
ENSURE_EVEN_SIZE(struct even);
/* compiler error: */
ENSURE_EVEN_SIZE(struct uneven);
}
If sizeof(e) % 2 == 1 is true, the bitfield int:-!!(e) would have a negative size, which is forbidden. (Ideone)

Here is the version which allows using same assertion macro multiple times in the same
file.
/*
General purpose static assert.
Works in/out -side of scope:
STATIC_ASSERT(sizeof(long)==8);
int main()
{
STATIC_ASSERT(sizeof(int)==4);
}
*/
#define STATIC_ASSERT(X) STATIC_ASSERT2(X,__LINE__)
/*
These macros are required by STATIC_ASSERT to make token pasting work.
Not really useful by themselves.
*/
#define STATIC_ASSERT2(X,L) STATIC_ASSERT3(X,L)
#define STATIC_ASSERT3(X,L) STATIC_ASSERT_MSG(X,at_line_##L)
/*
Static assertion with special error message.
Note: It depends on compiler whether message is visible or not!
STATIC_ASSERT_MSG(sizeof(long)==8, long_is_not_eight_bytes);
*/
#define STATIC_ASSERT_MSG(COND,MSG) \
typedef char static_assertion_##MSG[(!!(COND))*2-1]

Related

Creating iterable list of register in embedded C

I have been programming with python java and c++, which all have list objects predefined. I'm now working on a microcontroller in C embedded, but objects such as lists and functions such as printf simply don't exist.
What I am trying to do is the following. I have multiple registers which I attach to defines. I want to put all my defines in a list I can access.
#include <stdlib.h>
#include <stdio.h>
#include <xc.h>
#define KP_ROW1 LATBbits.LATB0
#define KP_ROW2 LATBbits.LATB1
#define KP_ROW3 LATBbits.LATB2
#define KP_ROW4 LATBbits.LATB3
#define KP_COL1 LATBbits.LATB4
#define KP_COL2 LATBbits.LATB5
#define KP_COL3 LATBbits.LATB6
#define KP_COL4 LATBbits.LATB7
KP_ROW = [KP_ROW1, KP_ROW2, KP_ROW3, KP_ROW4]; //error on this line
KP_COL = [KP_COL1, KP_COL2, KP_COL3, KP_COL4]; //error
for(int i=0;i<4;i++)
{
if (KP_COL[i] == 1){return 1;}
}
Since I have no previous experience in C embedded, I assumed that google could help me, but it seems that all the solutions I find need the good understanding of struct and the fabrication of really complex functions.
The reasons for this is that I will have maybe 100 pins and I don,t want to make "if" statements for each, I want to iterate throughout a list. I'm using a PIC18F with the XC8 compiler.
I'm asking for advice. How would you do it? Is there a faster, simpler way than making your own list class?
It's quite hard to understand what is desired here. I understand that you wish to read the values of 100 register in your program. You have to know their addresses. Then, you can create a series of macros:
#define REG1 0xabcd
#define REG2 0x1234
...
or whichever addresses. Then, create more macros to access the values directly:
#define REG1_VAL (*(volatile uint8_t *) REG1)
#define REG2_VAL (*(volatile uint8_t *) REG2)
...
Then you can write assignments such as REG1_VAL = 1 or tests such as REG1_VAL == 1. Otherwise, you can create macros that access their values through their addresses:
#define READ(reg) (*(volatile uint8_t *) reg)
Usage of the above to obtain a register value is then:
READ(REG1);
You could then allocate an array of 100 elements:
volatile uint8_t regs[100] = {REG1, REG2, ..., REG100};
and iterate through that array:
for (i = 0; i < 100; i ++) {
if (READ(regs[i]) == 1) {
...
}
}
Hope this helps you!
Note: These macros would need fine tuning to avoid subtle errors, but I hope they are still useful to illustrate what I mean.
I found the answer!
It was not working because I was trying to get the adress of the bit instead of the register.... my bad. It should rather be:
#define KP_ROW LATB
And then add some bitmasks to write the pins I want.
Sorry for this beginner's error!

How to separate #defined arguments in C

I'm working on an embedded project and I have all the ports and pins defined like this:
#define SENSOR_1 gpioPortA,15
Is there a way to extract just the integer without creating a new #define?
So far I used a typedef and assigned the pin value from the #define like so:
typedef struct
{
GPIO_Port_TypeDef port;
uint8_t pin;
}sensor_t;
sensor_t sensor1 = {SENSOR_1};
/* Now sensor1.pin is the pin value */
but I want to pass the pin to a switch case and I can only use constant values for that and I'd rather avoid using an if statement.
Is there a way to pass the pin to a switch case without a new #define?
Short answer is no.
You could however use sensor1.pin when needed but not in case of switch statement as switch does not support variables there. In this case use if-else statement.
Or do it like this:
#define SENSOR_1_PIN 10
#define SENSOR_1 my_port,SENSOR_1_PIN
and then in switch you use SENSOR_1_PIN in case part.
switch (condition) {
case SENSOR_1_PIN:
//Do stuff
break;
//....
}
Just to remind again as it was posted in first comment on your question. Doing that is very dangerous.
Most proper way would be to do it like this:
#define SENSOR_1_PORT GPIOB
#define SENSOR_1_PIN 15
//If structure order changes here, you may lead to wrong data interpretation
sensor_t sensor1 = {SENSOR_1_PORT, SENSOR_1_PIN};
If you are C99 compliant, you may do it even more safer like this:
//If structure order changes here, your data are still properly assigned to fields
sensor_t sensor1 = {.port = SENSOR_1_PORT, .pin = SENSOR_1_PIN};
You can define a macro to extract the value from your definition.
#define GET_SECOND(x, y) y
#define PIN_VALUE(x) GET_SECOND(x)
switch (pin) {
case PIN_VALUE(SENSOR_1):
/* ... */
break;
}
PIN_VALUE must allow SENSOR_1 to be expanded via helper macro so that the second part can be extracted.
The proper solution at this point is to re-design the definitions into something that makes more sense, or alternatively create new constants.
As a last resort, if you are stuck with these macros, you can parse them in the following way:
#include <stdio.h>
#include <stdint.h>
typedef int GPIO_Port_TypeDef; // whatever type this happens to be
typedef struct
{
GPIO_Port_TypeDef port;
uint8_t pin;
}sensor_t;
#define GET_FIELD(field,...) (sensor_t){__VA_ARGS__}.field
#define SENSOR_1 gpioPortA,15
int main (void)
{
int gpioPortA = 1;
printf("%d %d", GET_FIELD(port, SENSOR_1), GET_FIELD(pin, SENSOR_1));
}
The type-generic version would be:
#define GET_FIELD(type, field, ...) (type){__VA_ARGS__}.field
...
printf("%d %d", GET_FIELD(sensor_t, port, SENSOR_1), GET_FIELD(sensor_t, pin, SENSOR_1));
This scales variably no matter how many fields there are. This is however not recommended practice. Macros in general, and variadic macros in particular, should be avoided.
As for how to use run-time variables in case - you can't. Use an if-else if statement instead.
What about inserting a define inside a define ? Instead of directly adding 15, you could make a define holding 15 and insert it elsewhere.
E.g:
#define SENSORVAL 15
#define SENSOR_1 gpioPortA,SENSORVAL
typedef struct
{
GPIO_Port_TypeDef port;
uint8_t pin;
}sensor_t;
sensor_t sensor1 = {SENSOR_1};
/* Now sensor1.pin is the pin value */

compile time check for enums [duplicate]

Is there a compile-time way to detect / prevent duplicate values within a C/C++ enumeration?
The catch is that there are multiple items which are initialized to explicit values.
Background:
I've inherited some C code such as the following:
#define BASE1_VAL (5)
#define BASE2_VAL (7)
typedef enum
{
MsgFoo1A = BASE1_VAL, // 5
MsgFoo1B, // 6
MsgFoo1C, // 7
MsgFoo1D, // 8
MsgFoo1E, // 9
MsgFoo2A = BASE2_VAL, // Uh oh! 7 again...
MsgFoo2B // Uh oh! 8 again...
} FOO;
The problem is that as the code grows & as developers add more messages to the MsgFoo1x group, eventually it overruns BASE2_VAL.
This code will eventually be migrated to C++, so if there is a C++-only solution (template magic?), that's OK -- but a solution that works with C and C++ is better.
There are a couple ways to check this compile time, but they might not always work for you. Start by inserting a "marker" enum value right before MsgFoo2A.
typedef enum
{
MsgFoo1A = BASE1_VAL,
MsgFoo1B,
MsgFoo1C,
MsgFoo1D,
MsgFoo1E,
MARKER_1_DONT_USE, /* Don't use this value, but leave it here. */
MsgFoo2A = BASE2_VAL,
MsgFoo2B
} FOO;
Now we need a way to ensure that MARKER_1_DONT_USE < BASE2_VAL at compile-time. There are two common techiques.
Negative size arrays
It is an error to declare an array with negative size. This looks a little ugly, but it works.
extern int IGNORE_ENUM_CHECK[MARKER_1_DONT_USE > BASE2_VAL ? -1 : 1];
Almost every compiler ever written will generate an error if MARKER_1_DONT_USE is greater than BASE_2_VAL. GCC spits out:
test.c:16: error: size of array ‘IGNORE_ENUM_CHECK’ is negative
Static assertions
If your compiler supports C11, you can use _Static_assert. Support for C11 is not ubiquitous, but your compiler may support _Static_assert anyway, especially since the corresponding feature in C++ is widely supported.
_Static_assert(MARKER_1_DONT_USE < BASE2_VAL, "Enum values overlap.");
GCC spits out the following message:
test.c:16:1: error: static assertion failed: "Enum values overlap."
_Static_assert(MARKER_1_DONT_USE < BASE2_VAL, "Enum values overlap.");
^
I didn't see "pretty" in your requirements, so I submit this solution implemented using the Boost Preprocessor library.
As an up-front disclaimer, I haven't used Boost.Preprocessor a whole lot and I've only tested this with the test cases presented here, so there could be bugs, and there may be an easier, cleaner way to do this. I certainly welcome comments, corrections, suggestions, insults, etc.
Here we go:
#include <boost/preprocessor.hpp>
#define EXPAND_ENUM_VALUE(r, data, i, elem) \
BOOST_PP_SEQ_ELEM(0, elem) \
BOOST_PP_IIF( \
BOOST_PP_EQUAL(BOOST_PP_SEQ_SIZE(elem), 2), \
= BOOST_PP_SEQ_ELEM(1, elem), \
BOOST_PP_EMPTY()) \
BOOST_PP_COMMA_IF(BOOST_PP_NOT_EQUAL(data, BOOST_PP_ADD(i, 1)))
#define ADD_CASE_FOR_ENUM_VALUE(r, data, elem) \
case BOOST_PP_SEQ_ELEM(0, elem) : break;
#define DEFINE_UNIQUE_ENUM(name, values) \
enum name \
{ \
BOOST_PP_SEQ_FOR_EACH_I(EXPAND_ENUM_VALUE, \
BOOST_PP_SEQ_SIZE(values), values) \
}; \
\
namespace detail \
{ \
void UniqueEnumSanityCheck##name() \
{ \
switch (name()) \
{ \
BOOST_PP_SEQ_FOR_EACH(ADD_CASE_FOR_ENUM_VALUE, name, values) \
} \
} \
}
We can then use it like so:
DEFINE_UNIQUE_ENUM(DayOfWeek, ((Monday) (1))
((Tuesday) (2))
((Wednesday) )
((Thursday) (4)))
The enumerator value is optional; this code generates an enumeration equivalent to:
enum DayOfWeek
{
Monday = 1,
Tuesday = 2,
Wednesday,
Thursday = 4
};
It also generates a sanity-check function that contains a switch statement as described in Ben Voigt's answer. If we change the enumeration declaration such that we have non-unique enumerator values, e.g.,
DEFINE_UNIQUE_ENUM(DayOfWeek, ((Monday) (1))
((Tuesday) (2))
((Wednesday) )
((Thursday) (1)))
it will not compile (Visual C++ reports the expected error C2196: case value '1' already used).
Thanks also to Matthieu M., whose answer to another question got me interested in the Boost Preprocessor library.
I don't believe there's a way to detect this with the language itself, considering there are conceivable cases where you'd want two enumeration values to be the same. You can, however, always ensure all explicitly set items are at the top of the list:
typedef enum
{
MsgFoo1A = BASE1_VAL, // 5
MsgFoo2A = BASE2_VAL, // 7
MsgFoo1B, // 8
MsgFoo1C, // 9
MsgFoo1D, // 10
MsgFoo1E, // 11
MsgFoo2B // 12
} FOO;
So long as assigned values are at the top, no collision is possible, unless for some reason the macros expand to values which are the same.
Usually this problem is overcome by giving a fixed number of bits for each MsgFooX group, and ensuring each group does not overflow it's allotted number of bits. The "Number of bits" solution is nice because it allows a bitwise test to determine to which message group something belongs. But there's no built-in language feature to do this because there are legitimate cases for an enum having two of the same value:
typedef enum
{
gray = 4, //Gr[ae]y should be the same
grey = 4,
color = 5, //Also makes sense in some cases
couleur = 5
} FOO;
I don't know of anything that will automatically check all enum members, but if you want to check that future changes to the initializers (or the macros they rely on) don't cause collisions:
switch (0) {
case MsgFoo1A: break;
case MsgFoo1B: break;
case MsgFoo1C: break;
case MsgFoo1D: break;
case MsgFoo1E: break;
case MsgFoo2A: break;
case MsgFoo2B: break;
}
will cause a compiler error if any of the integral values is reused, and most compilers will even tell you what value (the numeric value) was a problem.
You could roll a more robust solution of defining enums using Boost.Preprocessor - wether its worth the time is a different matter.
If you are moving to C++ anyway, maybe the (proposed) Boost.Enum suits you (available via the Boost Vault).
Another approach might be to use something like gccxml (or more comfortably pygccxml) to identify candidates for manual inspection.
While we do not have full on reflection, you can solve this problem if you can relist the enumeration values.
Somewhere this is declared:
enum E { A = 0, B = 0 };
elsewhere, we build this machinery:
template<typename S, S s0, S... s>
struct first_not_same_as_rest : std::true_type {};
template<typename S, S s0, S s1, S... s>
struct first_not_same_as_rest : std::integral_constant< bool,
(s0 != s1) && first_not_same_as_rest< S, s0, s... >::value
> {};
template<typename S, S... s>
struct is_distinct : std::true_type {};
template<typename S, S s0, S... s>
struct is_distinct : std::integral_constant< bool,
std::is_distinct<S, s...>::value &&
first_not_same_as_rest< S, s0, s... >::value
> {};
Once you have that machinery (which requires C++11), we can do the following:
static_assert( is_distinct< E, A, B >::value, "duplicate values in E detected" );
and at compile time we will ensure that no two elements are equal.
This requires O(n) recursion depth and O(n^2) work by the compiler at compile time, so for extremely large enums this could cause problems. A O(lg(n)) depth and O(n lg(n)) work with a much larger constant factor can be done by sorting the list of elements first, but that is much, much more work.
With the enum reflection code proposed for C++1y-C++17, this will be doable without relisting the elements.
I didn't completely like any of the answers already posted here, but they gave me some ideas. The crucial technique is to rely on Ben Voight's answer of using a switch statement. If multiple cases in a switch share the same number, you'll get a compile error.
Most usefully to both myself and probably the original poster, this doesn't require any C++ features.
To clean things up, I used aaronps's answer at How can I avoid repeating myself when creating a C++ enum and a dependent data structure?
First, define this in some header someplace:
#define DEFINE_ENUM_VALUE(name, value) name=value,
#define CHECK_ENUM_VALUE(name, value) case name:
#define DEFINE_ENUM(enum_name, enum_values) \
typedef enum { enum_values(DEFINE_ENUM_VALUE) } enum_name;
#define CHECK_ENUM(enum_name, enum_values) \
void enum_name ## _test (void) { switch(0) { enum_values(CHECK_ENUM_VALUE); } }
Now, whenever you need to have an enumeration:
#define COLOR_VALUES(GEN) \
GEN(Red, 1) \
GEN(Green, 2) \
GEN(Blue, 2)
Finally, these lines are required to actually make the enumeration:
DEFINE_ENUM(Color, COLOR_VALUES)
CHECK_ENUM(Color, COLOR_VALUES)
DEFINE_ENUM makes the enum data type itself. CHECK_ENUM makes a test function that switches on all the enum values. The compiler will crash when compiling CHECK_ENUM if you have duplicates.
Here's a solution using X macro without Boost. First define the X macro and its helper macros. I'm using this solution to portably make 2 overloads for the X macro so that you can define the enum with or without an explicit value. If you're using GCC or Clang then it can be made shorter
#define COUNT_X_ARGS_IMPL2(_1, _2, count, ...) count
#define COUNT_X_ARGS_IMPL(args) COUNT_X_ARGS_IMPL2 args
#define COUNT_X_ARGS(...) COUNT_X_ARGS_IMPL((__VA_ARGS__, 2, 1, 0))
/* Pick the right X macro to invoke. */
#define X_CHOOSE_HELPER2(count) X##count
#define X_CHOOSE_HELPER1(count) X_CHOOSE_HELPER2(count)
#define X_CHOOSE_HELPER(count) X_CHOOSE_HELPER1(count)
/* The actual macro. */
#define X_GLUE(x, y) x y
#define X(...) X_GLUE(X_CHOOSE_HELPER(COUNT_X_ARGS(__VA_ARGS__)), (__VA_ARGS__))
Then define the macro and check it
#define BASE1_VAL (5)
#define BASE2_VAL (7)
// Enum values
#define MY_ENUM \
X(MsgFoo1A, BASE1_VAL) \
X(MsgFoo1B) \
X(MsgFoo1C) \
X(MsgFoo1D) \
X(MsgFoo1E) \
X(MsgFoo2A, BASE2_VAL) \
X(MsgFoo2B)
// Define the enum
#define X1(enum_name) enum_name,
#define X2(enum_name, enum_value) enum_name = enum_value,
enum foo
{
MY_ENUM
};
#undef X1
#undef X2
// Check duplicates
#define X1(enum_name) case enum_name: break;
#define X2(enum_name, enum_value) case enum_name: break;
static void check_enum_duplicate()
{
switch(0)
{
MY_ENUM
}
}
#undef X1
#undef X2
Use it
int main()
{
// Do something with the whole enum
#define X1(enum_name) printf("%s = %d\n", #enum_name, enum_name);
#define X2(enum_name, enum_value) printf("%s = %d\n", #enum_name, enum_value);
// Print the whole enum
MY_ENUM
#undef X1
#undef X2
}

Best way to define offsets via C preprocessor

I would like to define a macro that will help me to auto generate offsets. Something like this:
#define MEM_OFFSET(name, size) ...
MEM_OFFSET(param1, 1);
MEM_OFFSET(param2, 2);
MEM_OFFSET(param3, 4);
MEM_OFFSET(param4, 1);
should generate the following code:
const int param1_offset = 0;
const int param2_offset = 1;
const int param3_offset = 3;
const int param4_offset = 7;
or
enum {
param1_offset = 0,
param2_offset = 1,
param3_offset = 3,
param4_offset = 7,
}
or even (not possible using C-preprocessor only for sure, but who knows ;)
#define param1_offset 0
#define param2_offset 1
#define param3_offset 3
#define param4_offset 7
Is it possible to do without running external awk/bash/... scripts?
I'm using Keil C51
It seems I've found a solution with enum:
#define MEM_OFFSET(name, size) \
name ## _offset, \
___tmp__ ## name = name ## _offset + size - 1, // allocate right bound offset and introduce a gap to force compiler to use next available offset
enum {
MEM_OFFSET(param1, 1)
MEM_OFFSET(param2, 2)
MEM_OFFSET(param3, 4)
MEM_OFFSET(param4, 1)
};
In the comments to your post you mention that you're managing an EEPROM memory map, so this answer relates to managing memory offsets rather than answering your specific question.
One way to manage EEPROM memory is with the use of a packed struct. ie, one where there is no space between each of the elements. The struct is never instantiated, it is only used for offset calculations.
typedef struct {
uint8_t param1;
#ifdef FEATURE_ENABLED
uint16_t param2;
#endif
uint8_t param3;
} __packed eeprom_memory_layout_t;
You could then use code like the following to determine the offset of each element as needed(untested). This uses the offsetof stddef macro.
uint16_t read_param3(void) {
uint8_t buf;
eeprom_memory_layout_t * ee;
/* eeprom_read(offset, size, buf) */
eeprom_read(offsetof(eeprom_memory_layout_t, param3), sizeof(ee->param3), &buf);
return buf;
}
Note that the struct is never instantiated. Using a struct like this makes it easy to see your memory map at a glance, and macros can easily be used to abstract away the calls to offsetof and sizeof during access.
If you want to create several structures based on some preprocessor declarations, you could do something like:
#define OFFSET_FOREACH(MODIFIER) \
MODIFIER(1) \
MODIFIER(2) \
MODIFIER(3) \
MODIFIER(4)
#define OFFSET_MODIFIER_ENUM(NUM) param##NUM##_offset,
enum
{
OFFSET_FOREACH(OFFSET_MODIFIER_ENUM)
};
The preprocessor would then produce the following code:
enum
{
param1_offset,
param2_offset,
param3_offset,
param4_offset,
}
I'm sure somebody will figure a nice preprocessor trick to compute the offset values with the sum of its predecessors :)
If you are doing this in C code, you have to keep in mind that const int declarations do not declare constants in C. To declare a named constant you have to use either enum or #define.
If you need int constants specifically, then enum will work well, although I the auto-generation part might be tricky in any case. Off the top of my head I can only come up with something as ugly as
#define MEM_OFFSET_BEGIN(name, size)\
enum {\
name##_OFFSET = 0,\
name##_SIZE__ = size,
#define MEM_OFFSET(name, size, prev_name)\
name##_OFFSET = prev_name##_OFFSET + prev_name##_SIZE__,\
name##_SIZE__ = size,
#define MEM_OFFSET_END()\
};
and then
MEM_OFFSET_BEGIN(param1, 1)
MEM_OFFSET(param2, 2, param1)
MEM_OFFSET(param3, 4, param2)
MEM_OFFSET(param4, 1, param3)
MEM_OFFSET_END()
Needless to say, the fact that it requires the next offset declaration to refer to the previous offset declaration by name defeats most of the purpose of this construct.
Try something like:
#define OFFSET(x) offsetof(struct {\
char param1[1], param2[2], param3[4], param4[1];\
},x)
Then you can use OFFSET(param1), etc. and it's even an integer constant expression.

Hooking mmap system to provide real-time type conversion?

I'm working on some stuff where I want to memory map some large files containing numeric data. The problem is that the data can be a number of formats, including real byte/short/int/long/float/double and complex byte/short/int/long/float/double. Naturally handling all those types all the time quickly gets unwieldy, so I was thinking of implementing a memory mapping interface that can do real-time type conversion for the user.
I really like the idea of mapping a file so you get a pointer in memory back, doing whatever you need and then unmapping it. No bufferology or anything else needed. So a function that reads the data and does the type conversion for me would take a lot away from that.
I was thinking I could memory map the file being operated on, and then simultaneously mapping an anonymous file, and somehow catching page fetches/stores and doing the type conversion on demand. I'll be working on 64-bit so this would give you a 63-bit address space in these cases, but oh well.
Does anyone know if this sort of mmap hooking would be possible, and if so, how might it be accomplished?
Yes(-ish). You can create inaccessible mmap regions. Whenever anybody tries to touch one, handle the SIGSEGV raised by fixing its permissions, filling it, and resuming.
long *long_view =
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
double *double_view =
mmap(NULL, 4096, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
static void on_segv(int signum, siginfo_t *info, void *data) {
void *addr = info->si_addr;
if ((uintptr_t)addr - (uintptr_t)long_view < 4096) {
mprotect(long_view, 4096, PROT_READ|PROT_WRITE);
/* translate from double_view to long_view */
mprotect(double_view, 4096, PROT_NONE);
} else if ((uintptr_t)addr - (uintptr_t)double_view < 4096) {
mprotect(double_view, 4096, PROT_READ|PROT_WRITE);
/* translate from long_view to long_view */
mprotect(double_view, 4096, PROT_NONE);
} else {
abort();
}
}
struct sigaction segv_action = {
.sa_sigaction = on_segv,
.sa_flags = SA_RESTART | SA_SIGINFO,
};
sigaction(SIGSEGV, &segv_action, NULL);
long_view[0] = 42;
/* hopefully, this will trigger the code to fixup double_view and resume */
printf("%g\n", double_view[0]);
(Untested, but something along these lines ought to work...)
If you don't want to fill a whole page at once, that's still doable I think... the third argument can be cast to a ucontext_t *, with which you can decode the instruction being executed and fix it up as if it had performed the expected operation, while leaving the memorry PROT_NONE to catch further accesses... but it'll be a lot slower since you're trapping every access rather than just the first.
The reading part sounds somewhat doable to me. I have no experience in that, but in principle having a signal handler fetch your data and translate it as soon as you access a page that is not yet present in your user-presented buffer should be possible. But it could be that such a thing would be quite inefficient, you'd have a context switch on every page.
The other way round would be much harder I guess. Per default writes are asynchronous, so it will be difficult to capture them.
So "half" of what you want could be possible: always write the data in a new file in the format that the user wants, but translate it automatically on the fly when reading such a file.
But what would be much more important for you, I think, that you have a clear semantic on your different storage representation, and that you encapsulate a read or write of a data item properly. If you have such an interface (something like "store element E at position i with type T") you could easily trigger the conversion with respect to the target format.
Is there a reason why you're not using accessor functions?
There are two basic cases: structured data, and plain data. Structured data has mixed data types, plain data only one format of the ones you listed. You could also have support for transparent endianness correction, if you can store a prototype value (with distinct byte components) for each used type (28 or 30 bytes total for all types you listed -- complex types are just pairs, and have the same byte order as the base components). I have used this approach to store and access atom data from molecular dynamics simulations, and is quite efficient in practice -- the fastest portable one I've tested.
I would use a structure to describe the "file" (the backing file, memory map, endianness, and data format if plain data):
struct file {
int descriptor;
size_t size;
unsigned char *data;
unsigned int endian; /* Relative to current architecture */
unsigned int format; /* endian | format, for unstructured files */
};
#define ENDIAN_I16_MASK 0x0001
#define ENDIAN_I16_12 0x0000
#define ENDIAN_I16_21 0x0001
#define ENDIAN_I32_MASK 0x0006
#define ENDIAN_I32_1234 0x0000
#define ENDIAN_I32_4321 0x0002
#define ENDIAN_I32_2143 0x0004
#define ENDIAN_I32_3412 0x0006
#define ENDIAN_I64_MASK 0x0018
#define ENDIAN_I64_1234 0x0000
#define ENDIAN_I64_4321 0x0008
#define ENDIAN_I64_2143 0x0010
#define ENDIAN_I64_3412 0x0018
#define ENDIAN_F16_MASK 0x0020
#define ENDIAN_F16_12 0x0000
#define ENDIAN_F16_21 0x0020
#define ENDIAN_F32_MASK 0x00C0
#define ENDIAN_F32_1234 0x0000
#define ENDIAN_F32_4321 0x0040
#define ENDIAN_F32_2143 0x0080
#define ENDIAN_F32_3412 0x00C0
#define ENDIAN_F64_MASK 0x0300
#define ENDIAN_F64_1234 0x0000
#define ENDIAN_F64_4321 0x0100
#define ENDIAN_F64_2143 0x0200
#define ENDIAN_F64_3412 0x0300
#define FORMAT_MASK 0xF000
#define FORMAT_I8 0x1000
#define FORMAT_I16 0x2000
#define FORMAT_I32 0x3000
#define FORMAT_I64 0x4000
#define FORMAT_P8 0x5000 /* I8 pair ("complex I8") */
#define FORMAT_P16 0x6000 /* I16 pair ("complex I16") */
#define FORMAT_P32 0x7000 /* I32 pair ("complex I32") */
#define FORMAT_P64 0x8000 /* I64 pair ("complex I64") */
#define FORMAT_R16 0x9000 /* BINARY16 IEEE-754 floating-point */
#define FORMAT_R32 0xA000 /* BINARY32 IEEE-754 floating-point */
#define FORMAT_R64 0xB000 /* BINARY64 IEEE-754 floating-point */
#define FORMAT_C16 0xC000 /* BINARY16 IEEE-754 complex */
#define FORMAT_C32 0xD000 /* BINARY32 IEEE-754 complex */
#define FORMAT_C64 0xE000 /* BINARY64 IEEE-754 complex */
The accessor functions can be implemented in various ways. In Linux, functions marked static inline are as fast as macros, too.
Since the double type does not fully cover 64-bit integer types (since it has only 52 bits in the mantissa), I'd define a number structure,
#include <stdint.h>
struct number {
int64_t ireal;
int64_t iimag;
double freal;
double fimag;
};
and have the accessor functions always fill in the four fields. Using GCC, you can also create a macro to define a struct number using automatic type detection:
#define Number(x) \
( __builtin_types_compatible_p(__typeof__ (x), double) ? number_double(x) : \
__builtin_types_compatible_p(__typeof__ (x), _Complex double) ? number_complex_double(x) : \
__builtin_types_compatible_p(__typeof__ (x), _Complex long) ? number_complex_long(x) : \
number_int64(x) )
static inline struct number number_int64(const int64_t x)
{
return (struct number){ .ireal = (int64_t)x,
.iimag = 0,
.freal = (double)x,
.fimag = 0.0 };
}
static inline struct number number_double(const double x)
{
return (struct number){ .ireal = (int64_t)x,
.iimag = 0,
.freal = x,
.fimag = 0.0 };
}
static inline struct number number_complex_long(const _Complex long x)
{
return (struct number){ .ireal = (int64_t)(__real__ (x)),
.iimag = (int64_t)(__imag__ (x)),
.freal = (double)(__real__ (x)),
.fimag = (double)(__imag__ (x)) };
}
static inline struct number number_complex_double(const _Complex double x)
{
return (struct number){ .ireal = (int64_t)(__real__ (x)),
.iimag = (int64_t)(__imag__ (x)),
.freal = __real__ (x),
.fimag = __imag__ (x) };
}
This means that Number(value) constructs a correct struct number as long as value is an integer or floating-point real or complex type.
Note how the integer and floating-point components are set to the same values, as far as type conversions allow. (For very large integers in magnitude, the floating-point value will be an approximation. You could also use (int64_t)round(...) to round instead of truncate the floating-point parameter, when setting the integer component(s).
You'll need four accessor functions: Two for structured data, and two for unstructured data. For unstructured (plain) data:
static inline struct number file_get_number(const struct file *const file,
const size_t offset)
{
...
}
static inline void file_set_number(const struct file *const file,
const size_t offset,
const struct number number)
{
...
}
Note that offset above is not the byte offset, but the index of the number. For a structured file, you'll need to use the byte offset, and add a parameter specifying the number format used in the file:
static inline struct number file_get(const struct file *const file,
const size_t byteoffset,
const unsigned int format)
{
...
}
static inline void file_set(const struct file *const file,
const size_t byteoffset,
const unsigned int format,
const struct number number)
{
...
}
The conversions needed in the function bodies I omitted (...) are quite straightforward. There are a few tricks you can do for optimization, too. For example, I like to adjust the endianness constants so that the low bit is always a byte swap (ab -> ba, abcd -> badc, abcdefgh -> badcfehg), and the high bit is a short swap (abcd -> cdab, abcdefgh ->cdabghef). You might need a third bit for 64-bit values (abcdefgh -> efghabcd), if you want to be completely certain.
The if or case statements within the function body do cause a small access overhead, but it should be small enough to ignore in practice. All ways to avoid it will lead to much more complex code. (For maximum throughput, you'd need to open-code all access variants, and use __builtin_types_compatible_p() in a function or macro to determine the correct one to use. If you consider the endianness conversion, that means quite a few functions. I believe the very small access overhead -- a few clocks at most per access -- is much more preferable. (All my tests have been I/O bound anyway, even at 200 Mb/s, so to me the overhead is completely irrelevant.)
In general, for automatic endianness conversion using prototype values you simply test each possible conversion for the type. As long as each byte component of the prototype values are unique, then only one conversion will produce the correct expected value. On some architectures integers and floating-point values have different endianness; this is why the ENDIAN_ constants are for each type and size separately.
Assuming you have implemented all of the above, in your application code the data access would look something like
struct file f;
/* Set first number to zero. */
file_set_number(&f, 0, Number(0));
/* Set second number according to variable v,
* which can be just about any numeric type. */
file_set_number(&f, 1, Number(v));
I hope you find this useful.

Resources