I'm investigating the Linux kernel (specifically the load balance area).
in the kernel (sched.h) there is a declaration of a struct :
struct sched_group
which looks like this:
struct sched_group {
struct sched_group *next; /* Must be a circular list */
atomic_t ref;
unsigned int group_weight;
struct sched_group_power *sgp;
/*
* The CPUs this group covers.
*
* NOTE: this field is variable length. (Allocated dynamically
* by attaching extra space to the end of the structure,
* depending on how many CPUs the kernel has booted up with)
*/
unsigned long cpumask[0];
};
what I don't understand is the use of a cpumask array with the size 0.
any explanation would be much appreciated :)
The size of cpumask should be variant based on different platforms that has different number of CPUs, that's why it cannot define a fixed length array. GNU C supports a variable-length object, it should be the last element of a structure.
Related
May be you can advise me how to create a hierarchical structure in OpenCL. It is easy if you have "new" or "malloc", but I don't know how do it in GPGPU. So I created 3 kernels:
To send sizeof struct to host.
To initialize data.
And kernel which uses data initialized in second kernel.
I have this struct in OpenCL:
typedef struct some some;
struct some{
char data[4];
some* children[8];
};
First kernel says that size of this structure is 36 bytes(4 for data and 32 for pointers).
Next I allocate memory on GPU based on previous information and call second kernel:
kernel void import(global some *buffer){
for(int i=0;i<4;i++){
buffer[0].data[i]=255; //For example, doesn't matter
}
//Now I need to assign pointer to next element of array(buffer) to first element
buffer[0].children[0]=&buffer[1];
}
But kernel not compiles. Also I tried:
*buffer[0].children[0]=buffer[0];
It compiles, but crashes of course. It is logically wrong) Without assignments of pointers everything works fine. Very cool program for 1 element)
Try using offsets or array indices instead of pointers.
typedef struct some some;
struct some{
char data[4];
size_t children[8]; // an array of subscripts
};
...
// buffer[0].children[0]=&buffer[1]; becomes
buffer[0].children[0] = 1;
So now you can reference a child via its subscript
buffer[ buffer[0].children[0] ].char[0]
If your device supports OpenCL 2.0 then you can use Shared Virtual Memory. The pointers created on the host will be valid on the device too.
The description of Shared Virtual Memory concept and the examples you can find here and here.
Is it possible to use a structure with a pointer to another structure inside a memory mapped file instead of storing the offset in some integral type and calculate the pointer?
e.g. given following struct:
typedef struct _myStruct_t {
int number;
struct _myStruct_t *next;
} myStruct_t;
myStruct_t* first = (myStruct_t*)mapViewHandle;
myStruct_t* next = first->next;
instead of this:
typedef struct _myStruct_t {
int number;
int next;
} myStruct_t;
myStruct_t* first = (myStruct_t*)mappedFileHandle;
myStruct_t* next = (myStruct_t*)(mappedFileHandle+first->next);
I read about '__based' keyword, but this is Microsoft specific and therefore Windows-bound.
Looking for something working with GCC compiler.
I'm pretty sure there's nothing akin to the __based pointer from Visual Studio in GCC. The only time I'd seen anything like that built-in was on some pretty odd hardware. The Visual Studio extension provides an address translation layer around all operations involving the pointer.
So it sounds like you're into roll-your-own territory; although I'm willing to be told otherwise.
The last time I was dealing with something like this it was on the palm platform, where, unless you locked down memory, there was the possibility of it being moved around. You got memory handles from allocations and you had to MemHandleLock before you used it, and MemPtrUnlock it after you were finished using it so the block could be moved around by the OS (which seemed to happen on ARM based palm devices).
If you're insistent on storing pointer-esque values in a memory mapped structure the first recommendation would be to store the value in an intptr_t, which is an int size that can contain a pointer value. While your offsets are unlikely to exceed 4GB, it pays to stay safe.
That said, this is probably easy to implement in C++ using a template class, it's just that marking the question as C makes things a lot messier.
C++: It is very doable and portable (the code, but maybe not the data).
It was a while ago, but I created a template for a self-relative pointer classes.
I had tree structures inside blocks of memory that might move.
Internally, the class had a single intptr_t, but = * . -> operators were overloaded so it appeared like a regular pointer. Handling null took some attention.
I also did versions using int, short and not very useful char for space-saving pointers that were unable to point far away (outside memory block).
In C you could use macros to wrap get and set
// typedef OBJ { int p; } OBJ;
#define OBJPTR(P) ((OBJ*)((P)?(int)&(P)+(P):0))
#define SETOBJPTR(P,V) ((P)=(V)?(int)(V)-(int)&(P):0)
The above C macros are for self-relative pointers that can be slightly more efficient than based pointers.
Here is a working example of a tree in a small block of relocatable memory using 2-byte (short) pointers to save space. int is okay for casting from pointers since it is 32 bit code:
#include <stdio.h>
#include <memory.h>
typedef struct OBJ
{
int val;
short left;
short right;
#define OBJPTR(P) ((OBJ*)((P)?(int)&(P)+(P):0))
#define SETOBJPTR(P,V) ((P)=(V)?(int)(V)-(int)&(P):0)
} OBJ;
typedef struct HEAD
{
short top; // top of tree
short available; // index of next available place in data block
char data[0x7FFF]; // put whole tree here
} HEAD;
HEAD * blk;
OBJ * Add(int val)
{
short * where = &blk->top; // find pointer to "pointer" to place new node
OBJ * nd;
while ( ( nd = OBJPTR(*where) ) != 0 )
where = val < nd->val ? &nd->left : &nd->right;
nd = (OBJ*) ( blk->data + blk->available ); // allocate node
blk->available += sizeof(OBJ); // finish allocation
nd->val = val;
nd->left = nd->right = 0;
SETOBJPTR( *where, nd );
return nd;
}
void Dump(OBJ*top,int indent)
{
if ( ! top ) return;
Dump( OBJPTR(top->left), indent + 3 );
printf( "%*s %d\n", indent, "", top->val );
Dump( OBJPTR(top->right), indent + 3 );
}
void main(int argc,char*argv)
{
blk = (HEAD*) malloc(sizeof(HEAD));
blk->available = (int) &blk->data - (int) blk;
blk->top = 0;
Add(23); Add(2); Add(45); Add(99); Add(0); Add(12);
Dump( OBJPTR(blk->top), 3 );
{ // PROOF a copy at a different address still has the tree:
HEAD blk2 = *blk;
Dump( OBJPTR(blk2.top), 3 );
}
}
A note about based verses self-relative "*" operator.
Based can involve 2 addresses and 2 memory fetches.
Self-relative involves 1 address and 1 memory fetch.
Pseudo assembly:
load reg1,address of pointer
load reg2,fetch reg1
add reg3,reg2+reg1
load reg1,address of pointer
load reg2,fetch reg1
load reg3,address of base
load reg4,fetch base
add reg5,reg2+reg4
The first is extremely unlikely to work.
Remember that a pointer, such as struct _myStruct_t * is a pointer to a location in memory. Suppose that this structure was located at address 1000 in memory: that would mean that the next structure, located just after it, might be located at address 1008, and that's what's stored in ->next (the numbers don't matter; what matters is that they are memory addresses). Now you save that structure to a file (or un-map it). Then you map it again, but this time, it ends up starting at address 2000, but the ->next pointer is still 1008.
You have (generally) no control over where files are mapped in memory, so no control over the actual memory locations of the elements within the mapped structure. Therefore you can only depend on relative offsets.
Note that your second version may or may not work as you expect, depending on the declared type of mappedFileHandle. If it's a pointer to myStruct_t, then adding an integer n to it will produce a pointer to an address which is n*sizeof(myStruct_t) bytes higher in memory (as opposed to being n bytes higher).
If you declared mappedFileHandle as
myStruct_t* mappedFileHandle;
then you can subscript it like an array. If the mapped file is laid out as a sequence of myStruct_t blocks, and the next field refers to other blocks by index within that sequence, then (supposing myStruct_t* b is a block of interest)
mappedFileHandle[b->next].number
is the number field of the b->nextth block in the sequence.
(This is just a consequence of the way that arrays are defined in C: mappedFileHandle[b->next] is defined to be equivalent to *(mappedFileHandle + b->next), which is an object of type myStruct_t, which you can therefore get the number field of).
Given a complex data structure where each sub-structure has a variable that has a domain of {true or false},
(e.g.)
struct dataBlock{
struct {
/* more members */
char status;
} node1;
struct {
/* more members */
char status;
} node2;
/* More nodes */
};
It would be a waste to have 1 byte just for a value of 1 or 0. Is there a C language technique that status in each node will only occupy a bit in a byte? What I can think of is by using MACROS but macros cannot be contained in a local scope right? So having macro status will mean only one macro status in the program. Hence, calling node1.status and node2.status uses the same macro.
You can use a bitfield - this syntax allows you to define how many bits each int in a strcut should occupy.
Note, however, that C can only allocate full bytes, so the size of the struct would be rounded up to the nearest multiplication of 8 bits in any case.
E.g.:
struct {
int whole_int; /* a whole int, let's assume it's 16 bits. */
int half_int : 8; /* only half an int */
int another_half_int : 8;
} some_struct /* Total size is 2 bytes*/
Having said that, I sincerely doubt you'll notice any performance gain from using this technique, and as Fredrick Gauss commented, it's probably not worth the hassle.
C has a built in feature called bit fields that will get the job done.
Basically, bit fields automatically optimizes a structure to use only as much memory as needed for each given member. In your case, you would do something like this.
struct statusNode {
/* ... */
/* only use 1 bit for this member */
unsigned int status : 1;
/* for example, test only needs 4 bits (range of 0 to 15) */
unsigned int test : 4;
};
struct dataBlock {
struct statusNode node1;
/* ... */
struct statusNode node2;
};
You can assign each members a certain number of bits based on the highest value that you'll ever come across.
You can find more information about bit fields here.
Is it possible to create a generic Vector like data structure in C, with out using heap. Basically I need a array data type but a more generalized version on if it.
typedef struct {
/* some data types*/
}TYPE1;
typedef struct {
/* some data types*/
}TYPE2;
typedef struct _GCACHE_T
{
const int element_size;
const int count;
struct _ELEMENT {
UBYTE data[element_size];
BOOLEAN is_valid;
}element[count];
}GCACHE_T;
GCACHE_T f_cache1 = {sizeof(TYPE1), 15, {0} };
GCACHE_T f_cache2 = {sizeof(TYPE2), 10, {0} };
The above code will not compile but I have provided it for a better clarity on my requirement.
This would have been easy implemted provided heap memory was allowed to use. Since the code is meant for small micros heap memory usage is not allowed.
I could have used right away, but just checking if it can be done in a generic way.
TYPE1 f_cache1[15];
TYPE2 f_cache2[10];
The Vector will not grow in size. I could have also used a union but there is a memory trade off so not willing to use it.
Such parametric (template, generic) types are not supported by C. You can take an approach similar to the one used by the BSD socket subsystem. There different network addresses (e.g. IP address and TCP/UDP port number) are stored in structures of varying size (depending on the address family, e.g. IPv4 structures are shorter than IPv6 ones) but with similar layout in the beginning. Whenever an address is required, a pointer to the generic struct sockaddr type is passed instead and the correct structure type is inferred from the address family of the socket.
C supports the so-called flexible array members, but it cannot be simply applied to your case because not only is the number of struct _ELEMENT entries different, but the size of those elements could differ depending on the value of element_size. This makes it hard to compute the address of cache.element[i].data[j] in a portable way whithout refering to the actual type. What you can do is put an additional field in the beginning of the GCACHE_T type that helps you identify the true size of struct _ELEMENT:
typedef struct _GCACHE_T
{
int element_size;
int count;
size_t element_stride;
struct _ELEMENT {
BOOLEAN is_valid;
UBYTE data[];
} element[];
} GCACHE_T;
element_stride keeps the size of the concrete element type (including any padding). Note that is_valid is moved before data[] as C allows only the last element of a structure to be a flexible one.
You would then create specific types, e.g.
typedef struct _GCACHE_TYPE1_15_T
{
int element_size;
int count;
size_t element_stride;
struct {
BOOLEAN is_valid;
UBYTE data[sizeof(TYPE1)];
} element[15];
} GCACHE_TYPE1_15_T;
GCACHE_TYPE1_15_T f_cache1 = {
sizeof(TYPE1),
15,
// An awful hack to obtain the size of a structure member
sizeof(((GCACHE_TYPE1_15_T *)0)->element[0])
};
do_something((GCACHE_T *)&f_cache1);
Macros would come handy if you need to declare many different cache types. Now in do_something() you can compute the address of f_cache1.element[i].data[j] because you know the offset of the data field inside struct _ELEMENT and you can compute the offset of element[i] because the size of a single element is stored in the element_stride field.
Yeah, I know, it is a real pain... And I am not sure how much of the pointer arithmetic required works on a Harvard architecture device like PIC.
edit: a better way of phrasing this: What's the correct [modern] way to ensure that a struct is a specific size in bytes?
just spending a relaxing saturday afternoon debugging a legacy codebase, and having a bit of trouble figuring this out. The compiler error I get is this:
INC/flx.h:33: error: dereferencing pointer to incomplete type
the code at line 33 looks like this
typedef struct flx_head {
FHEAD_COMMON;
LONG frames_in_table; /* size of index */
LONG index_oset; /* offset to index */
LONG path_oset; /* offset to flipath record chunk */
/* this will insure that a Flx_head is the same size as a fli_head but won't
* work if there is < 2 bytes left (value <= 0) */
PADTO(sizeof(Fli_head),flx_head,flxpad); /* line 33 is this one */
} Flx_head;
well okay so I can see that the struct is referring to itself to pad it out somehow. But I don't know an alternative way of doing what PADTO does without the self reference.
here's what PADTO is defined as
#define MEMBER(struc,field) \
((struc*)NULL)->field
/* returns offset of field within a given struct name,
* and field name ie: OFFSET(struct sname,fieldname) */
#define OFFSET(struc,field) \
(USHORT)((ULONG)((PTR)&MEMBER(struc,field)-(PTR)NULL))
/* offset to first byte after a field */
#define POSTOSET(struc,field) \
(OFFSET(struc,field)+sizeof(MEMBER(struc,field)))
/* macro for defining pad sizes in structures can not define a pad of
* less than two bytes one may use pname for the offset to it but
* sizeof(struc->pname) will not be valid
*
* struct sname {
* char fld1[64];
* PADTO(68,sname,pname);
* };
* will make:
*
* struct sname {
* char fld1[64];
* UBYTE pname[1];
* UBYTE __pname[3];
* };
*/
#define PADTO(sz,struc,padfld) \
UBYTE padfld[1];UBYTE __##padfld[(sz)-OFFSET(struct struc,padfld)-1]
here is FHEAD_COMMON
#define FHEAD_COMMON \
CHUNKID_FIELDS;\
USHORT frame_count;\
USHORT width;\
USHORT height;\
USHORT bits_a_pixel;\
SHORT flags;\
LONG speed;\
USHORT unused;\
Fli_id id;\
USHORT aspect_dx;\
USHORT aspect_dy;\
UBYTE commonpad[38] /* should be total of 80 bytes (48 for unique) */
and flihead
typedef struct fli_head {
FHEAD_COMMON;
LONG frame1_oset;
LONG frame2_oset;
UBYTE padfill[40];
} Fli_head;
this is Autodesk animator pro. what I am working on is the "reference" implementation for the FLI file format- which you can see a spec for here:
http://www.compuphase.com/flic.htm
Incidentally, I'm pretty sure that what the /source code/ there refers to as "flx" is actually what that webpage there calls "flc" , not what it calls "flx"
update:
better source for format info http://drdobbs.com/architecture-and-design/184408954
It isn't pretty, but one possibility is to define another identical structure and use its size to determine the padding for the one you actually want to use:
#define FLX_HEAD \
FHEAD_COMMON;\
LONG frames_in_table; /* size of index */ \
LONG index_oset; /* offset to index */ \
LONG path_oset /* offset to flipath record chunk */
struct flx_head_unpadded {
FLX_HEAD;
};
typedef struct flx_head {
FLX_HEAD;
char __flxpad[sizeof(Fli_head)-sizeof(struct flx_head_unpadded)];
} Flx_head;
I suppose the answer depends on what you're trying to achieve. In most cases, the correct, modern way to pad a struct is not to. The only situation I can think of where it's legitimate to pad a struct is when you have a library interface where the caller creates objects of a structure type and passes pointers to them to the library, and where you want to leave room to add additional fields to the structure without breaking the ABI. In this case, I would start out with something like char pad[256]; and change it to char pad[256-3*sizeof(long)]; or similar as you add fields (making sure to avoid internal padding when you add fields).
Define it in a union with a byte/char array of the desired size?
I can think quickly of some scenarios where this is needed:
1) Compatibility with old software that uses flat binary files to store data, (as in OP).
2) Interaction with drivers and/or hardware
3) Forcing structs to be an exact multiple of the cache line size to prevent false sharing in inter-thread comms.
If you only want to achieve specific size you can use (for sure working in GCC):
typedef union {
struct {
FHEAD_COMMON;
LONG frames_in_table; /* size of index */
LONG index_oset; /* offset to index */
LONG path_oset; /* offset to flipath record chunk */
};
uint8_t __padding[128];
} Flx_head;
void test() {
Flx_head boo;
boo.frames_in_table= 0;
}
I am not sure if this is modern enough. If youc compiler does not support anonymous struct (the one inside union) it will get "messy".
Also you must remember that struct is now padded, but not aligned to specific data size.
thanks everyone. not sure who to award the green checkmark to, since I found this solution as a result of everyone kind of hinting and pointing in the right direction. After looking at the problem, it struck me that the struct just needs to be exactly 128 bytes. I "hand parsed" the macro, cross referencing with the spec and ended up with this:
typedef struct flx_head {
FHEAD_COMMON;
LONG frames_in_table; /* size of index */
LONG index_oset; /* offset to index */
LONG path_oset; /* offset to flipath record chunk */
UBYTE flxpad[36];
} Flx_head;
which is 128-(80+4+4+4) = 36