Is it possible to use a structure with a pointer to another structure inside a memory mapped file instead of storing the offset in some integral type and calculate the pointer?
e.g. given following struct:
typedef struct _myStruct_t {
int number;
struct _myStruct_t *next;
} myStruct_t;
myStruct_t* first = (myStruct_t*)mapViewHandle;
myStruct_t* next = first->next;
instead of this:
typedef struct _myStruct_t {
int number;
int next;
} myStruct_t;
myStruct_t* first = (myStruct_t*)mappedFileHandle;
myStruct_t* next = (myStruct_t*)(mappedFileHandle+first->next);
I read about '__based' keyword, but this is Microsoft specific and therefore Windows-bound.
Looking for something working with GCC compiler.
I'm pretty sure there's nothing akin to the __based pointer from Visual Studio in GCC. The only time I'd seen anything like that built-in was on some pretty odd hardware. The Visual Studio extension provides an address translation layer around all operations involving the pointer.
So it sounds like you're into roll-your-own territory; although I'm willing to be told otherwise.
The last time I was dealing with something like this it was on the palm platform, where, unless you locked down memory, there was the possibility of it being moved around. You got memory handles from allocations and you had to MemHandleLock before you used it, and MemPtrUnlock it after you were finished using it so the block could be moved around by the OS (which seemed to happen on ARM based palm devices).
If you're insistent on storing pointer-esque values in a memory mapped structure the first recommendation would be to store the value in an intptr_t, which is an int size that can contain a pointer value. While your offsets are unlikely to exceed 4GB, it pays to stay safe.
That said, this is probably easy to implement in C++ using a template class, it's just that marking the question as C makes things a lot messier.
C++: It is very doable and portable (the code, but maybe not the data).
It was a while ago, but I created a template for a self-relative pointer classes.
I had tree structures inside blocks of memory that might move.
Internally, the class had a single intptr_t, but = * . -> operators were overloaded so it appeared like a regular pointer. Handling null took some attention.
I also did versions using int, short and not very useful char for space-saving pointers that were unable to point far away (outside memory block).
In C you could use macros to wrap get and set
// typedef OBJ { int p; } OBJ;
#define OBJPTR(P) ((OBJ*)((P)?(int)&(P)+(P):0))
#define SETOBJPTR(P,V) ((P)=(V)?(int)(V)-(int)&(P):0)
The above C macros are for self-relative pointers that can be slightly more efficient than based pointers.
Here is a working example of a tree in a small block of relocatable memory using 2-byte (short) pointers to save space. int is okay for casting from pointers since it is 32 bit code:
#include <stdio.h>
#include <memory.h>
typedef struct OBJ
{
int val;
short left;
short right;
#define OBJPTR(P) ((OBJ*)((P)?(int)&(P)+(P):0))
#define SETOBJPTR(P,V) ((P)=(V)?(int)(V)-(int)&(P):0)
} OBJ;
typedef struct HEAD
{
short top; // top of tree
short available; // index of next available place in data block
char data[0x7FFF]; // put whole tree here
} HEAD;
HEAD * blk;
OBJ * Add(int val)
{
short * where = &blk->top; // find pointer to "pointer" to place new node
OBJ * nd;
while ( ( nd = OBJPTR(*where) ) != 0 )
where = val < nd->val ? &nd->left : &nd->right;
nd = (OBJ*) ( blk->data + blk->available ); // allocate node
blk->available += sizeof(OBJ); // finish allocation
nd->val = val;
nd->left = nd->right = 0;
SETOBJPTR( *where, nd );
return nd;
}
void Dump(OBJ*top,int indent)
{
if ( ! top ) return;
Dump( OBJPTR(top->left), indent + 3 );
printf( "%*s %d\n", indent, "", top->val );
Dump( OBJPTR(top->right), indent + 3 );
}
void main(int argc,char*argv)
{
blk = (HEAD*) malloc(sizeof(HEAD));
blk->available = (int) &blk->data - (int) blk;
blk->top = 0;
Add(23); Add(2); Add(45); Add(99); Add(0); Add(12);
Dump( OBJPTR(blk->top), 3 );
{ // PROOF a copy at a different address still has the tree:
HEAD blk2 = *blk;
Dump( OBJPTR(blk2.top), 3 );
}
}
A note about based verses self-relative "*" operator.
Based can involve 2 addresses and 2 memory fetches.
Self-relative involves 1 address and 1 memory fetch.
Pseudo assembly:
load reg1,address of pointer
load reg2,fetch reg1
add reg3,reg2+reg1
load reg1,address of pointer
load reg2,fetch reg1
load reg3,address of base
load reg4,fetch base
add reg5,reg2+reg4
The first is extremely unlikely to work.
Remember that a pointer, such as struct _myStruct_t * is a pointer to a location in memory. Suppose that this structure was located at address 1000 in memory: that would mean that the next structure, located just after it, might be located at address 1008, and that's what's stored in ->next (the numbers don't matter; what matters is that they are memory addresses). Now you save that structure to a file (or un-map it). Then you map it again, but this time, it ends up starting at address 2000, but the ->next pointer is still 1008.
You have (generally) no control over where files are mapped in memory, so no control over the actual memory locations of the elements within the mapped structure. Therefore you can only depend on relative offsets.
Note that your second version may or may not work as you expect, depending on the declared type of mappedFileHandle. If it's a pointer to myStruct_t, then adding an integer n to it will produce a pointer to an address which is n*sizeof(myStruct_t) bytes higher in memory (as opposed to being n bytes higher).
If you declared mappedFileHandle as
myStruct_t* mappedFileHandle;
then you can subscript it like an array. If the mapped file is laid out as a sequence of myStruct_t blocks, and the next field refers to other blocks by index within that sequence, then (supposing myStruct_t* b is a block of interest)
mappedFileHandle[b->next].number
is the number field of the b->nextth block in the sequence.
(This is just a consequence of the way that arrays are defined in C: mappedFileHandle[b->next] is defined to be equivalent to *(mappedFileHandle + b->next), which is an object of type myStruct_t, which you can therefore get the number field of).
Related
struct s{
int arr[10];
};
void func(struct s);
Now when I pass a structure variable of struct s type the whole array gets copied.
struct s demo;
func(demo){
demo.arr[0] = 20;
}
I want to know how this works i.e. what is happening behind the scenes as in does the whole array gets copied in actual etc.
I am looking for an answer from compiler's view. How does the compiler implement it.
It does get copied by value indeed, there's nothing else to it. This is actually the only way of copying arrays per assignment in C.
But this means that the whole array will get copied onto the stack when you call func. That's most often a bad idea, because it takes execution time to copy data and it takes up a lot of memory as well.
Normally, the rule of thumb is to always pass structs through pointers: void func(struct s* data);, because this goes way faster, takes up less memory and allows you to modify the original struct from inside the function.
Both approaches work: passing full array or struct object, and passing addresses only. But passing a pointer is less costly and more flexible than passing the object in its entirety.
Passing pointers is:
Less costly because when passing the full array, or struct variable the entire memory content for either is copied to a new location on the stack. And because typically struct and array variable are created as collections of large amounts related data, the cost benefit can be substantial. The size of data copied will determine how long it takes to copy, and how much memory will be used to accommodate.
Passing a pointer to either data type, no matter how much data the variable is defined to contain, will only cost the size of a pointer. If targeting 32bit addressing, the size of a pointer variable will be 4 bytes. If targeting 64bit addressing, then the cost is 8 bytes.
More flexible because, for these data types in particular, designing your code to pass pointers adds the ability to add struct members, or array elements without impacting the prototype of the functions that accept them as arguments. For example, the following function prototype will accept both of the following struct definitions:
void acceptStructPointer(S *data);
Will accept either struct definition without impact:
typedef struct {
int val[10];
}S;
Or:
typedef struct {
int val[10];
float b[100];
char string[100];
}S;
Additionally, when memory needs are not know until run-time, for example when reading from a data base, or when spawning multiple instances of socket sessions, passing pointers means that memory needs can be sized based on actual run-time needs:
void acceptStructPointer(S *data)
{
...
data = malloc(someDemand*sizeof(S));
if(data)
{
....
The following is a small code snippet showing in particular the size/speed advantage of passing pointers. Note that the larger, and/or more complex the data object, the bigger the advantage becomes in terms of run-time speed and memory usage.
#define ARY_SIZE 10
typedef struct {
int val[10];
}S;
//struct
S sData = {1,2,3,4,5,6,7,8,9,0};
//pointer to struct
S *pSdata = NULL;
//array
int aData[ARY_SIZE] = {9,8,7,6,5,4,3,2,1,0};
//pointer to array
int *pAdata = NULL;
void acceptPointerVaraibles(S *pA, int *pD);
void acceptNonPointerVariables(S a, int d[]);
int main(void)
{
pSdata = &sData;
pAdata = &aData[0];
printf("Size of struct sData: %d\n", sizeof(sData));
printf("Size of struct pSdata: %d\n", sizeof(pSdata));
printf("Size of struct aData: %d\n", sizeof(aData));
printf("Size of struct pAdata: %d\n", sizeof(pAdata));
//passing pointer
acceptPointerVaraibles(pSdata, pAdata);
//passing non pointer
acceptNonPointerVariables(sData, aData);
return 0;
}
void acceptPointerVaraibles(S *pA, int *pD)
{
for(int i=0;i<ARY_SIZE;i++)
{
printf("Value of struct val element %d: %d\n", i, pA->val[i]);
printf("Value of array element %d: %d\n", i, pD[i]);
}
return;
}
void acceptNonPointerVariables(S a, int d[])
{
for(int i=0;i<ARY_SIZE;i++)
{
printf("Value of struct val element %d: %d\n", i, a.val[i]);
printf("Value of array element %d: %d\n", i, d[i]);
}
return;
}
In C you pass arrays around by the use of pointers. You can’t pass the array itself, giving the call a copy of the array, because everything in C is passed by value, there is no passing by reference, not at all. So the pointer you are passing to the callee is passed by value; if you changed this value in the callee, it will not affect the caller.
Your problem will be how to determine the length of the referenced array in the callee. One common way (at least in C) is to end the array using zero (or NULL) value. This is how we know where a string ends. Also, the standard C library uses this feature elsewhere, like the environ array, containing the environment variables, which is an array of key/value pairs that is terminated by a NULL entry. You can use this same technique to mark the end of your arrays.
Another way is passing a second numerical argument that contains the size of the array (or the number of items, to be more accurate). Depending on the case, you might prefer the 1st or the 2nd option. For example, if the function is a recursive function you might want to use the first option, to limit the number of bytes you chop off the stack. But if the array is expected to be large, and you will need to recalculate its length multiple times, you might want to use the second approach, as it will save your computational power and time. It’s the classic space vs time optimization problem.
I'm working my way through the learn c the hard way book and have run into a few issues on Exercise 19. The author said that ex19 was intended for the learners to get to know the macro in c. I have no problem in understanding the concept of that, but I just don't understand everything else. I can't understand how the object prototype is created.
Especilly,what does the following sentense mean?
Since C puts the Room.proto field first, that means the el pointer is
really only pointing at enough of the block of memory to see a full
Object struct. It has no idea that it's even called proto.
the relevant code is this:
// this seems weird, but we can make a struct of one size,
// then point a different pointer at it to "cast" it
Object *el = calloc(1, size);
*el = proto;
can anyone tell me how on earth malloc/calloc exactly works? As far as i know, it just allocate the required number of memory and return the first address. If so, how can the computer know the data struct of the allocated memory? like in the code, after Room *arena = NEW(Room, "The arena, with the minotaur");,you can do this directly arena->bad_guy = NEW(Monster, "The evil minotaur"); how does the computer know there is a bad_guy??
what on earth is the content of *el after the above two statements(Object *el = calloc(1, size); and *el = proto;)?
Any help will be appreciated!!
the link to the exercise: http://c.learncodethehardway.org/book/ex19.html
calloc has the additional feature that it fills the allocated memory with zero bytes, whereas using the equivalent malloc call would require an additional step if all or some of the allocation needs to be zero initially.
In the code
arena->bad_guy = NEW(Monster, "The evil minotaur");
the compiler knows the layout of the struct because the access is through the arena variable, which is declared as a pointer to Room, which is presumably a typedef of a struct.
For the other part, the guarantee of ordering within structs allows a limited form of inheritance in composite structs, or extended structs.
struct A {
int x;
};
struct B {
int foo;
double baloney;
};
struct B (or a pointer to it) can be cast to a (pointer to a) struct A because they both begin with an int. Of course, if you cast the other way, the struct A must have been originally a struct B or access to the baloney field will be undefined. In other words, struct B essentially begins with a struct A.
This may be easier to see if I rewrite my example like this:
struct A {
int x;
};
struct B {
struct A foo;
double baloney;
};
Now you can get a struct A out of struct B in different ways.
struct A a;
struct B b;
a = b.foo; // regular member variable access
struct A *ap = &a;
struct B *bp = &b;
ap = (struct A *)bp; // cast the pointer
ap = & b.foo; // take a pointer from the member variable
ap = & bp->foo; // take a pointer from the member variable via a pointer
All it does is to alloc 1*size bytes. There's nothing magic with malloc/calloc. He is passing the sizeof(T) to the function through that NEW macro and putting it in Object_new's size parameter. So all the function knows is the size in bytes.
Suppose I have the following structs:
typedef struct plane_t Plane;
struct plane_t{
Point p1;
Point p2;
Point p3;
};
typedef struct arrangement_t* Arrangement;
struct arrangement_t{
//TODO add fields here
int maxPlanes;
int curPlanes;
Plane *planes;
};
And I have the following function:
Plane planeCreate(Point point1, Point point2, Point point3){
Plane newPlane = {{point1.x, point1.y, point1.z}, {point2.x, point2.y, point2.z}, {point3.x, point3.y, point3.z}};
return newPlane;
}
Arrangement arrangementCreate(int maxPlanes){
if (maxPlanes < 1) return NULL;
Arrangment newArrangment = malloc(sizeof struct arrangement_t);
if (newArrangment == NULL) return NULL;
newArrangment->planes = malloc(sizeof(Plane)*maxPlanes);
if (newArrangment->planes == NULL) {
free(newArrangment);
return NULL;
}
newArrangment->maxPlanes = maxPlanes;
newArrangment->curPlanes = 0;
return newArrangment;
}
Will the following line mean that every cell within the array will have a stuct of type Plane, or I still have to go over each cell in manually create them one by one?
newArrangment->planes = malloc(sizeof(Plane)*maxPlanes);
malloc will allocate enough space for an array of maxPlanes structs, but it's up to you to initialize them.
In other words, if malloc succeeds, you will be able to access the structs newArrangement->planes[0] through newArrangement->planes[maxPlanes-1]. The structs are laid out end-to-end in memory as one contiguous block.
Your code is probably (see below) fine (by "fine" I mean functional, not well-designed). Your malloc(sizeof(Plane)*maxPlanes) will allocate space for maxPlanes planes, and you do not have to allocate each cell.
The cells themselves don't "have a struct of type Plane", it is merely a block of memory large enough to hold the information stored in maxPlanes Plane structs. Since you are accessing that block via a Plane pointer, the data will be interpreted as a Plane struct.
I say probably because your Plane struct members will be uninitialized (will contain random data), and since you don't show what a Point is, there is not enough information to know if what you have is sufficient. You also do not show what you are going to end up doing with the arrangement_t, so I presume that it is acceptable to leave the values uninitialized and that you are setting the values to something meaningful later.
I'd also make a number of recommendations to clarify your code:
You typedef Arrangement as a pointer but Plane as a struct. I suggest perhaps qualifying your Arrangement type as ArrangementPtr or something instead to make the distinction and reduce confusion.
Consider using calloc instead of malloc for clarity: calloc(maxPlanes, sizeof(Plane))
If zeroing the memory is acceptable to you for initialization, you can do a quick memset(planes, 0, sizeof(Plane)*maxPlanes).
Unless you have a specific reason to use C (there are many), you may wish to consider C++ (with STL classes if they are available to you), which will greatly reduce a lot of the work and possible sources of error in your code (note this is still not ideal because public members can break invariants that you may have, but just as an example):
struct Plane {
Point p1;
Point p2;
Point p3;
Plane ();
Plane (const Point &, const Point &, const Point &);
};
struct Arrangement {
int maxPlanes;
int curPlanes;
std::vector<Plane> planes;
explicit Arrangement (int maxPlanes);
};
Plane::Plane () {
}
Plane::Plane (const Point &p1, const Point &p2, const Point &p3) :
p1(p1), p2(p2), p3(p3)
{
}
Arrangement::Arrangement (int maxPlanes) :
maxPlanes(maxPlanes),
curPlanes(0),
planes(maxPlanes)
{
}
That will handle all memory management for you; your job would be to catch std::bad_alloc to check for memory allocation errors, or add any necessary parameter validation.
Your code has a number of other problems, or potential for problems, but that's outside the scope of this question I think.
I was wanting a simple string table that will store a bunch of constants and I thought "Hey! Lua does that, let me use some of there functions!"
This is mainly in the lstring.h/lstring.c files (I am using 5.2)
I will show the code I am curious about first. Its from lobject.h
/*
** Header for string value; string bytes follow the end of this structure
*/
typedef union TString {
L_Umaxalign dummy; /* ensures maximum alignment for strings */
struct {
CommonHeader;
lu_byte reserved;
unsigned int hash;
size_t len; /* number of characters in string */
} tsv;
} TString;
/* get the actual string (array of bytes) from a TString */
#define getstr(ts) cast(const char *, (ts) + 1)
/* get the actual string (array of bytes) from a Lua value */
#define svalue(o) getstr(rawtsvalue(o))
As you see, the data is stored outside of the structure. To get the byte stream, you take the size of TString, add 1, and you got the char* pointer.
Isn't this bad coding though? Its been DRILLED into m in my C classes to make clearly defined structures. I know I might be stirring a nest here, but do you really lose that much speed/space defining a structure as header for data rather than defining a pointer value for that data?
The idea is probably that you allocate the header and the data in one big chunk of data instead of two:
TString *str = (TString*)malloc(sizeof(TString) + <length_of_string>);
In addition to having just one call to malloc/free, you also reduce memory fragmentation and increase memory localization.
But answering your question, yes, these kind of hacks are usually a bad practice, and should be done with extreme care. And if you do, you'll probably want to hide them under a layer of macros/inline functions.
As rodrigo says, the idea is to allocate the header and string data as a single chunk of memory. It's worth pointing out that you also see the non-standard hack
struct lenstring {
unsigned length;
char data[0];
};
but C99 added flexible array members so it can be done in a standard compliant way as
struct lenstring {
unsigned length;
char data[];
};
If Lua's string were done in this way it'd be something like
typedef union TString {
L_Umaxalign dummy;
struct {
CommonHeader;
lu_byte reserved;
unsigned int hash;
size_t len;
const char data[];
} tsv;
} TString;
#define getstr(ts) (ts->tsv->data)
It relates to the complications arising from the more limited C language. In C++, you would just define a base class called GCObject which contains the garbage collection variables, then TString would be a subclass and by using a virtual destructor, both the TString and it's accompanying const char * blocks would be freed properly.
When it comes to writing the same kind of functionality in C, it's a bit more difficult as classes and virtual inheritance do not exist.
What Lua is doing is implementing garbage collection by inserting the header required to manage the garbage collection status of the part of memory following it. Remember that free(void *) does not need to know anything other than the address of the memory block.
#define CommonHeader GCObject *next; lu_byte tt; lu_byte marked
Lua keeps a linked list of these "collectable" blocks of memory, in this case an array of characters, so that it can then free the memory efficiently without knowing the type of object it is pointing to.
If your TString pointed to another block of memory where the character array was, then it require the garbage collector determine the object's type, then delve into its structure to also free the string buffer.
The pseudo code for this kind of garbage collection would be something like this:
GCHeader *next, *prev;
GCHeader *current = firstObject;
while(current)
{
next = current->next;
if (/* current is ready for deletion */)
{
free(current);
// relink previous to the next (singly-linked list)
if (prev)
prev->next = next;
}
else
prev = current; // store previous undeleted object
current = next;
}
Often stacks in C are dependent upon datatype used to declare them. For example,
int arr[5]; //creates an integer array of size 5 for stack use
char arr[5]; //creates a character array of size 5 for stack use
are both limited to holding integer and character datatypes respectively and presumes that the programmer knows what data is generated during the runtime. What if I want a stack which can hold any datatype?
I initially thought of implementing it as a union, but the approach is not only difficult but also flawed. Any other suggestions?
I would use a structure like this:
struct THolder
{
int dataType; // this is a value representing the type
void *val; // this is the value
};
Then use an array of THolder to store your values.
This is really just a variant of Pablo Santa Cruz' answer, but I think it looks neater:
typedef enum { integer, real, other } type_t;
typedef struct {
type_t type;
union {
int normal_int; /* valid when type == integer */
double large_float; /* valid when type == real */
void * other; /* valid when type == other */
} content;
} stack_data_t;
You still need to use some way to explicitly set the type of data stored in each element, there is no easy way around that.
You could look into preprocessor magic relying on the compiler-dependent typeof keyword to do that automagically, but that will probably not do anything but ruin the portability.
Some people have suggested a void* member. In addition to that solution I'd like to offer an alternative (assuming your stack is a linked list of heap-allocated structures):
struct stack_node
{
struct stack_node *next;
char data[];
};
The data[] is a C99 construct. data must be the last member; this takes advantage of the fact that we can stuff arbitrary quantities after the address of the struct. If you're using non-C99 compiler you might have to do some sketchy trick like declare it as data[0].
Then you can do something like this:
struct stack_node*
allocate_stack_node(size_t extra_size)
{
return malloc(sizeof(struct stack_node) + extra_size);
}
/* In some other function... */
struct stack_node *ptr = allocate_stack_node(sizeof(int));
int *p = (int*)ptr->data;
If this looks ugly and hacky, it is... But the advantage here is that you still get the generic goodness without introducing more indirection (thus slightly quicker access times for ptr->data than if it were void* pointing to a different location from the structure.)
Update: I'd also like to point out that the code sample I give may have problems if your machine happens to have different alignment requirements for int than char. This is meant as an illustrative example; YMMV.
You could use macros and a "container" type to reduce "type" from being per-element, to whole-container. (C99 code below)
#define GENERIC_STACK(name, type, typeid, elements) \
struct name##_stack { \
unsigned int TypeID; \
type Data[elements]; \
} name = { .TypeID = typeid }
Of course, your "TypeID" would have to allow every possible agreed-upon type you expect; might be a problem if you intend to use whole structs or other user-defined types.
I realize having a uniquely named struct type for every variable is odd and probably not useful... oops.
I created an library that works for any data type:
List new_list(int,int);
creates new list eg:
List list=new_list(TYPE_INT,sizeof(int));
//This will create an list of integers
Error append(List*,void*);
appends an element to the list. *Append accpts two pointers as an argument, if you want to store pointer to the list don't pass the pointer by pointer
eg:
//using the int list from above
int a=5;
Error err;
err=append(&list,&a)
//for an list of pointers
List listptr=new_list(TYPE_CUSTOM,sizeof(int*));
int num=7;
int *ptr=#
append(&listptr,ptr);
//for list of structs
struct Foo
{
int num;
float *ptr;
};
List list=new_list(TYPE_CUSTOM,sizeof(struct Foo));
struct Foo x;
x.num=9;
x.ptr=NULL;
append(&list,&x);
Error get(List*,int);
Gets data at index specified. When called list's current poiter will point to the data.
eg:
List list=new_list(TYPE_INT,sizeof(int));
int i;
for(i=1;i<=10;i++)
append(&list,&i);
//This will print the element at index 2
get(&list,2);
printf("%d",*(int*)list.current);
Error pop(List*,int);
Pops and element from the specified index
eg:
List list=new_list(TYPE_INT,sizeof(int));
int i;
for(i=1;i<=10;i++)
append(&list,&i);
//element in the index 2 will be deleted,
//the current pointer will point to a location that has a copy of the data
pop(&list,2);
printf("%d",*(int*)list.current);
//To use the list as stack, pop at index list.len-1
pop(&list,list.len-1);
//To use the list as queue, pop at index 0
pop(&list,0);
Error merge(List ,List);
Merges two list of same type. If types are different will return a error message in the Error object it returns;
eg:
//Merge two elements of type int
//List 2 will come after list 1
Error err;
err=merge(&list1,&list2);
Iterator get_iterator(List*);
Get an iterator to an list. when initialized will have a pointer to the first element of the list.
eg:
Iterator ite=get_iterator(&list);
Error next(Iterator*);
Get the next element of the list.
eg:
//How to iterate an list of integers
Iterator itr;
for(itr=get_iterator(&list); ite.content!=NULL; next(ite))
printf("%d",*(int*)ite.content);
https://github.com/malayh/C-List