Im am trying to make some code, which mimics a simple malloc-function (in C), though it should only control the memory of a big array, and not the actual physical memory. To control the "memory", I would like to store segments of META-data in the memory-array. The META-data is stored as a struct.
My question is, how do I correctly store the struct in the bytes of the array? In the example shown here, I try to store some initial META-data on the starting element of the memory-array; however I have syntax wrong to do this.
typedef struct _xMetaData{
size_t xSize;
int* piNextBlock;
int iBlockFree;
}xMetaData;
int8_t memory[ALLOCATE_SIZE];
// Pointer to struct
xMetaData* pxMetaPtr;
xMetaData xInitialData = {BLOCKSIZE, &memory[INITIAL_BLOCK_ADDRESS], BLOCK_FREE};
&memory[0] = xInitialData;
You need to cast the block of memory to xMetaData:
*(xMetaData *) (&memory[0]) = xInitialData;
You should also be aware of structure padding if you're using a struct for this kind of thing (for example, make sure ALLOCATE_SIZE uses sizeof(xMetaData) and not a hardcoded length, and make sure you always access the memory using the struct.)
Related
Suppose I am given a (void*) ptr (my basic understanding is, it represents a pointer to a region of unknown data type) passed through the parameter of a function. I am trying to figure out how to access and check if a struct exists a few addresses behind.
To clarify, I am working with a big char array (not malloced) and the ptr passed into the function should point to an address of an unspecified data type within the array. Located before this data is a struct for which I am trying to access.
void function(void *ptr)
{
void *structPtr = (void*)((void*)ptr - sizeof(struct block));
}
Would this work to get me a pointer to the address of the struct located behind the initial "ptr"? And if so, how could I check if it is the block struct?
Apologizes in advance, I know this code is not specific as I am fairly new to the concepts entirely but also, I am in the process of coming up with an algorithm and not yet implementing it. Any references to possibly useful information are much appreciated.
what you are trying to do is risky as you must be sure that you address a correct place in memory. Usually, we add some magic number in struct block so that we can test here that we are not going anywhere.
This pattern is generally used in memory allocators,
have a look to https://sourceware.org/glibc/wiki/MallocInternals for an example.
The usual way of writing this is something like:
...function(void *ptr) {
struct block *hdr = (struct block *)ptr - 1;
relying on pointer arithmetic automatically scaling by the size of the pointed at type. As long as all the pointers passed in here were originally created by taking a pointer to a valid struct block and adding 1 to it (to get the memory after it), you should be fine.
I practice in realization a memory manager in C.
I want the structure, that has a various length and self-described.
So, I peep at a POSIX textbook something, like that:
struct layout
{
uint32_t size; // array size in bytes, include space after the struct
uchar_t data[1];
};
// But, is next line correct?
layout *val = malloc (array_memory_in_bytes + sizeof (uint32_t) - 1);
// Where does a static array keep the pointer for using it?
If I have several these structures one-after-one in uninterrupted piece of memory, and I want be able to iterate through them. Can I write something, like that:
layout *val1 = pointer;
layout *val2 = val1 + val1.size + sizeof (val1.size);
Or can you recommend me a better approach?
The Standard C version of this is called flexible array member and it looks like:
struct layout
{
uint32_t size;
uchar_t data[];
};
// allocate one of these blocks (in a function)
struct layout *val = malloc( sizeof *val + number_of_bytes );
val->size = number_of_bytes;
The code val1->data + val1->size will get you a pointer one-past-the-end of the space you just malloc'd.
However you cannot iterate off the end of one malloc'd block and hope to hit another malloc'd block. To implement this idea you would have to malloc a large block and then place various struct layout objects throughout it, being careful about alignment.
In this approach, it's probably best to also store an index of where each struct layout is. In theory you could go through the list from the start each time, adding on size and then doing your alignment adjustment; but that would be slow and also it would mean you could not cope with a block in the middle being freed and re-"allocated".
If this is meant to be a drop-in replacement for malloc then there are in fact two alignment considerations:
alignment for struct layout
data must be aligned for any possible type
The simplest way to cope with this is to align struct layout for any possible type also. This could look like (note: #include <stdint.h> required):
struct layout
{
uint64_t size; // may as well use 64 bits since they're there
_Alignas(max_align_t) uchar_t data[];
};
An alternative might be to keep size at 32-bit and throw in a pragma pack to prevent padding; then you'll need to use some extra complexity to make sure that the struct layout is placed 4 bytes before a max_align_t-byte boundary, and so on. I'd suggest doing it the easy way first and get your code running; then later you can go back and try this change in order to save a few bytes of memory if you want.
Alternative approaches:
Keep each instance of a struct layout plus its trailing data in a separate allocation.
Change data be a pointer to malloc'd space; then you could keep all of the struct layout objects in an array.
The general idea will work, but that specific struct will only work if the most-severe boundary alignment case is an int.
A memory manager, particularly one that might be a back-end for an implementation of malloc(), must know what that worst-case boundary is. The actual start of data must be on that boundary in order to satisfy the general requirement that the allocated memory be suitably aligned for the storage of any data type.
The easiest way to get that done is to make the length allocation header described by the layout struct and the actual allocation sizes all multiples of that alignment unit.
No matter what, you can't describe the start of data as a struct member and have the size of that struct be the size of the header. C doesn't support zero-length fields. You should use something to put that array on boundary, and use the offsetof() macro from <stddef.h>.
Personally, I'd use a union, based on both old habits and occasional use of Visual C++ for C. But uint32_t is a C99 type and if you also have C11 support you can use _Alignas(). With that, your struct could look something like:
#define ALIGN_TYPE double /* if this is the worst-case type */
#define ALIGN_UNIT ((sizeof)(ALIGN_TYPE))
#define ALIGN_SIZE(n) (((size_t)(n) + ALIGN_UNIT - 1) & ~(ALIGN_UNIT-1))
typedef struct layout
{
size_t size; /* or use uint32_t if you prefer */
_Alignas(ALIGN_UNIT) char data[1];
} layout;
#define HEADER_SIZE (offsetof(layout, data))
That makes most everything symbolic except for the worst-case alignment type. You'd allocate the combined header plus data array with:
layout *ptr = (layout*) malloc(HEADER_SIZE + ALIGN_SIZE(number_of_bytes));
ptr->size = HEADER_SIZE;
The ALIGN_SIZE type really isn't a symbolic constant, though, unless C99/C11 changed the definition of sizeof. You can't use to compute ordinary array dimensions, for example. You can hard code a literal number, like 8 for a typical double, if that's a problem. Beware that long double has a problematical size (10 bytes) on many x86 implementations. If you're going to base the allocation unit on a type, then long double might not be your best choice.
I'm currently having an issue with the following struct:
typedef struct __attribute__((__packed__)) rungInput{
operation inputOperation;
inputType type;
char* name;
char numeroInput;
u8 is_not;
} rungInput;
I create multiple structs like above inside a for loop, and then fill in their fields according to my program logic:
while (a < 5){
rungInput input;
(...)
Then when I'm done filling the struct's fields appropriately, I then attempt to copy the completed struct to an array as such:
rungArray[a] = input; //memcpy here instead?
And then I iterate again through my loop. I'm having a problem where my structs seem to all have their name value be the same, despite clearly having gone through different segments of code and assigning different values to that field for every loop iteration.
For example, if I have three structs with the following names: "SW1" "SW2" SW3", after I am done adding them to my array I seem to have all three structs point me to the value "SW3" instead. Does this mean I should call malloc() to allocate manually each pointer inside each struct to ensure that I do not have multiple structs that point to the same value or am I doing something else wrong?
When you write rungArray[i] = input;, you are copying the pointer that is in the input structure into the rungArray[i] structure. If you subsequently overwrite the data that the input structure is pointing at, then you also overwrite the data that the rungArray[i] structure is pointing at. Using memcpy() instead of assignment won't change this at all.
There are a variety of ways around this. The simplest is to change the structure so that you allocate a big enough array in the structure to hold the name:
enum { MAX_NAME_SIZE = 32 };
…
char name[MAX_NAME_SIZE];
…
However, if the extreme size of a name is large but the average size is small, then this may waste too much space. In that case, you continue using a char *, but you do indeed have to modify the copying process to duplicate the string with dynamically allocated memory:
rungArray[i] = input;
rungArray[i].name = strdup(input.name);
Remember to free the memory when you discard the rungArray. Yes, this code copies the pointer and then overwrites it, but it is more resilient to change because all the fields are copied, even if you add some extra (non-pointer) fields, and then the pointer fields are handled specially. If you write the assignments to each member in turn, you have to remember to track all the places where you do this (that would be a single assignment function, wouldn't it?) and add the new assignments there. With the code shown, that mostly happens automatically.
You should malloc memory for your struct and then store the pointers to the structs inside your array. You could also turn your structs into a linked list by adding a pointer to each struct that points to the next instance of your struct.
http://www.cprogramming.com/tutorial/c/lesson15.html
This is one of those I think this should work, but it's best to check questions. It compiles and works fine on my machine.
Is this guaranteed to do what I expect (i.e. allow me to access the first few elements of the array with a guarantee that the layout, alignment, padding etc of the struct is the same as the array)?
struct thingStruct
{
int a;
int b;
int c;
};
void f()
{
int thingsArray[5];
struct thingStruct *thingsStruct = (struct thingStruct *)&thingsArray[0];
thingsArray[0] = 100;
thingsArray[1] = 200;
thingsArray[2] = 300;
printf("%d", thingsStruct->a);
printf("%d", thingsStruct->b);
printf("%d", thingsStruct->c);
}
EDIT: Why on earth would I want to do something like this? I have an array which I'm mmapping to a file. I'm treating the first part of the array as a 'header', which stores various pieces of information about the array, and the rest of it I'm treating as a normal array. If I point the struct to the start of the array I can access the pieces of header data as struct members, which is more readable. All the members in the struct would be of the same type as the array.
While I have seen this done frequently, you cannot (meaning it is not legal, standard C) make assumptions about the binary layout of a structure, as it may have padding between fields.
This is explained in the comp.lang.c faq: http://c-faq.com/struct/padding.htmls
Although it's likely to work in most places, it's still a bit iffy. If you want to give symbolic names to parts of the header, why not just do:
enum { HEADER_A, HEADER_B, HEADER_C };
/* ... */.
printf("%d", thingsArray[HEADER_A]);
printf("%d", thingsArray[HEADER_B]);
printf("%d", thingsArray[HEADER_C]);
As Evan commented on the question, this will probably work in most cases (again, probably best if you use #pragma pack to ensure their is no padding) assuming all the types in your struct are the same type as your array. Given the rules of C, this is legal.
My question to you is "why?" This isn't a particularly safe thing to do. If a float gets thrown into the middle of the struct, this all falls apart. Why not just use the struct directly? This really ins't a technique that I'd recommend in most cases.
Another solution for representing a header and the rest of file data is using a structure like this:
struct header {
long headerData1;
int headerData2;
int headerData3;
int fileData[ 1 ]; // <- data begin here
};
Then you allocate the memory block with a file contents and cast it as struct header *myFileHeader (or map the memory block on a file) and access all your file data with
myFileHeader->fileData[ position ]
for arbitrary big position. The language imposes no restriction on the index value, so it's only your responsibility to keep your arbitrary big posistion within the actual size of the memory block you allocated (or the mapped file's size).
One more important note: apart from switching off the struct members padding, which has been already described by others, you should carefully choose data types for the header members, so that they fit the actual file data layout despite compiler you use (say, int won't change from 32 to 64 bits...)
OK, I hope I explain this one correctly.
I have a struct:
typedef struct _MyData
{
char Data[256];
int Index;
} MyData;
Now, I run into a problem. Most of the time MyData.Data is OK with 256, but in some cases I need to expand the amount of chars it can hold to different sizes.
I can't use a pointer.
Is there any way to resize Data at run time? How?
Code is appreciated.
EDIT 1:
While I am very thankful for all the comments, the "maybe try this..." or "do that", or "what you are dong is wrong..." comments are not helping. Code is the help here. Please, if you know the answer post the code.
Please note that:
I cannot use pointers. Please don't try to figure out why, I just can't.
The struct is being injected into another program's memory that's why no pointers can be used.
Sorry for being a bit rough here but I asked the question here because I already tried all the different approaches that thought might work.
Again, I am looking for code. At this point I am not interested in "might work..." or " have you considered this..."
Thank you and my apologies again.
EDIT 2
Why was this set as answered?
You can use a flexible array member
typedef struct _MyData
{
int Index;
char Data[];
} MyData;
So that you can then allocate the right amount of space
MyData *d = malloc(sizeof *d + sizeof(char[100]));
d->Data[0..99] = ...;
Later, you can free, and allocate another chunk of memory and make a pointer to MyData point to it, at which time you will have more / less elements in the flexible array member (realloc). Note that you will have to save the length somewhere, too.
In Pre-C99 times, there isn't a flexible array member: char Data[] is simply regarded as an array with incomplete type, and the compiler would moan about that. Here i recommend you two possible ways out there
Using a pointer: char *Data and make it point to the allocated memory. This won't be as convenient as using the embedded array, because you will possibly need to have two allocations: One for the struct, and one for the memory pointed to by the pointer. You can also have the struct allocated on the stack instead, if the situation in your program allows this.
Using a char Data[1] instead, but treat it as if it were bigger, so that it overlays the whole allocated object. This is formally undefined behavior, but is a common technique, so it's probably safe to use with your compiler.
The problem here is your statement "I can't use a pointer". You will have to, and it will make everything much easier. Hey, realloc even copies your existing data, what do you want more?
So why do you think you can't use a pointer? Better try to fix that.
You would re-arrange the structure like that
typedef struct _MyData
{
int Index;
char Data[256];
} MyData;
And allocate instances with malloc/realloc like that:
my_data = (MyData*) malloc ( sizeof(MyData) + extra_space_needed );
This is an ugly approach and I would not recommend it (I would use pointers), but is an answer to your question how to do it without a pointer.
A limitation is that it allows for only one variable size member per struct, and has to be at the end.
Let me sum up two important points I see in this thread:
The structure is used to interact between two programs through some IPC mechanism
The destination program cannot be changed
You cannot therefore change that structure in any way, because the destination program is stuck trying to read it as currently defined. I'm afraid you are stuck.
You can try to find ways to get the equivalent behavior, or find some evil hack to force the destination program to read a new structure (e.g., modifying the binary offsets in the executable). That's all pretty application specific so I can't give much better guidance than that.
You might consider writing a third program to act as an interface between the two. It can take the "long" messages and do something with them, and pass the "short" messages onward to the old program. You can inject that in between the IPC mechanisms fairly easily.
You may be able to do this like this, without allocating a pointer for the array:
typedef struct _MyData
{
int Index;
char Data[1];
} MyData;
Later, you allocate like this:
int bcount = 256;
MyData *foo;
foo = (MyData *)malloc(sizeof(*foo) + bcount);
realloc:
int newbcount = 512;
MyData *resized_foo;
resized_foo = realloc((void *)foo, sizeof(*foo) + newbcount);
It looks like from what you're saying that you definitely have to keep MyData as a static block of data. In which case I think the only option open to you is to somehow (optionally) chain these data structures together in a way that can be re-assembled be the other process.
You'd need and additional member in MyData, eg.
typedef struct _MyData
{
int Sequence;
char Data[256];
int Index;
} MyData;
Where Sequence identifies the descending sequence in which to re-assemble the data (a sequence number of zero would indicate the final data buffer).
The problem is in the way you're putting the question. Don't think about C semantics: instead, think like a hacker. Explain exactly how you are currently getting your data into the other process at the right time, and also how the other program knows where the data begins and ends. Is the other program expecting a null-terminated string? If you declare your struct with a char[300] does the other program crash?
You see, when you say "passing data" to the other program, you might be [a] tricking the other process into copying what you put in front of it, [b] tricking the other program into letting you overwrite its normally 'private' memory, or [c] some other approach. No matter which is the case, if the other program can take your larger data, there is a way to get it to them.
I find KIV's trick quite usable. Though, I would suggest investigating the pointer issue first.
If you look at the malloc implementations
(check this IBM article, Listing 5: Pseudo-code for the main allocator),
When you allocate, the memory manager allocates a control header and
then free space following it based on your requested size.
This is very much like saying,
typedef struct _MyData
{
int size;
char Data[1]; // we are going to break the array-bound up-to size length
} MyData;
Now, your problem is,
How do you pass such a (mis-sized?) structure to this other process?
That brings us the the question,
How does the other process figure out the size of this data?
I would expect a length field as part of the communication.
If you have all that, whats wrong with passing a pointer to the other process?
Will the other process identify the difference between a pointer to a
structure and that to a allocated memory?
You cant reacolate manualy.
You can do some tricks wich i was uning when i was working aon simple data holding sistem. (very simple filesystem).
typedef struct
{
int index ;
char x[250];
} data_ztorage_250_char;
typedef struct
{
int index;
char x[1000];
} data_ztorage_1000_char;
int main(void)
{
char just_raw_data[sizeof(data_ztorage_1000_char)];
data_ztorage_1000_char* big_struct;
data_ztorage_250_char* small_struct;
big_struct = (data_ztorage_1000_char*)big_struct; //now you have bigg struct
// notice that upper line is same as writing
// big_struct = (data_ztorage_1000_char*)(&just_raw_data[0]);
small_struct = (data_ztorage_250_char*)just_raw_data;//now you have small struct
//both structs starts at same locations and they share same memory
//addresing data is
small_struct -> index = 250;
}
You don't state what the Index value is for.
As I understand it you are passing data to another program using the structure shown.
Is there a reason why you can't break your data to send into chunks of 256bytes and then set the index value accordingly? e.g.
Data is 512 bytes so you send one struct with the first 256 bytes and index=0, then another with the next 256 bytes in your array and Index=1.
How about a really, really simple solution? Could you do:
typedef struct _MyData
{
char Data[1024];
int Index;
} MyData;
I have a feeling I know your response will be "No, because the other program I don't have control over expects 256 bytes"... And if that is indeed your answer to my answer, then my answer becomes: this is impossible.