I am trying to make a struct for a circular buffer that contains an array of type "quote." However, the quote array must start out at a size of 10. I am trying to figure out if I declare the size of 10 in my .h file or in my .c file. My two files are as follows:
.h file:
typedef struct{
unsigned int time;
double rate;
}quote;
typedef struct{
unsigned int testNum;
quote quoteBuffer[];
}cbuf;
cbuf* cbuf_init();
.c file:
cbuf* cbuf_init(){
cbuf *buffer = (cbuf *)calloc(1,sizeof(cbuf));
buffer->testNum = 1;
quote newQuote = {1,1.00};
buffer->quoteBuffer[1] = newQuote;
return buffer;
}
These are obviously just test values, however if I wanted to specifically make the quote array in the cbuf struct start out at a size of 10, would I declare that in the .h file as:
typedef struct{
unsigned int testNum;
quote quoteBuffer[10];
}cbuf;
or in the .c file some other way?
There are two ways of having dynamic arrays in structures. The obvious is of course to have it as a pointer, and dynamically allocate (or reallocate) when needed.
The other is to have an array of size 1, and then allocate a larger size than the structure, to accommodate for the array:
typedef struct {
unsigned int testNum;
quote quoteBuffer[1];
} cbuf;
cbuf *cbuf_init(const size_t num_quotes) {
/* Allocate for the `cbuf` structure, plus a number of `quote`
* structures in the array
*/
cbuf *buffer = malloc(sizeof(cbuf) + (num_quotes - 1) * sizeof(quote));
/* Other initialization */
return buffer;
}
/* If more quotes are needed after initial allocation, use this function */
cbuf *cbuf_realloc(cbuf *buffer, const size_t new_num_quotes) {
buffer = realloc(buffer, sizeof(cbuf) + (new_num_quotes - 1) * sizeof(quote));
/* Other initialization */
return buffer;
}
Now you can use the array as a normal array:
cbuf *buffer = cbuf_init();
buffer->quoteBuffer[5].time = 123;
Note: I only allocate extra space for 9 quote structures, but state that I allocate ten. The reason is that the cbuf structure already contains one quote structure in its array. 1 + 9 = 10. :)
Note 2: I put the quote array in the cbuf structure with one entry already in it for backwards compatibility. Having an array without a size in the structure is quite new in the C world.
you can also do this if you want 10 quotes in a cbuf but a statically allocated like quote buffer[10] would work too:
cbuf* cbuf_init(int numQuotes)
{
cbuf *b = calloc(1, sizeof(cbuf) + numQuotes * sizeof(quote));
return b;
}
If you want a statically sized circular buffer then your could declare the size in the header file. Using a #define for the buffer size will make the code more readable and maintainable, as you'll reference the size elsewhere in your code.
If you want the circular buffer to be growable then define the size in your C file. You'll then have to take care of tracking the size and destroying the memory that you will have to allocate dynamically.
In your example, I think you need to allocate more room for your quote structs...
cbuf *buffer = (cbuf *)calloc(1,sizeof(cbuf) + NUM_QUOTES*sizeof(struct quote));
---------------------------------
The reason for this is that in your struct def...
quote quoteBuffer[];
... quoteBuffer doesn't add size to the struct. quoteBuffer will point to one byte past the end of the struct, hence the need to allocate memory for the struct + memory for the array.
EDIT: Daniel Fischer's comment (thanks Daniel) - quoteBuffer may, in some cases, add size to the struct if it introduces padding. The reason is that the compiler will probably strive to get the most optimal alignment for quoteBuffer. For example, ints normally aligned of 4-byte boundaries. E.g. a struct like:
struct {
char a;
int b;
}
is probably changed by compiler to
struct {
char a;
char pad[3]; // compiler adds padding behind the scenes
int b; // align b on a 4-byte boundary
}
This probs doesn't apply in your case as your struct leaves quoteBuffer[] on a 4 byte boundary.
The reason that the compiler does this is two fold.
1. On some architectures (not so common nowadays I think?), unaligned accesses aren't supported.
2. Aligned accesses are more efficient, even if architecture allows non-aligned accesses as it is one memory read as opposed to two memory reads plus a manipulation.
Related
I am working on a micro controller, so, no malloc. Actually, I want to create a memory manager, so I am kinda implementing the malloc function for later use and using the BLOCK strategy to get it, like FreeRTOS does.
typedef struct BLOCK {
unsigned char used; // If 1 block is used, if 0 it's free to use
unsigned long offset; // Offset of the starting block
unsigned long size; // Size of this block
struct BLOCK * next; // Pointer to the next block
} BLOCK_t;
#define MAX_PROGRAMS 3
#define BLOCKS_NEEDED (MAX_PROGRAMS * 2) + 1
BLOCK_t blocks[BLOCKS_NEEDED]; // Not allocating 7 * sizeof(BLOCK)
This BLOCK is a linked list and I want to create (allocate) a fixed amount of them and set the first blocks[0] only. The next ones will be created in execution time, when memory is allocated.
Thanks in advance.
EDIT: In case the title is not clear enough, I want to compiler to assign some memory space to my array (fixed location and size) but I don't want to initialize it with data because I will get the data in run-time, so I want an array of 7 BLOCKs with empty data. The code above shows my attempt to do it, I declared the array, but I assume that declaring an array doesn't give you the space needed. How can I achieve this ? How can I get the compiler to give me that space ?
EDIT 2: This would be tha Java code to do it.
private static int MAX_PROGRAMS = 3;
private static int BLOCKS_NEEDED = (MAX_PROGRAMS * 2) + 1:
Block myBlockList[] = new Block[BLOCKS_NEEDED];
This get the space for myBlockList even though the list is empty and each item is uninitialized, but I have the space already.
All you want to do is allocate memory automatically on the stack.
#include <stdio.h>
#define blockcontent_size 1024
#define blockcount 3
typedef struct
{
unsigned char used;
unsigned long offset;
unsigned long size;
unsigned data[blockcontent_size];
} BLOCK;
BLOCK blocks[blockcount];
int main()
{
printf("memory available in one block %u\n", sizeof(blocks[0].data));
printf("memory used for one block %u\n", sizeof(BLOCK));
printf("memory used for all blocks %u\n", sizeof(blocks));
return 0;
}
You actually do not need a linked list, you can just use the index.
Is this close to what you are asking?
#LPs quote:
Using c writing BLOCK_t blocks[BLOCKS_NEEDED]; you are declaring the array and sizeof(BLOCK_t)*BLOCKS_NEEDED bytes are occupied by the array.
So my statement :
BLOCK_t blocks[BLOCKS_NEEDED]; // Not allocating 7 * sizeof(BLOCK)
was false, it actually does allocate the space.
If I want to reduce malloc()s (espacially if the data is small and allocated often) I would like to allocate the pointer and pointee at once.
If you assume something like the following:
struct entry {
size_t buf_len;
char *buf;
int something;
};
I would like to allocate memory in the following way (don't care about error checking here):
size_t buf_len = 4; // size of the buffer
struct entry *e = NULL;
e = malloc( sizeof(*e) + buf_len ); // allocate struct and buffer
e->buf_len = buf_len; // set buffer size
e->buf = e + 1; // the buffer lies behind the struct
This could even be extende, so that a whole array is allocated at once.
How would you assess such a technuique with regard to:
Portability
Maintainability / Extendability
Performance
Readability
Is this reasonable? If it is ok to use, are there any ideas on how to design a possible interface for that?
You could use a flexible array member instead of a pointer:
struct entry {
size_t buf_len;
int something;
char buf[];
};
// ...
struct entry *e = malloc(sizeof *e + buf_len);
e->buf_len = buf_len;
Portability and performance are fine. Readability: not perfect but good enough.
Extendability: you can't use this for more than one member at a time, you'd have to fall back to your explicit pointer version. Also, the explicit pointer version means that you have to muck around to ensure correct alignment if you use it with a type that doesn't have an alignment of 1.
If you are seriously thinking about this I'd consider revisiting your entire data structure's design to see if there is another way of doing it. (Maybe this way is actually the best way, but have a good think about it first).
As to portability, I am unaware of any issues, as long as the sizes are found via suitable calls to sizeof(), as in your code.
Regarding maintainability, extendability and readability, you should certainly wrap allocation and de-allocation in a well-commented function. Calls to...
entry *allocate_entry_with_buffer();
void deallocate_entry_with_buffer(entry **entry_with_buffer);
...do not need to know implementation details of how the memory actually gets handled. People use stranger things like custom allocators and memory pools quite frequently.
As for speed, this is certainly faster than making lots of small allocations. I used to allocate whole 2D matrices with a similar strategy...
It should work, but in fact you are using a pointer for a useless indirection. Windows API (for example) uses another method for variable size structs : the variable size buffer is last in struct and is declared to be char buf[1].
Your struct would become :
struct entry {
size_t buf_len;
int something;
char buf[1];
};
The allocation is (still no error checking) :
size_t buf_len = 4; // size of the buffer
struct entry *e;
e = malloc( sizeof(*e) + buf_len - 1); // struct already has room for 1 char
e->buf_len = buf_len; // set buffer size
That's all e.buf is guaranteed to be a char array of size buf_len.
That way ensures that even if the variable part was not a character array but a int, long, or anything array, the alignement would be given by the last element being a array of proper type and size 1.
For starters, the line:
e->buf = e + sizeof(*e); // the buffer lies behind the struct
Should be:
e->buf = e + 1; // the buffer lies behind the struct
This is because e + 1 will be equal to the address at the end of the structure. As you have it, it will only be the number of bytes into the structure equal to the number of bytes in a pointer.
And, yes, it's reasonable. However, I prefer this approach:
struct entry {
size_t buf_len;
int something;
char buf[1];
};
This way, you don't mess with the pointers. Just append as many bytes as needed, and they will grow the size of your buf array.
Note: I wrote a text editor using an approach similar to this but used a Microsoft c++ extension that allowed me to declare the last member as char buf[]. So it was an empty array that was exactly as long as the number of extra bytes I allocated.
seems fine to me - put comments in though
Or you could do this - which is quite common
struct entry {
size_t buf_len;
int something;
char buf;
};
ie make the struct itself variable length. and do
size_t buf_len = 4; // size of the buffer
struct entry *e = NULL;
// check that it packs right
e = malloc(sizeof(size_t) + sizeof(int) + buf_len ); // allocate struct and buffer
e->buf_len = buf_len; // set buffer size
...... later
printf(&e.buf);
I am sure this is a basic question but I haven't been able to find whether or not this is a legitimate memory allocation strategy or not. I am reading in data from a file and I am filling in a struct. The size of the members are variable on each read so my struct elements are pointers like so
struct data_channel{
char *chan_name;
char *chan_type;
char *chan_units;
};
So before reading I figure out what the size of each string is so I can allocate memory for them my question is can I allocate the memory for the struct and the strings all in one malloc and then fill the pointer in?
Say the size of chan_name is 9, chan_type 10, and chan_units 5. So I would allocate the and do something like this.
struct data_channel *chan;
chan = malloc(sizeof(struct data_channel) + 9 + 10 + 5);
chan->chan_name = chan[1];
chan->chan_type = chan->chan_name + 9;
chan->chan_units = chan->chan_type + 10;
So I read a couple of articles on memory alignment but I don't know if doing the above is a problem or not or what kind of unintended consequences it could have. I have already implemented it in my code and it seems to work fine. I just don't want to have to keep track of all those pointers because in reality each of my structs has 7 elements and I could have upwards of 100 channels. That of course means 700 pointers plus the pointers for each struct so total 800. The I also have to devise a way to free them all. I also want to apply this strategy to arrays of strings of which I then need to have an array of pointers to. I don't have any structures right now that would mix data types could that be a problem but I might could that be a problem?
If chan_name is a 8 character string, chan_type is a 9 character string and chan_units is a 4 character string, then yes it will work fine when you fix the compilation error you have when assigning to chan_name.
If you allocate enough memory for the structure plus all the strings (including their string terminator) then it's okay to use such a method. Maybe not recommended by all, but it will work.
It depends in part on the element types. You will certainly be able to do it with character strings; with some other types, you have to worry about alignment and padding issues.
struct data_channel
{
char *chan_name;
char *chan_type;
char *chan_units;
};
struct data_channel *chan;
size_t name_size = 9;
size_t type_size = 10;
size_t unit_size = 5;
chan = malloc(sizeof(struct data_channel) + name_size + type_size + unit_size);
if (chan != 0)
{
chan->chan_name = (char *)chan + sizeof(*chan);
chan->chan_type = chan->chan_name + name_size;
chan->chan_units = chan->chan_type + type_size;
}
This will work OK in practice — it was being done for ages before the standard was standardized. I can't immediately see why the standard would disallow this.
What gets trickier is if you needed to allocate an array of int, say, as well as two strings. Then you have to worry about alignment issues.
struct data_info
{
char *info_name;
int *info_freq;
char *info_unit;
};
size_t name_size = 9;
size_t freq_size = 10;
size_t unit_size = 5;
size_t nbytes = sizeof(struct data_info) + name_size + freq_size * sizeof(int) + unit_size;
struct data_info *info = malloc(nbytes);
if (info != 0)
{
info->info_freq = (int *)((char *)info + sizeof(*info));
info->info_name = (char *)info->info_freq + freq_size * sizeof(int);
info->info_unit = info->info_name + name_size;
}
This has adopted the simple expedient of allocating the most stringently aligned type (the array of int) first, then allocating the strings afterwards. This part is, however, where you have to make judgement calls about portability. I'm confident that the code is portable in practice.
C11 has alignment facilities (_Alignof and _Alignas and <stdalign.h>, plus max_align_t in <stddef.h>) that could alter this answer (but I've not studied them sufficiently so I'm not sure how, yet), but the techniques outlined here will work in any version of C provided you are careful about the alignment of data.
Note that if you have a single array in the structure, then C99 provides an alternative to the older 'struct hack' called a flexible array member (FAM). This allows you to have an array explicitly as the last element of the structure.
struct data_info
{
char *info_name;
char *info_units;
int info_freq[];
};
size_t name_size = 9;
size_t freq_size = 10;
size_t unit_size = 5;
size_t nbytes = sizeof(struct data_info) + name_size + freq_size * sizeof(int) + unit_size;
struct data_info *info = malloc(nbytes);
if (info != 0)
{
info->info_name = ((char *)info + sizeof(*info) + freq_size * sizeof(int));
info->info_units = info->info_name + name_size;
}
Note that there was no step to initialize the FAM, info_freq in this example. You cannot have multiple arrays like this.
Note that the techniques outlined cannot readily be applied to arrays of structures (at least, arrays of the outer structure). If you go to considerable effort, you can make it work. Also, beware of realloc(); if you reallocate space, you have to fix up the pointers if the data has moved.
One other point: especially on 64-bit machines, if the sizes of the strings are uniform enough, you'd probably do better allocating the arrays in the structure, instead of using the pointers.
struct data_channel
{
char chan_name[16];
char chan_type[16];
char chan_units[8];
};
This occupies 40 bytes. On a 64-bit machine, the original data structure would occupy 24 bytes for the three pointers and another 24 bytes for the (9 + 10 + 5) bytes of data, for a total of 48 bytes allocated.
I know there is a sure way to do this when you have ONE array at the end of a structure, but since all your arrays have the same type, you may be in luck. The sure method is:
#include <stddef.h>
#include <stdlib.h>
struct StWithArray
{
int blahblah;
float arr[1];
};
struct StWithArray * AllocWithArray(size_t nb)
{
size_t size = nb*sizeof(float) + offsetof(structStWithArray, arr);
return malloc(size);
}
The use of an actual array in the structure guarantees alignment is respected.
Now to apply it to your case:
#include <stddef.h>
#include <stdlib.h>
struct data_channel
{
char *chan_name;
char *chan_type;
char *chan_units;
char actualCharArray[1];
};
struct data_channel * AllocDataChannel(size_t nb)
{
size_t size = nb*sizeof(char) + offsetof(data_channel, actualCharArray);
return malloc(size);
}
struct data_channel * CreateDataChannel(size_t length1, size_t length2, size_t length3)
{
struct data_channel * pt = AllocDataChannel(length1 + length2 + length3);
if(pt != NULL)
{
pt->chan_name = &pt->actualCharArray[0];
pt->chan_type = &pt->actualCharArray[length1];
pt->chan_name = &pt->actualCharArray[length1+length2];
}
return pt;
}
Joachim and Jonathan's answers are nice. Only addition I would like to mention is this.
Separate mallocs and frees buy you some basic protection like buffer overrun, access after
free, etc. I mean basic and not Valgrind like features. Allocating one single chunk and internally doling it out will lead to a loss of this feature.
In future, if the mallocs are for different sizes totally, then separate mallocs may buy you the efficiency of coming from different allocation buckets inside of the malloc implementation, especially if you are going to free them at different times.
The last thing you have to consider is how frequently you are calling mallocs. If it is frequent, then cost of multiple mallocs can be costly.
I tried to replace a void* member of a struct with a flexible array member using the more accepted idiom:
typedef struct Entry {
int counter;
//void* block2; // This used to be what I had
unsigned char block[1];
}
I then add entries into a continuous memory block:
void *memPtr = mmap(NULL, someSize*1024, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
as such:
int number = 0;
int AddEntry(void *data) {
Entry *entry;
entry = malloc(sizeof(Entry) + ((SECTOR_SIZE-1) * sizeof(unsigned char));
entry->counter = 1;
memcpy(entry->block, data, SECTOR_SIZE);
// make sure number doesn't overflow space, etc...
memcpy(&memPtr[number], entry, sizeof(Entry) + ((SECTOR_SIZE-1) * sizeof(unsigned char));
number++;
return 0;
}
The problem is unpacking this data once I need it. For example, if I do:
void * returnBlock(int i) {
Entry * entry = &memPtr[i];
printf("Entry counter is %d\n", entry->counter); // returns 1, reliably
return entry->block; // Gives me gibberish but not if I uncomment void* block2.
}
Is there a reason this could be? I don't necessarily think I'm stomping on stuff anywhere, and it used to work with the void* approach. The weird thing is that if I put a dummy void* back into the struct, it works. It doesn't work if I put in a dummy int.
Edit: actually, it also fails if number in AddEntry is not 0. What am I stepping on, if anything?
Your problem is here:
&memPtr[number]
Since memPtr is a void * pointer, this isn't actually allowed in C at all. Some compilers do allow arithmetic on void * pointers as a language extension - however they treat it as if it were a char * pointer.
That means that &memPtr[number] is likely indexing only number bytes into your memory block - so the second Entry structure copied in will overlap the first one, and so on.
Your allocation line appears to be assuming 1024 bytes per Entry (if someSize is a number of Entry structures), so you probably want something like:
((char *)memPtr + number * 1024)
(and similar in the returnBlock() function).
However, if you do this you will notice that there is no point in using the flexible array member - because you're creating a contiguous array of these structures, and don't have a separate index, you have to assume each one is a fixed size. This means that you might as well make each one a fixed size:
typedef struct Entry {
int counter;
unsigned char block[1024 - sizeof counter];
}
I am writing a light weight serialization function and need to include two variable sized arrays within this.
How should I track the size of each?
How should I define the struct?
Am I going about this all wrong?
EDIT: the result must be a contiguous block of memory
This resolves to something like
typedef struct
{
size_t arr_size_1, arr_size_2;
char arr_1[0/*arr_size_1 + arr_size_2*/];
} ...;
The size(s) should be in the front of the dynamic sized data, so that it doesn't move when expanding your array.
You cannot have 2 unknown sized arrays in your struct, so you must collapse them into one and then access the data relative from the first pointer.
typedef struct MyStruct_s
{
int variable_one_size;
void* variable_one_buf;
int variable_two_size;
void* variable_two_buf;
} MyStruct;
MyStruct* CreateMyStruct (int size_one, int size_two)
{
MyStruct* s = (MyStruct*)malloc (sizeof (MyStruct));
s->variable_one_size = size_one;
s->variable_one_buf = malloc (size_one);
s->variable_two_size = size_two;
s->variable_two_buf = malloc (size_two);
}
void FreeMyStruct (MyStruct* s)
{
free (s->variable_one_buf);
free (s->variable_two_buf);
free (s);
}
Since the data should be continuous in memory it is necessary to malloc a chunk of memory of the right size and manage it's contents more or less manually. You probably best create a struct that contains the "static" information and related management functions that do the memory management and give access to the "dynamic" members of the struct:
typedef struct _serial {
size_t sz_a;
size_t sz_b;
char data[1]; // "dummy" array as pointer to space at end of the struct
} serial;
serial* malloc_serial(size_t a, size_t b) {
serial *result;
// malloc more memory than just sizeof(serial), so that there
// is enough space "in" the data member for both of the variable arrays
result = malloc(sizeof(serial) - 1 + a + b);
if (result) {
result->sz_a = a;
result->sz_b = b;
}
return result;
}
// access the "arrays" in the struct:
char* access_a(serial *s) {
return &s->data[0];
}
char* access_b(serial *s) {
return &s->data[s->sz_a];
}
Then you could do things like this:
serial *s = ...;
memcpy(access_a(s), "hallo", 6);
access_a(s)[1] = 'e';
Also note that you can't just assign one serial to another one, you need to make sure that the sizes are compatible and copy the data manually.
In order to serialize variably-sized data, you have to have a boundary tag of some sort. The boundary tag can be either a size written right before the data, or it can be a special value that is not allowed to appear in the data stream and is written right after the data.
Which you choose depends on how much data you are storing, and if you are optimizing for size in the output stream. It is often easier to store a size before-hand, because you know how big to make the receiving buffer. If you don't then you have to gradually resize your buffer on load.
In some ways, I'd do things like Dan Olson. However:
1) I'd create the final struct by having two instances of a simpler struct that has just one variable array.
2) I'd declare the array with byte* and use size_t for its length.
Having said this, I'm still not entirely clear on what you're trying to do.
edit
If you want it contiguous in memory, just define a struct with two lengths. Then allocate a block big enough for both blocks that you want to pass, plus the struct itself. Set the two lengths and copy the two blocks immediately after. I think it should be clear how the lengths suffice to make the struct self-describing.