I've read all other questions but response aren't helping me so much.
Once again, I'm not a developer but a wood worker trying to make something far too complicated for his own brain.
I work on a PIC its alignment is 4 bytes, all structures are define with attribute((packed)).
I've found a way to do it but it uses malloc and str/mem-cpy, methods that aren't safe for interrupt, or for malloc that I shouldn't be using at all (cf. my previous question)
Said structure contains 16 unsigned char, one s16 (2 bytes) et three s32 (4 bytes), so its 30 bytes long, with 4bytes alignement making it 32bytes long.
(char are coded on 8bits).
1) Am I wrong until here?
int len =sizeof(data_out.point[i]);
unsigned char* raw;
raw = malloc(len);
memcpy(raw, &data_out.point[i], len);
Data_res[count].pData = malloc(len);
for (int k = 0; k<len; k++)
{
Data_res[count].pData[k] = (uint8_t)(raw[k]);
}
Data_res[count].DataLen=len;
count++;
}
data_out is a stucture (members are not relevant except following one)
point is the structure described before.
Data_res is a structure that store an uchar buffer (pData) and contains lenght and status (locked, writing, etc to avoid multi-access).
1) its not working 100%, sometimes result are really strange.
2) since yesterday I understand why its bad (malloc on not shared memory, casting malloc, interruption safety, etc).
How to do the same thing without mallocs/memcpy ?
note: I need this for debug output, I was going to just let it go, but I don't like to keep things unfinished...
Don't cast the result of malloc in C. If you have to cast, you're using a C++ compiler, and you should be using new instead of malloc.
sizeof is an operator (not a function) that determines the size of objects of that type (as a size_t, not an int), so if data.point[i] is a char * for example, sizeof data.point[i] is the same value as sizeof (char *). It is a common mistake to assume sizeof determines the size of the object pointed at by a pointer; that hold true for array types, but not pointer types.
You're leaking memory pointed at by raw when your function returns. You should be using automatic storage duration rather than dynamic allocation there. i.e. unsigned char raw[len];.
You should be performing machine-independent serialisation, e.g. translating each field to a value that doesn't depend upon machine architecture, if you're planning on transporting this data to other machines.
Finally, to answer your question, the only way you can substitute Data_res[count].pData = malloc(len); seems that you need some other object (see automatic storage duration above) to point to instead, i.e. Data_res[count].pData = raw;... You might need to declare raw in the caller function, to avoid it being destroyed when the function you provided returns.
Related
Hello everyone so i want to put a size_t variable of a number up to 4096 in order to keep track of how much "memory" i used in the array (yes it has to be a char array). So for example
#include <stdio.h>
#include <unistd.h>
int main(void)
{
char mem[4096] = {0};
size_t size = 4095;
mem[0] = size;
//then somehow be able to size = mem[0] to be able to recall the size later.
return 0;
}
I just want to be able to put a number in the first array elment in order to be able to keep track of how many elements ive used.
Im implementating a heap memory and i also have to implement malloc(), and free() i have a char array that simulates the heap memory, and i cant use global/ static local variables but i want to be able to keep track of the heap size. My best thought of action was to store the heap size in the "Heap" (char array) to be able to keep track of it.
If you must use a char array for storing the number of bytes used, you have (at least) two choices:
create a union of an uint16_t with the array. This will allow you to use the first two bytes to hold the number of bytes used.
essentially the same as #1 but do the math manually, and maintain the bytes yourself.
I'd recommend #1.
About the most straightforward way to do this that is defined by the C standard is with to copy the bytes from the size object to the memory:
memcpy(&mem[index], &size, sizeof size);
Good compilers with optimization enabled will optimize this to a store from size to memory if they can determine &mem[index] is suitably aligned (or an unaligned store instruction is available and a good method). The reverse, to retrieve the size, is memcpy(&size, &mem[index], sizeof size);.
It can be done with unions, but that may be cumbersome in this situation. The union would have to include the entire array.
Old C code would simply cast an address in mem to the desired type, as with * (size_t *) &mem[index] = size;. However, the C standard does not define the behavior when an array defined as an array of char is accessed through a size_t lvalue as the above does. This worked with old compilers, and modern compilers may still support it if an appropriate switch is given to ask the compiler to support it, but it is needless, so new code should avoid this in favor of more portability.
Note that you do not need to use a size_t if the size will only be 4096. For this, you can use uint16_t.
Another option is to calculate the two bytes:
static void RecordSize(unsigned char *memory, uint16_t size)
{
memory[0] = size >> 8; // Record high bits.
memory[1] = size; // Record low bits.
// This will automatically “chop off” high bits.
}
static uint16_t RetrieveSize(unsigned char *memory)
{
return (uint16_t) memory[0] << 8 | memory[1];
}
…
RecordSize(&mem[index], size);
…
size = RetrieveSize(&mem[index];
That would be used more rarely than the direct copy, but it could be useful if data is being recorded in memory to transmit to another computer system that may use a different byte ordering for its integers. This method explicitly stores the high bits first and the low bits second (or the code could be written to do the opposite, as desired).
Note that this code uses unsigned char. You should generally prefer to use unsigned char for manipulations of bytes as arbitrary data, as it avoids some problems that can occur with signed types. (The C standard leaves it implementation-defined whether the char type is signed or unsigned.)
For an experiment I created a function to initialize an array that have a built-in length like in java
int *create_arr(int len) {
void *ptr = malloc(sizeof(int[len + 1]));
int *arr = ptr + sizeof(int);
arr[-1] = len;
return arr;
}
that can be later be used like this
int *arr = create_arr(12);
and allow to find the length at arr[-1]. I was asking myself if this is a common practice or not, and if there is an error in what i did.
First of all, your code has some bugs, mainly that in standard C you can't do arithmetic on void pointers (as commented by MikeCAT). Probably a more typical way to write it would be:
int *create_arr(int len) {
int *ptr = malloc((len + 1) * sizeof(int));
if (ptr == NULL) {
// handle allocation failure
}
ptr[0] = len;
return ptr + 1;
}
This is legal but no, it's not common. It's more idiomatic to keep track of the length in a separate variable, not as part of the array itself. An exception is functions that try to reproduce the effect of malloc, where the caller will later pass back the pointer to the array but not the size.
One other issue with this approach is that it limits your array length to the maximum value of an int. On, let's say, a 64-bit system with 32-bit ints, you could conceivably want an array whose length did not fit in an int. Normally you'd use size_t for array lengths instead, but that won't work if you need to fit the length in an element of the array itself. (And of course this limitation would be much more severe if you wanted an array of short or char or bool :-) )
Note that, as Andrew Henle comments, the pointer returned by your function could be used for an array of int, but would not be safe to use for other arbitrary types as you have destroyed the alignment promised by malloc. So if you're trying to make a general wrapper or replacement for malloc, this doesn't do it.
Apart from the small mistakes that have already been pointed in comments, this is not common, because C programmers are used to handle arrays as an initial pointer and a size. I have mainly seen that in mixed programming environments, for example in Windows COM/DCOM where C++ programs can exchange data with VB programs.
Your array with builtin size is close to winAPI BSTR: an array of 16 bits wide chars where the allocated size is at index -1 (and is also a 16 bit integer). So there is nothing really bad with it.
But in the general case, you could have an alignment problem. malloc does return a pointer with a suitable alignment for any type. And you should make sure that the 0th index of your returned array also has a suitable alignment. If int has not the larger alignment, it could fail...
Furthermore, as the pointer is not a the beginning of the allocated memory, the array would require a special function for its deallocation. It should probaby be documented in a red flashing font, because this would be very uncommon for most C programmers.
This technique is not as uncommon as people expect. For example stb header only library for image processing uses this method to implement type safe vector like container in C. See https://github.com/nothings/stb/blob/master/stretchy_buffer.h
It would be more idiomatic to do something like:
struct array {
int *d;
size_t s;
};
struct array *
create_arr(size_t len)
{
struct array *a = malloc(sizeof *a);
if( a ){
a->d = malloc(len * sizeof *a->d);
a->s = a->d ? len : 0;
}
return a;
}
According to this,
https://gcc.gnu.org/onlinedocs/gcc/Zero-Length.html
It is said that the benefit is
They are very useful as the last element of a structure that is really
a header for a variable-length object
What does it mean?
The zero-length array is a GCC extension (read as: not standard) which you should not use.
While recent versions of C allow for someting similar (flexible array member with empty brackets), C++ knows no such thing. As people often mix C and C++, this is a possible source of confusion.
Instead, an array of length 1 should be used, which is standards-compliant under both C and C++, and which just works with every compiler.
What is this useful for at all?
Sometimes you need to access "invalid" out-of-bounds data knowing that it is valid in reality. In the strictest sense, this is undefined behavior (since you are accessing out-of-bounds values which are indeterminate, and using indeterminate values is UB), but that is only for what the compiler knows, not for what it fact, so it nevertheless "works fine".
For example, you might receive framed data on the network consisting of a tag word, a length, and an amount of data corresponding to the length given. Or an operating system function might return a variable amount of results to you (a couple of Win32 API functions work that way, for example).
In either case, you have a unknown (unknown at compile time) number of elements at the end of this structure, so it is not possible to define a single legitimate structure to hold everything.
That is what flexible array members are for. And with this, it is explained why they must be the last member as well. It doesn't make sense for something that could have "any size" to be anywhere but at the end -- it's impossible for the compiler to lay out any members after it, not knowing its size.
(In case you wonder how the compiler can ever free the storage not knowing the objects's size... it cannot! There normally exists an explicit function for freeing such an object as part of the API, which takes care of this exact problem.)
It's probably best to demonstrate with a small example:
#include <stdio.h>
#include <stdlib.h>
#define BLOB_TYPE_FOO 0xBEEF
struct blob {
/* Part of your object header... perhaps describing the type of blob. */
int type;
/* This is actually the length of the "data" field below */
unsigned length;
/* The data */
unsigned char data[];
};
struct blob *
create_blob(int type, size_t size)
{
/* Allocate enough space for the "header" and "size" bytes of data. */
struct blob *x = calloc(1, sizeof(struct blob) + size);
x->type = type;
x->length = size;
return x;
}
int
main(void)
{
/* Note that sizeof(struct blob) doesn't include the data field. */
printf("sizeof(struct blob): %zu\n", sizeof(struct blob));
struct blob *x = create_blob(BLOB_TYPE_FOO, 1000);
/*
You can manipulate data here, but be careful not to exceed the
allocated size.
*/
size_t i;
for (i = 0; i < 1000; i++)
{
x->data[i] = 'A' + (i % 26);
}
/*
Since data was allocated with the rest of the header, everything is
freed.
*/
free(x);
return 0;
}
The nice part about this setup is that sizeof(struct blob) represents the size of the "object header" (on my machine, that's 8 bytes), and that since you allocate the whole object together, a single free() is all that is needed to release the memory.
Like others have stated here, this is a non-standard extension and you should really consider using it with care. Damon's answer is the better way to go, though the sizeof() operation is not quite the right size (it's a bit too large to represent the size of the actual header). It's not too hard to workaround that problem though.
You cannnot have the array of 0 length because if you try to make a zero length array then it would mean that you are trying to create a pointer to nothing which is not correct. The C standard says:
Flexible array members are written as contents[] without the 0.
Flexible array members have incomplete type, and so the sizeof operator may not be applied. As a quirk of the original implementation of zero-length arrays, sizeof evaluates to zero.
Flexible array members may only appear as the last member of a struct that is otherwise non-empty.
A structure containing a flexible array member, or a union containing such a structure (possibly recursively), may not be a member of a structure or an element of an array. (However, these uses are permitted by GCC as extensions.
I've got a function that receives data of an unknown type allocated on the heap that I'd like to copy it into a specific position in an array. The issue is that this function can't have access to the type of the data; just to its address and size.
This is the information the function has access to:
unsigned int count; /* length of the array */
unsigned int pos; /* index in the array */
void *data; /* source data */
size_t elemsize; /* sizeof data type */
void *array = malloc(elemcount * elemsize);
The solution that I've reached was to use memcpy and cast the array to char* in order to be able to address it on a byte-level.
memcpy((char*) array + elemsize * pos, (char*) data, elemsize);
However, I've read about possible problems when using memcpy with unaligned pointers in certain architectures.
Would casting the source data to char* make this operation safe?
Are there any situations in which addressing the array by converting it to char* (i.e. any system in which sizeof(char) != 1) or using memcpy in this way could cause problems?
Would casting the source data to char* make this operation safe?
That makes no difference.
Are there any situations in which addressing the array by converting it to char* (i.e. any system in which sizeof(char) != 1) or using memcpy in this way could cause problems?
There are no systems with sizeof(char) != 1. The standard specifies that it must be 1.
There is never any problem using memcpy so long as the bounds are not exceeded of the objects being read and written, and the destination is writable, and the objects don't overlap. In particular there is not any alignment problem.
(There may be a problem when you attempt to access the object that you copied into, if you didn't successfully build up a valid representation for that object).
I've read about possible problems when using memcpy with unaligned pointers in certain architectures.
Either you misunderstood what you read, or the information was wrong. If you are talking about the code in the linked ARM article:
void example (unsigned int * const unaligned_ptr)
{
if unaligned_ptr is not correctly aligned for unsigned int then the behaviour is already undefined before execution enters the body of the function. The page seems to be offering a way to attempt to recover from undefined behaviour . This is a rather bad idea IMHO, it'd be better to not initiate undefined behaviour in the first place; or to detect it and then throw some sort of fatal exception so that the problem can be debugged and corrected.
I am writing a library for various mathematical computations in C. Several of these need some "scratch" space -- memory that is used for intermediate calculations. The space required depends on the size of the inputs, so it cannot be statically allocated. The library will typically be used to perform many iterations of the same type of calculation with the same size inputs, so I'd prefer not to malloc and free inside the library for each call; it would be much more efficient to allocate a large enough block once, re-use it for all the calculations, then free it.
My intended strategy is to request a void pointer to a single block of memory, perhaps with an accompanying allocation function. Say, something like this:
void *allocateScratch(size_t rows, size_t columns);
void doCalculation(size_t rows, size_t columns, double *data, void *scratch);
The idea is that if the user intends to do several calculations of the same size, he may use the allocate function to grab a block that is large enough, then use that same block of memory to perform the calculation for each of the inputs. The allocate function is not strictly necessary, but it simplifies the interface and makes it easier to change the storage requirements in the future, without each user of the library needing to know exactly how much space is required.
In many cases, the block of memory I need is just a large array of type double, no problems there. But in some cases I need mixed data types -- say a block of doubles AND a block of integers. My code needs to be portable and should conform to the ANSI standard. I know that it is OK to cast a void pointer to any other pointer type, but I'm concerned about alignment issues if I try to use the same block for two types.
So, specific example. Say I need a block of 3 doubles and 5 ints. Can I implement my functions like this:
void *allocateScratch(...) {
return malloc(3 * sizeof(double) + 5 * sizeof(int));
}
void doCalculation(..., void *scratch) {
double *dblArray = scratch;
int *intArray = ((unsigned char*)scratch) + 3 * sizeof(double);
}
Is this legal? The alignment probably works out OK in this example, but what if I switch it around and take the int block first and the double block second, that will shift the alignment of the double's (assuming 64-bit doubles and 32-bit ints). Is there a better way to do this? Or a more standard approach I should consider?
My biggest goals are as follows:
I'd like to use a single block if possible so the user doesn't have to deal with multiple blocks or a changing number of blocks required.
I'd like the block to be a valid block obtained by malloc so the user can call free when finished. This means I don't want to do something like creating a small struct that has pointers to each block and then allocating each block separately, which would require a special destroy function; I'm willing to do that if that's the "only" way.
The algorithms and memory requirements may change, so I'm trying to use the allocate function so that future versions can get different amounts of memory for potentially different types of data without breaking backward compatibility.
Maybe this issue is addressed in the C standard, but I haven't been able to find it.
The memory of a single malloc can be partitioned for use in multiple arrays as shown below.
Suppose we want arrays of types A, B, and C with NA, NB, and NC elements. We do this:
size_t Offset = 0;
ptrdiff_t OffsetA = Offset; // Put array at current offset.
Offset += NA * sizeof(A); // Move offset to end of array.
Offset = RoundUp(Offset, sizeof(B)); // Align sufficiently for type.
ptrdiff_t OffsetB = Offset; // Put array at current offset.
Offset += NB * sizeof(B); // Move offset to end of array.
Offset = RoundUp(Offset, sizeof(C)); // Align sufficiently for type.
ptrdiff_t OffsetC = Offset; // Put array at current offset.
Offset += NC * sizeof(C); // Move offset to end of array.
unsigned char *Memory = malloc(Offset); // Allocate memory.
// Set pointers for arrays.
A *pA = Memory + OffsetA;
B *pB = Memory + OffsetB;
C *pC = Memory + OffsetC;
where RoundUp is:
// Return Offset rounded up to a multiple of Size.
size_t RoundUp(size_t Offset, size_t Size)
{
size_t x = Offset + Size - 1;
return x - x % Size;
}
This uses the fact, as noted by R.., that the size of a type must be a multiple of the alignment requirement for that type. In C 2011, sizeof in the RoundUp calls can be changed to _Alignof, and this may save a small amount of space when the alignment requirement of a type is less than its size.
If the user is calling your library's allocation function, then they should call your library's freeing function. This is very typical (and good) interface design.
So I would say just go with the struct of pointers to different pools for your different types. That's clean, simple, and portable, and anybody who reads your code will see exactly what you are up to.
If you do not mind wasting memory and insist on a single block, you could create a union with all of your types and then allocate an array of those...
Trying to find appropriately aligned memory in a massive block is just a mess. I am not even sure you can do it portably. What's the plan? Cast pointers to intptr_t, do some rounding, then cast back to a pointer?
The latest C11 standard has the max_align_t type (and _Alignas specifier and _Alignof operator and <stdalign.h> header).
GCC compiler has a __BIGGEST_ALIGNMENT__ macro (giving the maximal size alignment). It also proves some extensions related to alignment.
Often, using 2*sizeof(void*) (as the biggest relevant alignment) is in practice quite safe (at least on most of the systems I heard about these days; but one could imagine weird processors and systems where it is not the case, perhaps some DSP-s). To be sure, study the details of the ABI and calling conventions of your particular implementation, e.g. x86-64 ABI and x86 calling conventions...
And the system malloc is guaranteed to return a sufficiently aligned pointer (for all purposes).
On some systems and targets and some processors giving a larger alignment might give performance benefit (notably when asking the compiler to optimize). You may have to (or want to) tell the compiler about that, e.g. on GCC using variable attributes...
Don't forget that according to Fulton
there is no such thing as portable software, only software that has been ported.
but intptr_t and max_align_t is here to help you....
Note that the required alignment for any type must evenly divide the size of the type; this is a consequence of the representation of array types. Thus, in the absence of C11 features to determine the required alignment for a type, you can just estimate conservatively and use the type's size. In other words, if you want to carve up part of an allocation from malloc for use storing doubles, make sure it starts at an offset that's a multiple of sizeof(double).