I have to create an array of char pointers each of them of size 10000000 using the best and optimized way to do this in C.
I think this will do (haven't checked for nulls though):
int i;
int num_arrays;
char **huge_char_array;
num_arrays = 10; //number of arrays you want.
huge_char_array = (char **)malloc(sizeof(char *) * num_arrays);
for(i = 0; i < num_arrays; i++)
{
huge_char_array[i] = (char *)malloc(sizeof(char) * 10000000);
}
I believe this is the optimal way because there is only one dynamic allocation, reducing overhead from fragmentation of the heap and the time taken to allocate. You can use the STRING_INDEX utility function to access the nth string.
Also, using calloc() instead of malloc() zeroes out buffer to ensure that all strings are NUL terminated.
#define STRING_SIZE 10000000
#define NUM_STRINGS 10
#define STRING_INDEX(array, string_idx) ((array) + (string_idx) * STRING_SIZE)
int main(int argc, char **argv) {
char *array_of_strings = calloc(NUM_STRINGS, STRING_SIZE);
// Access 8th character of 7th string
char c = STRING_INDEX(array_of_strings, 7)[8];
// Use array
// Free array when done
free(array_of_strings);
return 0;
}
I have been allocating 50K with each request I process this resulted in memory fragmentation. So I switched to allocate 50k * 100 = 5MB and reuse the pool when ever I need to allocate one more. If the requests increases beyond my pool capacity I double the pool size to avoid memory fragmentations. Based on this I would recommend allocating a huge chunk of memory maybe in your case 10 * 10000000 to reuse it. I am sorry but I cannot share the code that handle the pool here. Also use malloc and free and do not use new and delete.
malloc won't through exception it will simply return null if there are no memory available.
if you insist on using malloc for each element individually you can create a heap with maximum size to avoid fragmentations and in that case you will have to define the heap size.
please referee to MSDN for more details about how to create a heap and use it or the other allocation methods available for windows users.
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366533(v=vs.85).aspx
char *arr[SOME_SIZE];
for(int i = 0; i < SOME_SIZE; ++i) {
arr[i] = malloc(10000000);
if(!arr[i])
// allocation failed, do something
}
Just realize that you're allocating ~9.5MB for every element in the array (i.e., SOME_SIZE * 10000000 total bytes)
Related
I'm creating a C-library with .h and .c files for a ring buffer. Ideally, you would initialize this ring buffer library in the main project with something like ringbuff_init(int buff_size); and the size that is sent, will be the size of the buffer. How can I do this when arrays in C needs to be initialized statically?
I have tried some dynamically allocating of arrays already, I did not get it to work. Surely this task is possible somehow?
What I would like to do is something like this:
int buffSize[];
int main(void)
{
ringbuffer_init(100); // initialize buffer size to 100
}
void ringbuffer_init(int buff_size)
{
buffSize[buff_size];
}
This obviously doesn't compile because the array should have been initialized at the declaration. So my question is really, when you make a library for something like a buffer, how can you initialize it in the main program (so that in the .h/.c files of the buffer library) the buffer size is set to the wanted size?
You want to use dynamic memory allocation. A direct translation of your initial attempt would look like this:
size_t buffSize;
int * buffer;
int main(void)
{
ringbuffer_init(100); // initialize buffer size to 100
}
void ringbuffer_init(size_t buff_size)
{
buffSize = buff_size;
buffer = malloc(buff_size * sizeof(int));
}
This solution here is however extremely bad. Let me list the problems here:
There is no check of the result of malloc. It could return NULL if the allocation fails.
Buffer size needs to be stored along with the buffer, otherwise there's no way to know its size from your library code. It isn't exactly clean to keep these global variables around.
Speaking of which, these global variables are absolutely not thread-safe. If several threads call functions of your library, results are inpredictible. You might want to store your buffer and its size in a struct that would be returned from your init function.
Nothing keeps you from calling the init function several times in a row, meaning that the buffer pointer will be overwritten each time, causing memory leaks.
Allocated memory must be eventually freed using the free function.
In conclusion, you need to think very carefully about the API you expose in your library, and the implementation while not extremely complicated, will not be trivial.
Something more correct would look like:
typedef struct {
size_t buffSize;
int * buffer;
} RingBuffer;
int ringbuffer_init(size_t buff_size, RingBuffer * buf)
{
if (buf == NULL)
return 0;
buf.buffSize = buff_size;
buf.buffer = malloc(buff_size * sizeof(int));
return buf.buffer != NULL;
}
void ringbuffer_free(RingBuffer * buf)
{
free(buf.buffer);
}
int main(void)
{
RingBuffer buf;
int ok = ringbuffer_init(100, &buf); // initialize buffer size to 100
// ...
ringbuffer_free(&buf);
}
Even this is not without problems, as there is still a potential memory leak if the init function is called several times for the same buffer, and the client of your library must not forget to call the free function.
Static/global arrays can't have dynamic sizes.
If you must have a global dynamic array, declare a global pointer instead and initialize it with a malloc/calloc/realloc call.
You might want to also store its size in an accompanying integer variable as sizeof applied to a pointer won't give you the size of the block the pointer might be pointing to.
int *buffer;
int buffer_nelems;
char *ringbuffer_init(int buff_size)
{
assert(buff_size > 0);
if ( (buffer = malloc(buff_size*sizeof(*buffer)) ) )
buffer_nelems = buff_size;
return buffer;
}
You should use malloc function for a dynamic memory allocation.
It is used to dynamically allocate a single large block of memory with the specified size. It returns a pointer of type void which can be cast into a pointer of any form.
Example:
// Dynamically allocate memory using malloc()
buffSize= (int*)malloc(n * sizeof(int));
// Initialize the elements of the array
for (i = 0; i < n; ++i) {
buffSize[i] = i + 1;
}
// Print the elements of the array
for (i = 0; i < n; ++i) {
printf("%d, ", buffSize[i]);
}
I know I'm three years late to the party, but I feel I have an acceptable solution without using dynamic allocation.
If you need to do this without dynamic allocation for whatever reason (I have a similar issue in an embedded environment, and would like to avoid it).
You can do the following:
Library:
int * buffSize;
int buffSizeLength;
void ringbuffer_init(int buff_size, int * bufferAddress)
{
buffSize = bufferAddress;
buffSizeLength = buff_size;
}
Main :
#define BUFFER_SIZE 100
int LibraryBuffer[BUFFER_SIZE];
int main(void)
{
ringbuffer_init(BUFFER_SIZE, LibraryBuffer ) // initialize buffer size to 100
}
I have been using this trick for a while now, and it's greatly simplified some parts of working with a library.
One drawback: you can technically mess with the variable in your own code, breaking the library. I don't have a solution to that yet. If anyone has a solution to that I would love to here it. Basically good discipline is required for now.
You can also combine this with #SirDarius 's typedef for ring buffer above. I would in fact recommend it.
I have a char array, we know that that a char size is 1 byte. Now I have to collect some char -> getchar() of course and simultaneously increase the array by 1 byte (without malloc, only library: stdio.h)
My suggestion would be, pointing to the array and somehow increase that array by 1 till there are no more chars to get OR you run out of Memory...
Is it possible to increase char array while using it, WITHOUT malloc?
No.
You cannot increase the size of a fixed size array.
For that you need realloc() from <stdlib.h>, which it seems you are not "allowed" to use.
Is it possible to increase char array while using it, WITHOUT malloc?
Quick answer: No it is not possible to increase the size of an array without reallocating it.
Fun answer: Don't use malloc(), use realloc().
Long answer:
If the char array has static or automatic storage class, it is most likely impossible to increase its size at runtime because keeping it at the same address that would require objects that are present at higher addresses to be moved or reallocated elsewhere.
If the array was obtained by malloc, it might be possible to extend its size if no other objects have been allocated after it in memory. Indeed realloc() to a larger size might return the same address. The problem is it is impossible to predict and if realloc returns a different address, the current space has been freed so pointers to it are now invalid.
The efficient way to proceed with this reallocation is to increase the size geometrically, by a factor at a time, 2x, 1.5x, 1.625x ... to minimize the number of reallocations and keep linear time as the size of the array grows linearly. You would a different variable for the allocated size of the array and the number of characters that you have stored into it.
Here is an example:
#include <stdio.h>
#include <stdlib.h>
int main(void) {
char *a = NULL;
size_t size = 0;
size_t count = 0;
int c;
while ((c = getchar()) != EOF && c != '\n') {
if (count >= size) {
/* reallocate the buffer to 1.5x size */
size_t newsize = size + size / 2 + 16;
char *new_a = realloc(a, new_size);
if (new_a == NULL) {
fprintf("out of memory for %zu bytes\n", new_size);
free(a);
return 1;
}
a = new_a;
size = new_size;
}
a[count++] = c;
}
for (i = 0; i < count; i++) {
putchar(a[i]);
}
free(a);
return 0;
}
There are two ways to create space for the string without using dynamic memory allocation(malloc...). You can use a static array or an array with automatic storage duration, you need to specify a maximum amount, you might never reach. But always check against it.
#define BUFFER_SIZE 0x10000
Static
static char buffer[BUFFER_SIZE];
Or automatic (You need to ensure BUFFER_SIZE is smaller than the stack size)
int main() {
char buffer[BUFFER_SIZE];
...
};
There are also optimizations done by the operating system. It might lazily allocate the whole (static/automatic) buffer, so that only the used part is in the physical memory. (This also applies to the dynamic memory allocation functions.) I found out that calloc (for big chunks) just allocates the virtual memory for the program; memory pages are cleared only, when they are accessed (probably through some interrupts raised by the cpu). I compared it to an allocation with malloc and memset. The memset does unnessecary work, if not all bytes/pages of the buffer are accessed by the program.
If you cannot allocate a buffer with malloc..., create a static/automatic array with enough size and let the operating system allocate it for you. It does not occupy the same space in the binary, because it is just stored as a size.
I allocate a big memory , char* test= malloc(10000000); , then I put value on this memory , and do some work for each value.
What I want is , each 1000 index, I want to release all the memory until it.
For ex.
for(long i=0; i<10000000;i++)
DoSomeWork(test[i]);
if(i%1000==0)
releaseMemory(i-1000,i);
How can I do it in c?
I know that free can only free all of my allocate, but I don't want to wait to the end of work the free all the memory.
I want each 1000 works free all the 1000 back
I must to allocate all the memory in the begining of program.
What you want can be achieved by allocating the program in smaller chunks.
You have to adjust your algorithm to handle a bunch of small sub-arrays which you then can release after use.
In this case, it might be useful to allocate the chunks in reversed direction to give the libc the chance to release the freed memory to the underlying OS.
Let me enhance a bit here:
Assume you want an array with 10000000 (10 million) entries. Instead of allocating it as one chunk as depicted in the question, it could be possible to have
#define CHUNKSIZE 10000
#define ENTRYSIZE 8
#define NUM_CHUNKS 1000
void test(void)
{
void** outer_array = malloc(NUM_CHUNKS * sizeof(void*))
for (int i = 0; i < NUM_CHUNKS; i++) {
void * chunk = malloc(CHUNKSIZE * ENTRYSIZE);
outer_array[NUM_CHUNKS - 1 - i] = chunk;
// allocate them in reverse order
}
// now, set item #123456
size_t item_index = 123456;
// TODO check if the index is below the maximum
size_t chunk_index = item_index / CHUNKSIZE;
size_t index_into_chunk = item_index % CHUNKSIZE;
void * item_address = &outer_array[chunk_index][index_into_chunk * ENTRY_SIZE];
// after having processed one chunk, you can free it:
free(outer_array[0]);
outer_array[0] = NULL;
}
There are (roughly) two possibilities how a program can enhance the heap in order to allocate memory:
It can obtain a completely new memory block from the OS, indepedent from the "main address space". Then it can use it for allocation and return it to the OS as soon as it is free()d. This happens in some allocators if the allocation size is above a certain threshold.
It can enhance the program address space. Then, the new memory is added at the end. After free()ing the last memory block, the program address space can be reduced again. This happens in some allocators if the allocation size is below a certain threshold.
This way, your program's memory footprint decreases over time.
I allocate a 2D array of char * and every string length is 12.
50 rows and 2000000 columns.
Lets calculate it:
50*2000000 * (12(length)+8(for pointer)). I use 64 bit.
50*2000000 * 20 =2000000000 bits .. -> 2 GB.
When I check the memory monitor it shows that the process takes 4 GB.
(All that happened after allocation)
This is the code:
int col=2000000,row=50,i=0,j=0;
char *** arr;
arr=(char***)malloc(sizeof(char**)*row);
for(i=0;i<row;i++)
{
arr[i]=(char ** )malloc(sizeof(char*)*col);
for(j=0;j<col;j++)
{
arr[i][j]=(char*)malloc(12);
strcpy(arr[i][j],"12345678901");
arr[i][j][11]='\0';
}
}
May that be from the paging in Linux?
Each call of malloc is taking more memory than you ask. Malloc needs to store somewhere its internal info about allocated place, like size of allocated space, some info about neighbors chunks, etc. Also (very probably) each returned pointer is aligned to 16 bytes. In my estimation each allocation of 12 bytes takes 32 bytes of memory. If you want to save memory allocate all strings in one malloc and split them into sizes per 12 at your own.
Try the following:
int col=2000000,row=50,i=0,j=0;
char *** arr;
arr= malloc(sizeof(*arr)*row);
for(i=0;i<row;i++)
{
arr[i]= malloc(sizeof(*arr[i])*col);
char *colmem = malloc(12 * col);
for(j=0;j<col;j++)
{
arr[i][j] = colmem + j*12;
strcpy(arr[i][j],"12345678901");
}
}
I would re-write the code from scratch. For some reason, around 99% of all C programmers don't know how to correctly allocate true 2D arrays dynamically. I'm not even sure I'm one of the 1% who do, but lets give it a shot:
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
int main()
{
const int COL_N = 2000000;
const int ROW_N = 50;
char (*arr)[ROW_N] = malloc( sizeof(char[COL_N][ROW_N]) );
if(arr == NULL)
{
printf("Out of memory");
return 0;
}
for(int row=0; row<ROW_N; row++)
{
strcpy(arr[row], "12345678901");
puts(arr[row]);
}
free(arr);
return 0;
}
The important parts here are:
You should always allocate multi-dimensional arrays in adjacent memory cells or they are not arrays, but rather pointer-based lookup tables. Thus you only need one single malloc call.
This should save a bit of memory since you only need one pointer and it is allocated on the stack. No pointers are allocated on the heap.
Casting the return value of malloc is pointless (but not dangerous on modern compilers).
Ensure that malloc actually worked, particularly when allocating ridiculous amounts of memory.
strcpy copies the null termination, you don't need to do it manually.
There is no need for nested loops. You want to allocate a 2D array, not a 3D one.
Always clean up your own mess with free(), even though the OS might do it for you.
Ok, this is a C programming homework question. But I'm truly stuck.
I ask the user to input words, and then I insert the input into an array, but I can't have any control over the number of words the user types.
I guess what I'm asking is how do you declare a an array in C without declaring its length and without asking the user what the length should be.
I know this has something to do with malloc, but if you could give me some examples of how to do this, I would really appreciate it.
You can malloc a block of memory large enough to hold a certain number of array items.
Then, before you exceed that number, you can use realloc to make the memory block bigger.
Here's a bit of C code that shows this in action, reallocating an integer array whenever it's too small to hold the next integer.
#include <stdio.h>
#include <stdlib.h>
int main (void) {
int *xyzzy = NULL; // Initially NULL so first realloc is a malloc.
int currsz = 0; // Current capacity.
int i;
// Add ten integers.
for (i = 0; i < 10; i++) {
// If this one will exceed capacity.
if (i >= currsz) {
// Increase capacity by four and re-allocate.
currsz += 4;
xyzzy = realloc (xyzzy, sizeof(int) * currsz);
// Should really check for failure here.
}
// Store number.
xyzzy[i] = 100 + i;
}
// Output capacity and values.
printf ("CurrSz = %d, values =", currsz);
for (i = 0; i < 10; i++) {
printf (" %d", xyzzy[i]);
}
printf ("\n");
return 0;
}
You can realloc it every time like:
int size = 0;
char **array = malloc(0);
while(/* something */)
{
char *string = // get input
size++;
array = realloc(array, size * sizeof(char*));
array[size - 1] = string;
}
Or in chunks if you care about speed.
Yes, you want malloc. Checkout this tut.
http://www.cprogramming.com/tutorial/dynamic_memory_allocation.html
This site is good in general for learning.
Here is an example of using realloc, it is basically exactly what you are asking to do.
http://www.cplusplus.com/reference/clibrary/cstdlib/realloc/
0) obviously you will need multiple buffers, so you will need a list like structure: perhaps a record with char array 100 chars and a pointer to next structure
1) You need to capture the words char by char and store them in your buffer
2) once the buffer is full you allocate another record, chain it with the previous one and keep going until you are out of mem or the process is over.
That should be better performance than realloc function. I believe malloc is trying to give contious block of memory. Therefore the list like structure will be faster and work better.