I am having difficulties to find a possible solution so I decided to post my question. I am writing a program in C, and:
i am generating a huge array containing a lot of pointers to ints, it is allocated dynamically and filled during runtime. So before I don't know which pointers will be added and how many. The problem is that they are just to many of them, so I need to shrink somehow the space.
IS there any package or tool available which could possibly encode my entries somehow or change the representation so that I save space?
Another question, I also thought about writing a file with my information, is this then kept in memory the whole time or just if I reopen the file again?
It seems like you are looking for a simple dynamic array (the advanced data type dynamic array, that is). There should be many implementations for this out there. You can simply start with a small dynamic array and push new items to the back just like you would do with a vector in c++ or java. One implementation would be GArray. You will only allocate the memory you need.
If you have to/want to do it manually, the usual method is to store the capacity and the size of the array you allocated along with the pointer in a struct and call realloc() from within push_back() whenever you need more space. Usually you should increase the size of your array by a factor of 1.3 to 1.4, but a factor of 2 will do if you're not expecting a HUGE array. If you call remove and your size is below a certain threshold (e.g. capacity/2) you shrink the array again with realloc();
Related
I read an article about Dynamically Sized Arrays on ITJungle and was wondering if this is not an "making easy thing much more complex" thing.
So as I understand if I define an static variable, including arrays, the runtime reserves the needed space at RUNTIME. So when defining a array of CHAR(10) DIM(10) the whole space would be reserved when starting the program.
So as the article says if I want to have a dynamically increasing array which resizes itself to fit the data like an List<String> in C#, I have to create a CHAR(10) DIM(10). I then have to re-allocate new space only if needed?
Why? The space is already reserved. What reason would someone have to base a array with (lets say) 100 bytes size on a pointer when only needing i.e. 80 bytes?
Am I just missing something? Is the "init-value" for sizing the array just to calm down the compiler so I don't get an error that the "compiler doesn't know the size at compile time"?
For normal arrays, you are correct that the space gets allocated at runtime as soon as the particular arrays scope is reached (start of the program for globals, start of subprocedure for subprocedures).
However, you will notice that the data structure is declared with based(pInfo). based is the keyword that will cause the memory to NOT be allocated. It will instead assume the all the memory for the data structure (included the array member) is already allocated at the location specified by the pointer passed to the based keyword (pInfo in this case).
Effectively, once you use the based keyword you are simply telling the compiler how you would like the memory at the specified pointer to be used but it is up to you to actually manage that memory.
In summary, if I understand your question properly, the statement you made about "knowing the size at compile time" is correct. RPG does not support pointer/array duality or array-like objects like some languages so you essentially just have to declare to RPG that you will NEVER go beyond "init-value" bounds.
I was recently in an interview that required me to choose over the two data structures for a problem, and now I have the question of:
What is the reasoning for using a Stack over an array if the only operations needed are push and pop? An array provides constant time for appending and popping the last element from it and it takes up generally less memory than implementing a Stack with a LinkedList. It also provides random access should it be required. Is the only reasoning because an array is typically of fixed size, so we need to dynamically resize the array for each element we put in? This still is in constant time though isn't it unless the penalty is disproportionate?
There are several aspects to consider here...
First, a Stack is an abstract data type. It doesn't define how to implement itself.
An array is (generally) a well defined concrete implementation, and might even be fixed size unless explicitly defined to be dynamic.
A dynamic array can be implemented such that it automatically grows by some factor when exhausted and also might shrink when fill rate drops. These operations are not constant time, but are actually amortized to constant time because the array doesn't grow or shrink in each operation. In terms of memory usage it's hard to imagine an array being more expensive then a linked list unless extremely under used.
The main problem with an array is large allocation size. This is both a problem of maximum limitation and memory fragmentation. Using a linked list avoids both issues because every entry has a small memory footprint.
In some languages like C++, the underlying container that the 'stack' class uses can actually be changed between a dynamic array (vector), linked list (list), or even a double ended queue (deque). I only mention this because its typically not fair to compare a stack vs an array (one is an interface, another is a data structure).
Most dynamic array implementations will allocate more space than is needed, and upon filling the array they will again resize to 2x the size and so on. This avoids allocations and keeps the performance of push generally constant time. However the occasional resize does require copying elements O(n), though this is usually said to amortized to constant time. So in general, you are correct in that this is efficient.
Linked lists on the other hand typically require allocations for every push, which can be somewhat expensive, and the node's they create are larger in size than a single element in the array.
One possible advantage of linked lists, however, is that they do not require contiguous memory. If you have many many elements, its possible that you can fail to allocate a large enough block of memory for an array. Having said that, linked lists take up more memory... so its a bit of a wash.
In C++ for example, the stack by default uses the deque container. The deque is typically implemented as a dynamic array of 'pages' of memory. Each page of memory is fixed in size, which allows the container to actually have random access properties. Moreover, since each page is separate, then the entire container does not require contiguous memory meaning that it can store many many elements. Resizing is also cheap for a deque because it simply allocates another page, making it a great choice for a large stack.
I'm looking for a way to allocate additional memory (in C) at runtime, for an existing structure (that already had its memory assigned initially). I have a feeling I might be able to use memmove or something similar but that's still just a copy operation, and doesn't increase the amount of memory available to a structure at runtime. Also I don't want to have to copy the entire structure every time I need to do this, which will be many hundreds of times during the program (the structure is already huge). Can anyone help?
UPDATE: Thanks everyone for the replies. To give more detail, what I am trying to do is run an MPI-parallelised code that creates many instances of the structure (call it 'S') initially. Each instance of the structure contains an array 'T' which records the time of a particular event happening as the code is run. These events occur at runtime, and the number of events differs for each instance of S. For example, S[0] might see 100 events (and therefore need an array of 100 elements in length) but S[1] might see only 1 event (and S[2] 30 events, etc.) Therefore it would be very wasteful to allocate huge amounts of memory at the start for every instance of S (for which there are millions) since some might fill the array but others would not even come close. Indeed I have tried this and it is too much for the machine I am running it on.
I will try some of the ideas here and post my progress. Many thanks!
You could probably use realloc().
There is no way to do what you describe, because there is no way to guarantee that there will be available memory next to the one that your structure is currently occupying.
The standard thing to do is to allocate more memory and copy your data.
Of course if you can know (an estimate of) the size of the memory allocation that you need you can preallocate it and avoid copying.
Note, however, that the structures in C have a fixed size once they are declared, so it seems you don't really need to allocate more memory for an existing structure...
realloc is the only way to expand the existing dynamic memory. realloc will tries to expand the existing buffer, if it fails in expansion it will allocate new buffer for the total size required and it will copy the data from old buffer. If you dont want to do realloc every time(which internally will memmove most of the time) then you can try to reallocate more memory than actually you required.
realloc(buf_ptr, (actual_size + additional_size) * 2);
This way will reduce the frequency of calling realloc (and memmove).
Note : Implementation of realloc is different in some architecture, it will never tries to expand the memory it always tries to allocate buffer for total size. So in those platforms memmove will be called for every call to realloc.
It sounds like you are looking for the C feature called flexible array member (example). It is only well-defined for C standard C99 or later.
The last member of the struct will have to be declared as a flexible array member, which you initially malloc, and later realloc (and of course memcpy to do the actual copying).
I have a 2D array that I have split up into several 1D arrays and made those 1D arrays into OpenCL buffers. Sometimes I need a kernel function to take the entire 2D array but since its size is determined at runtime I cannot just make enough kernel arguments as there are 1D arrays (plus there can be over 1000 1D arrays). I am hoping that there is some way I can take the 1D array holding OpenCL buffers and combine them into one large buffer that has the entire data and send it to my kernel. Right now the only way I can see of doing this is if I read the data from the 1D buffers back to my program, arrange them into a giant 1D array and write the new buffer back to my compute device this seems like it will be extremely slow, is there any other way?
Here are a couple of ideas (though I admit they are not ideal).
Instead of copying the buffers back to your program, then building new buffers from it, you can use the clEnqueueCopyBuffer() method (or the clEnqueueCopyBufferRect(), depending on your situation) to copy data from one buffer to another. I believe (but I wouldn't swear to it) that how this copy is performed is implementation dependent, but it seems that a buffer that resides in device memory could be copied to another buffer in device memory without the need to cross the bus back to host memory.
Of course (if I understand correctly), copying is not really what you wanted anyway. How about using the clCreateSubBuffer() method? This method can make a new buffer that simply points to a sub-section of an existing buffer (without actually making a copy of its own). To do this, (from my understanding of what you've described) you would need to make the large 2D buffer, then create a number of light-weight 1D sub-buffers that point to regions of this memory.
In this way, you can pass the buffer that represents the whole 2D array when necessary, but just pass one or more 1D sub-buffers when that is all that is required.
I tested clCreateSubBuffer (whith his release) and a saw that it was slower than copy, better than create/ release, but ...
:(
System:
OpenCL 1.1 AMD-APP-SDK-v2.5 (684.212) FULL_PROFILE
Radeon 5870
I know its possible to increase the size of a dynamically allocated array.
But can I increase the size of a statically allocated array?
If yes,how?
EDIT: Though this question is intended for C language, consider other languages too.Is it possible in any other language?
Simple answer is no, this cannot be done. Hence the name "static".
Now, lots of languages have things that look like statically allocated arrays but are actually statically allocated references to a dynamically allocated array. Those you could resize.
in VB .NET it would be:
Redim Preserve ArrayName(NewSize)
not sure what langauge you're after though...
And I wouldn't use this command a lot... its terribly inefficient. Linked lists and growing data structures are much more efficient.
No. It is not. There are two options here:
Use a dynamic one
Or,at the risk of wasting memory, if you have an idea about the maximum number of elements that the array will store, statically allocate accordingly
Yes, that was C.
If you're careful, you can use alloca(). The array is allocated on the stack, but in terms of the code style it's a lot like if you used malloc (you don't have to free it though, that's done automatically). I'll let you decide whether to call that a "static" array.
No. Static allocation gives the compiler permission to make all kinds of assumptions which are then baked into the program during compilation.
Among those assumptions are that:
it is safe to put other data immediately after the array (not leaving you room to grow), and
that the array starts at a certain address, which then becomes part of the machine code of the program; you can't allocate a new array somewhere (and use it) because the references to the address can't be updated.
(Well, references could be updated, if the program was stored in ram, but self-modifying programs are highly frowned upon, and surely more trouble than dynamic arrays.)
Technically, in C it isn´t even possible to increase the size of a dynamically allocated array.
In fact, realloc() does some kind of "create new object & copy the data" routine. It does not modify the size of an existant heap-memory object at all.
So the answer is simple as that, that you are not be able to change the size of any object or array of objects after it has been allocated, neither if it was dynamically or statically allocated.
What you can do is to use the same strategy by developing a function which is creating another static allocated array of objects with the desired size and copy the data. If the new array of objects is smaller than the old one, the values inside the difference are discarded.
The only difference is, that the size of the new array, equivalent to the size of the old array, need to be fixed at compile-time.