Creating a million (or more) string array in C - c

I'm running into a problem creating an array big enough to store words from a large text document (think books).
Usually, I would just do:
char wordList[1000000][30];
But, as expected the program crashes as soon as it tries to initialize the array. So, I tried a few different things, such as:
char *wordList[30]
int k=0;
while(k<1000000){
wordList[k]= malloc(sizeof(char)*30);
k++;
}
This, too, didn't work. So I'm wondering if there is an easier way. I know its possible. For the first option, my research as lead my to believe the the variable is initialized on the stack (which has small memory) and segfaults.
I'm not sure why the second one fails. Any suggestions? I've searched everywhere I could to find an answer, but most of the suggestions are in java or c++ where you just call new, or arraylist etc.

wordList is an array of 30 char* pointers. You are accessing way beyond the limit of this array. Specifically, you are accessing up to million spaces, but the array only has 30 spaces. This will cause undefined behaviour.
You need to instead make sure wordList has enough space for 1000000 pointers.
This should be instead:
char *wordList[1000000];
This allows flexibility on the length of the words. The only fixed size here is the array size.
If you use:
wordList[k]= malloc(sizeof(char)*30);
Moreover, this will run into issues if the words are more than 29 characters, excluding the \0 character at the end. Although, their are not many words longer than 29 characters. Words as long as:
supercalifragilisticexpialidocious
Are hard to come by.
Furthermore, it depends on how you are reading these words from your text document. If you parse the words, and instead use a temporary buffer to store them, then you can do this:
wordList[k]= malloc(strlen(word)+1); /* +1 for null-terminator */
Which will allocate memory for any sized word you copy into wordList[k]. This will be more efficient for smaller words like "the" and "or", instead of allocating 30 spaces for any word.
Note: Allocating a million pointers on the heap beforehand is also very wasteful, this process should be done on an as needed basis. It might even be better to use char **wordList, to allow more flexibility with how many words you allocate.
For example, you could allocate a starting size:
size_t start_size = 1000;
char **wordList = malloc(start_size * sizeof(*wordlist));
Then if more words are found, you can realloc() more space as needed.
realloc() resizes block of memory it points to, and returns a pointer.
An example would be:
if (start_size == word_count) {
start_size *= 2;
wordList = realloc(wordList, start_size * sizeof(*wordList));
if (wordList == NULL) {
/* handle exit */
Which would return a pointer which holds start_size * 2 spaces.
You should also check the return of malloc() and realloc(), as they can return NULL if unsuccessful. At the end of your program, you should also free() the pointers allocated from malloc().

In practice, a good rule of thumb when programming in C is that "big" data should be allocated in heap memory.
(I am guessing that you are coding for an ordinary laptop or desktop machine running some common operating system; I'm actually thinking of a desktop Linux computer, like the machine I am answering here; but you could adapt my answer to a desktop running some Windows, to a tablet running some Android, to an Apple computer running some MacOSX)
Notice that your current wordList is 30 megabytes (that is sizeof(wordList)==30000000). That is big! But in some cases not big enough (probably not enough for the whole Saint James bible, and certainly not for the entire laws, decrees, and juridictions of US or of France, or for the archive of a century-old newspaper). You can easily find textual data of more than several dozens of megabytes today, such as all the messages on StackOverflow.
You may need to understand more about current operating systems to understand all of my answer; I recommend reading Operating Systems: Three Easy Pieces and, if your computer runs Linux and you want to code for Linux, Advanced Linux Programming.
You don't want to allocate a big array (or any kind of big data) as local variables (or with alloca(3) ...) on your call stack, because the call stack is limited (typically to one megabyte, or a few of them). On some special computers (think of expensive servers running some specially configured Linux) you could raise that (machine call stack) limit, but perhaps not easily, to several dozens of megabytes. Expecting a gigabyte call stack is not reasonable.
You probably don't want to have huge global or static data (those allocated at compile time in your data segment) of fixed size. If you did that, your program might still lack of memory (because you under-estimated that fixed size) or could not even start on smaller computers (e.g. if your data segment had 20Gbytes, your executable might start on my desktop with 32Gbytes, but would fail to start -at execve(2) time- on your laptop with only 16Gbytes).
The remaining option is usual practice: allocate all "big" data in heap memory by indirectly using primitives growing your virtual address space. In standard C, you'll extensively use malloc and friends (e.g. calloc) - with free to release memory. FWIW, the underlying primitives (to grow the virtual address space) on Linux include mmap(2) and related system calls (and may be called by the malloc implementation on your system). But the standard C dynamic memory allocation techniques (that is malloc & free) are hiding these gory (implementation specific) details in the C standard library (so your code using malloc & free could become portable with efforts from your part).
Read some coding rules, e.g. the GNU ones, related to memory usage, and robust programs, notably:
Avoid arbitrary limits on the length or number of any data structure, including file names, lines, files, and symbols, by allocating all data structures dynamically
(emphasis is mine)
Practically speaking, your wordList (that is a poor name, it is not a list but a vector or a table) should probably be a dynamically allocated array of pointers to dynamically allocated strings. You could declare it as char**wordList; and you want to keep its allocated size and used length (perhaps in two other global variables, size_t allocatedSize, usedLength; ...). You might prefer to use a struct ending with a flexible array member
Don't forget to check against failure of malloc. Perhaps you want to initialize your data with something like:
allocatedSize=1000;
usedLength=0;
wordList= calloc(allocatedSize, sizeof(char*));
if (!wordList) { perror("initial calloc wordlist"); exit(EXIT_FAILURE); };
Here is a routine to add a new word to your wordList; that routine does not check if the word is indeed new (maybe you want to use some other data structure, like some hash-table, or some self balancing binary search tree); if you want to keep only unique words, read some Introduction to Algorithms. Otherwise, you could use:
void add_new_word(const char*w) {
if (usedLength >= allocatedSize) {
size_t newsize = 4*usedLength/3+10;
(heuristically, we don't want to re-allocate wordList too often; hence the "geometrical" growth above)
char**newlist = calloc(newsize*sizeof(char*));
if (!newlist) { perror("calloc newlist"); exit(FAILURE); };
memcpy (newlist, wordList, usedLength*sizeof(char*));
free (wordList);
wordList = newlist;
allocatedSize = newsize;
};
// here we are sure that wordList is not full,
// so usedLength < allocatedSize
char *dw = strdup(w);
if (!dw) { perror("strdup failure"); exit(EXIT_FAILURE); };
we are using the very common strdup(3) function to copy some string into a heap allocated one. If your system don't have that, it is really easy to write, using strlen, malloc, strcpy ...
wordList[usedLength++] = dw;
} // end of add_new_word
After a call to add_new_word you know that a new word has been added at index usedLength-1 of wordList.
Notice that my add_new_word is checking against failure of malloc (including the one called from strdup) and satisfy the "robustness" criteria: all data is heap allocated!
BTW, some computers are (IMHO wrongly) enabling memory overcommitment. This is a system administration issue. I dislike that feature (because when it is enabled, malloc would never fail, but programs would crash badly when memory resources are exhausted).
FWIW, here is the routine to release memory, to be called near end of program (or registered thru atexit(3))
void destroy_word_list(void) {
if (!wordList) return;
for (size_t ix=0; ix<usedLength; ix++) free(wordList[ix]);
free (wordList);
usedLength = 0;
allocatedSize = 0;
wordList = NULL;
} // end of destroy_word_list
You may want to use valgrind e.g. to debug memory leaks.

Related

Why does malloc need to be used for dynamic memory allocation in C?

I have been reading that malloc is used for dynamic memory allocation. But if the following code works...
int main(void) {
int i, n;
printf("Enter the number of integers: ");
scanf("%d", &n);
// Dynamic allocation of memory?
int int_arr[n];
// Testing
for (int i = 0; i < n; i++) {
int_arr[i] = i * 10;
}
for (int i = 0; i < n; i++) {
printf("%d ", int_arr[i]);
}
printf("\n");
}
... what is the point of malloc? Isn't the code above just a simpler-to-read way to allocate memory dynamically?
I read on another Stack Overflow answer that if some sort of flag is set to "pedantic", then the code above would produce a compile error. But that doesn't really explain why malloc might be a better solution for dynamic memory allocation.
Look up the concepts for stack and heap; there's a lot of subtleties around the different types of memory. Local variables inside a function live in the stack and only exist within the function.
In your example, int_array only exists while execution of the function it is defined in has not ended, you couldn't pass it around between functions. You couldn't return int_array and expect it to work.
malloc() is used when you want to create a chunk of memory which exists on the heap. malloc returns a pointer to this memory. This pointer can be passed around as a variable (eg returned) from functions and can be used anywhere in your program to access your allocated chunk of memory until you free() it.
Example:
'''C
int main(int argc, char **argv){
int length = 10;
int *built_array = make_array(length); //malloc memory and pass heap pointer
int *array = make_array_wrong(length); //will not work. Array in function was in stack and no longer exists when function has returned.
built_array[3] = 5; //ok
array[3] = 5; //bad
free(built_array)
return 0;
}
int *make_array(int length){
int *my_pointer = malloc( length * sizeof int);
//do some error checking for real implementation
return my_pointer;
}
int *make_array_wrong(int length){
int array[length];
return array;
}
'''
Note:
There are plenty of ways to avoid having to use malloc at all, by pre-allocating sufficient memory in the callers, etc. This is recommended for embedded and safety critical programs where you want to be sure you'll never run out of memory.
Just because something looks prettier does not make it a better choice.
VLAs have a long list of problems, not the least of which they are not a sufficient replacement for heap-allocated memory.
The primary -- and most significant -- reason is that VLAs are not persistent dynamic data. That is, once your function terminates, the data is reclaimed (it exists on the stack, of all places!), meaning any other code still hanging on to it are SOL.
Your example code doesn't run into this problem because you aren't using it outside of the local context. Go ahead and try to use a VLA to build a binary tree, then add a node, then create a new tree and try to print them both.
The next issue is that the stack is not an appropriate place to allocate large amounts of dynamic data -- it is for function frames, which have a limited space to begin with. The global memory pool, OTOH, is specifically designed and optimized for this kind of usage.
It is good to ask questions and try to understand things. Just be careful that you don't believe yourself smarter than the many, many people who took what now is nearly 80 years of experience to design and implement systems that quite literally run the known universe. Such an obvious flaw would have been immediately recognized long, long ago and removed before either of us were born.
VLAs have their place, but it is, alas, small.
Declaring local variables takes the memory from the stack. This has two ramifications.
That memory is destroyed once the function returns.
Stack memory is limited, and is used for all local variables, as well as function return addresses. If you allocate large amounts of memory, you'll run into problems. Only use it for small amounts of memory.
When you have the following in your function code:
int int_arr[n];
It means you allocated space on the function stack, once the function will return this stack will cease to exist.
Image a use case where you need to return a data structure to a caller, for example:
Car* create_car(string model, string make)
{
Car* new_car = malloc(sizeof(*car));
...
return new_car;
}
Now, once the function will finish you will still have your car object, because it was allocated on the heap.
The memory allocated by int int_arr[n] is reserved only until execution of the routine ends (when it returns or is otherwise terminated, as by setjmp). That means you cannot allocate things in one order and free them in another. You cannot allocate a temporary work buffer, use it while computing some data, then allocate another buffer for the results, and free the temporary work buffer. To free the work buffer, you have to return from the function, and then the result buffer will be freed to.
With automatic allocations, you cannot read from a file, allocate records for each of the things read from the file, and then delete some of the records out of order. You simply have no dynamic control over the memory allocated; automatic allocations are forced into a strictly last-in first-out (LIFO) order.
You cannot write subroutines that allocate memory, initialize it and/or do other computations, and return the allocated memory to their callers.
(Some people may also point out that the stack memory commonly used for automatic objects is commonly limited to 1-8 mebibytes while the memory used for dynamic allocation is generally much larger. However, this is an artifact of settings selected for common use and can be changed; it is not inherent to the nature of automatic versus dynamic allocation.)
If the allocated memory is small and used only inside the function, malloc is indeed unnecessary.
If the memory amount is extremely large (usually MB or more), the above example may cause stack overflow.
If the memory is still used after the function returned, you need malloc or global variable (static allocation).
Note that the dynamic allocation through local variables as above may not be supported in some compiler.

Whats the difference between these array declarations in C?

People seem to say how malloc is so great when using arrays and you can use it in cases when you don't know how many elements an array has at compile time(?). Well, can't you do that without malloc? For example, if we knew we had a string that had max length 10 doesn't the following do close enough to the same thing?... Besides being able to free the memory that is.
char name[sizeof(char)*10];
and
char *name = malloc(sizeof(char)*10);
The first creates an array of chars on the stack. The length of the array will be sizeof(char)*10, but seeing as char is defined by the standard of being 1 in size, you could just write char name[10];
If you want an array, big enough to store 10 ints (defined per standard to be at least 2 bytes in size, but most commonly implemented as 4 bytes big), int my_array[10] works, too. The compiler can work out how much memory will be required anyways, no need to write something like int foo[10*sizeof(int)]. In fact, the latter will be unpredictable: depending on sizeof(int), the array will store at least 20 ints, but is likely to be big enough to store 40.
Anyway, the latter snippet calls a function, malloc wich will attempt to allocate enough memory to store 10 chars on the heap. The memory is not initialized, so it'll contain junk.
Memory on the heap is slightly slower, and requires more attention from you, who is writing the code: you have to free it explicitly.
Again: char is guaranteed to be size 1, so char *name = malloc(10); will do here, too. However, when working with heap memory, I -and I'm not alone in this- prefer to allocate the memory like so some_ptr = malloc(10*sizeof *some_ptr); using *some_ptr, is like saying 10 times the size of whatever type this pointer will point to. If you happen to change the type later on, you don't have to refactor all malloc calls.
General rule of thumb, to answer your question "can you do without malloc", is that you don't use malloc, unless you have to.
Stack memory is faster, and easier to use, but it is less abundant. This site was named after a well-known issue you can run into when you've pushed too much onto the stack: it overflows.
When you run your program, the system will allocate a chunk of memory that you can use freely. This isn't much, but plenty for simple computations and calling functions. Once you run out, you'll have to resort to allocating memory from the heap.
But in this case, an array of 10 chars: use the stack.
Other things to consider:
An array is a contguous block of memory
A pointer doesn't know/can't tell you how big a block of memory was allocated (sizeof(an_array)/sizeof(type) vs sizeof(a_pointer))
An array's declaration does not require the use of sizeof. The compiler works out the size for you: <type> my_var[10] will reserve enough memory to hold 10 elements of the given type.
An array decays into a pointer, most of the time, but that doesn't make them the same thing
pointers are fun, if you know what you're doing, but once you start adding functions, and start passing pointers to pointers to pointers, or a pointer to a pointer to a struct, that has members that are pointers... your code won't be as jolly to maintain. Starting off with an array, I find, makes it easier to come to grips with the code, as it gives you a starting point.
this answer only really applies to the snippets you gave, if you're dealing with an array that grows over time, than realloc is to be preferred. If you're declaring this array in a recursive function, that runs deep, then again, malloc might be the safer option, too
Check this link on differences between array and pointers
Also take a look at this question + answer. It explains why a pointer can't give you the exact size of the block of memory you're working on, and why an array can.
Consider that an argument in favour of arrays wherever possible
char name[sizeof(char)*10]; // better to use: char name[10];
Statically allocates a vector of sizeof(char)*10 char elements, at compile time. The sizeof operator is useless because if you allocate an array of N elements of type T, the size allocated will already be sizeof(T)*N, you don't need to do the math. Stack allocated and no free needed. In general, you use char name[10] when you already know the size of the object you need (the length of the string in this case).
char *name = malloc(sizeof(char)*10);
Allocates 10 bytes of memory in the heap. Allocation is done at run time, you need to free the result.
char name[sizeof(char)*10];
The first one is allocated on the stack, once it goes out of scope memory gets automatically freed. You can't change the size of the first one.
char *name = malloc(sizeof(char)*10);
The second one is allocated on the heap and should be freed with free. It will stick around otherwise for the lifetime of your application. You can reallocate memory for the second one if you need.
The storage duration is different:
An array created with char name[size] exists for the entire duration of program execution (if it is defined at file scope or with static) or for the execution of the block it is defined in (otherwise). These are called static storage duration and automatic storage duration.
An array created with malloc(size) exists for just as long as you specify, from the time you call malloc until the time you call free. Thus, it can be made to use space only while you need it, unlike static storage duration (which may be too long) or automatic storage duration (which may be too short).
The amount of space available is different:
An array created with char name[size] inside a function uses the stack in typical C implementations, and the stack size is usually limited to a few megabytes (more if you make special provisions when building the program, typically less in kernel software and embedded systems).
An array created with malloc may use gigabytes of space in typical modern systems.
Support for dynamic sizes is different:
An array created with char name[size] with static storage duration must have a size specified at compile time. An array created with char name[size] with automatic storage duration may have a variable length if the C implementation supports it (this was mandatory in C 1999 but is optional in C 2011).
An array created with malloc may have a size computed at run-time.
malloc offers more flexibility:
Using char name[size] always creates an array with the given name, either when the program starts (static storage duration) or when execution reaches the block or definition (automatic).
malloc can be used at run-time to create any number of arrays (or other objects), by using arrays of pointers or linked lists or trees or other data structures to create a multitude of pointers to objects created with malloc. Thus, if your program needs a thousand separate objects, you can create an array of a thousand pointers and use a loop to allocate space for each of them. In contrast, it would be cumbersome to write a thousand char name[size] definitions.
First things first: do not write
char name[sizeof(char)*10];
You do not need the sizeof as part of the array declaration. Just write
char name[10];
This declares an array of 10 elements of type char. Just as
int values[10];
declares an array of 10 elements of type int. The compiler knows how much space to allocate based on the type and number of elements.
If you know you'll never need more than N elements, then yes, you can declare an array of that size and be done with it, but:
You run the risk of internal fragmentation; your maximum number of bytes may be N, but the average number of bytes you need may be much smaller than that. For example, let's say you want to store 1000 strings of max length 255, so you declare an array like
char strs[1000][256];
but it turns out that 900 of those strings are only 20 bytes long; you're wasting a couple of hundred kilobytes of space1. If you split the difference and stored 1000 pointers, then allocated only as much space as was necessary to store each string, then you'd wind up wasting a lot less memory:
char *strs[1000];
...
strs[i] = strdup("some string"); // strdup calls malloc under the hood
...
Stack space is also limited relative to heap space; you may not be able to declare arbitrarily large arrays (as auto variables, anway). A request like
long double huge[10000][10000][10000][10000];
will probably cause your code to crash at runtime, because the default stack size isn't large enough to accomodate it2.
And finally, most situations fall into one of three categories: you have 0 elements, you have exactly 1 element, or you have an unlimited number of elements. Allocating large enough arrays to cover "all possible scenarios" just doesn't work. Been there, done that, got the T-shirt in multiple sizes and colors.
1. Yes, we live in the future where we have gigabytes of address space available, so wasting a couple of hundred KB doesn't seem like a big deal. The point is still valid, you're wasting space that you don't have to.
2. You could declare very large arrays at file scope or with the static keyword; this will allocate the array in a different memory segment (neither stack nor heap). The problem is that you only have that single instance of the array; if your function is meant to be re-entrant, this won't work.

What happens to memory after '\0' in a C string?

Surprisingly simple/stupid/basic question, but I have no idea: Suppose I want to return the user of my function a C-string, whose length I do not know at the beginning of the function. I can place only an upper bound on the length at the outset, and, depending on processing, the size may shrink.
The question is, is there anything wrong with allocating enough heap space (the upper bound) and then terminating the string well short of that during processing? i.e. If I stick a '\0' into the middle of the allocated memory, does (a.) free() still work properly, and (b.) does the space after the '\0' become inconsequential? Once '\0' is added, does the memory just get returned, or is it sitting there hogging space until free() is called? Is it generally bad programming style to leave this hanging space there, in order to save some upfront programming time computing the necessary space before calling malloc?
To give this some context, let's say I want to remove consecutive duplicates, like this:
input "Hello oOOOo !!" --> output "Helo oOo !"
... and some code below showing how I'm pre-computing the size resulting from my operation, effectively performing processing twice to get the heap size right.
char* RemoveChains(const char* str)
{
if (str == NULL) {
return NULL;
}
if (strlen(str) == 0) {
char* outstr = (char*)malloc(1);
*outstr = '\0';
return outstr;
}
const char* original = str; // for reuse
char prev = *str++; // [prev][str][str+1]...
unsigned int outlen = 1; // first char auto-counted
// Determine length necessary by mimicking processing
while (*str) {
if (*str != prev) { // new char encountered
++outlen;
prev = *str; // restart chain
}
++str; // step pointer along input
}
// Declare new string to be perfect size
char* outstr = (char*)malloc(outlen + 1);
outstr[outlen] = '\0';
outstr[0] = original[0];
outlen = 1;
// Construct output
prev = *original++;
while (*original) {
if (*original != prev) {
outstr[outlen++] = *original;
prev = *original;
}
++original;
}
return outstr;
}
If I stick a '\0' into the middle of the allocated memory, does
(a.) free() still work properly, and
Yes.
(b.) does the space after the '\0' become inconsequential? Once '\0' is added, does the memory just get returned, or is it sitting there hogging space until free() is called?
Depends. Often, when you allocate large amounts of heap space, the system first allocates virtual address space - as you write to the pages some actual physical memory is assigned to back it (and that may later get swapped out to disk when your OS has virtual memory support). Famously, this distinction between wasteful allocation of virtual address space and actual physical/swap memory allows sparse arrays to be reasonably memory efficient on such OSs.
Now, the granularity of this virtual addressing and paging is in memory page sizes - that might be 4k, 8k, 16k...? Most OSs have a function you can call to find out the page size. So, if you're doing a lot of small allocations then rounding up to page sizes is wasteful, and if you have a limited address space relative to the amount of memory you really need to use then depending on virtual addressing in the way described above won't scale (for example, 4GB RAM with 32-bit addressing). On the other hand, if you have a 64-bit process running with say 32GB of RAM, and are doing relatively few such string allocations, you have an enormous amount of virtual address space to play with and the rounding up to page size won't amount to much.
But - note the difference between writing throughout the buffer then terminating it at some earlier point (in which case the once-written-to memory will have backing memory and could end up in swap) versus having a big buffer in which you only ever write to the first bit then terminate (in which case backing memory is only allocated for the used space rounded up to page size).
It's also worth pointing out that on many operating systems heap memory may not be returned to the Operating System until the process terminates: instead, the malloc/free library notifies the OS when it needs to grow the heap (e.g. using sbrk() on UNIX or VirtualAlloc() on Windows). In that sense, free() memory is free for your process to re-use, but not free for other processes to use. Some Operating Systems do optimise this - for example, using a distinct and independently releasble memory region for very large allocations.
Is it generally bad programming style to leave this hanging space there, in order to save some upfront programming time computing the necessary space before calling malloc?
Again, it depends on how many such allocations you're dealing with. If there are a great many relative to your virtual address space / RAM - you want to explicitly let the memory library know not all the originally requested memory is actually needed using realloc(), or you could even use strdup() to allocate a new block more tightly based on actual needs (then free() the original) - depending on your malloc/free library implementation that might work out better or worse, but very few applications would be significantly affected by any difference.
Sometimes your code may be in a library where you can't guess how many string instances the calling application will be managing - in such cases it's better to provide slower behaviour that never gets too bad... so lean towards shrinking the memory blocks to fit the string data (a set number of additional operations so doesn't affect big-O efficiency) rather than having an unknown proportion of the original string buffer wasted (in a pathological case - zero or one character used after arbitrarily large allocations). As a performance optimisation you might only bother returning memory if unusued space is >= the used space - tune to taste, or make it caller-configurable.
You comment on another answer:
So it comes down to judging whether the realloc will take longer, or the preprocessing size determination?
If performance is your top priority, then yes - you'd want to profile. If you're not CPU bound, then as a general rule take the "preprocessing" hit and do a right-sized allocation - there's just less fragmentation and mess. Countering that, if you have to write a special preprocessing mode for some function - that's an extra "surface" for errors and code to maintain. (This trade-off decision is commonly needed when implementing your own asprintf() from snprintf(), but there at least you can trust snprintf() to act as documented and don't personally have to maintain it).
Once '\0' is added, does the memory just get returned, or is it
sitting there hogging space until free() is called?
There's nothing magical about \0. You have to call realloc if you want to "shrink" the allocated memory. Otherwise the memory will just sit there until you call free.
If I stick a '\0' into the middle of the allocated memory, does (a.)
free() still work properly
Whatever you do in that memory free will always work properly if you pass it the exact same pointer returned by malloc. Of course if you write outside it all bets are off.
\0 is just one more character from malloc and free perspective, they don't care what data you put in the memory. So free will still work whether you add \0 in the middle or don't add \0 at all. The extra space allocated will still be there, it won't be returned back to the process as soon as you add \0 to the memory. I personally would prefer to allocate only the required amount of memory instead of allocating at some upper bound as that will just wasting the resource.
As soon as you get memory from heap by calling malloc(), the memory is yours to use. Inserting \0 is like inserting any other character. This memory will remain in your possession until you free it or until OS claims it back.
The \0is a pure convention to interpret character arrays as stings - it is independent of the memory management. I.e., if you want to get your money back, you should call realloc. The string does not care about memory (what is a source of many security problems).
malloc just allocates a chunk of memory .. Its upto you to use however you want and call free from the initial pointer position... Inserting '\0' in the middle has no consequence...
To be specific malloc doesnt know what type of memory you want (It returns onle a void pointer) ..
Let us assume you wish to allocate 10 bytes of memory starting 0x10 to 0x19 ..
char * ptr = (char *)malloc(sizeof(char) * 10);
Inserting a null at 5th position (0x14) does not free the memory 0x15 onwards...
However a free from 0x10 frees the entire chunk of 10 bytes..
free() will still work with a NUL byte in memory
the space will remain wasted until free() is called, or unless you subsequently shrink the allocation
Generally, memory is memory is memory. It doesn't care what you write into it. BUT it has a race, or if you prefer a flavor (malloc, new, VirtualAlloc, HeapAlloc, etc). This means that the party that allocates a piece of memory must also provide the means to deallocate it. If your API comes in a DLL, then it should provide a free function of some sort.
This of course puts a burden on the caller right?
So why not put the WHOLE burden on the caller?
The BEST way to deal with dynamically allocated memory is to NOT allocate it yourself. Have the caller allocate it and pass it on to you. He knows what flavor he allocated, and he is responsible to free it whenever he is done using it.
How does the caller know how much to allocate?
Like many Windows APIs have your function return the required amount of bytes when called e.g. with a NULL pointer, then do the job when provided with a non-NULL pointer (using IsBadWritePtr if it is suitable for your case to double-check accessibility).
This can also be much much more efficient. Memory allocations COST a lot. Too many memory allocations cause heap fragmentation and then the allocations cost even more. That's why in kernel mode we use the so called "look-aside lists". To minimize the number of memory allocations done, we reuse the blocks we have already allocated and "freed", using services that the NT Kernel provides to driver writers.
If you pass on the responsibility for memory allocation to your caller, then he might be passing you cheap memory from the stack (_alloca), or passing you the same memory over and over again without any additional allocations. You don't care of course, but you DO allow your caller to be in charge of optimal memory handling.
To elaborate on the use of the NULL terminator in C:
You cannot allocate a "C string" you can allocate a char array and store a string in it, but malloc and free just see it as an array of the requested length.
A C string is not a data type but a convention for using a char array where the null character '\0' is treated as the string terminator.
This is a way to pass strings around without having to pass a length value as a separate argument. Some other programming languages have explicit string types that store a length along with the character data to allow passing strings in a single parameter.
Functions that document their arguments as "C strings" are passed char arrays but have no way of knowing how big the array is without the null terminator so if it is not there things will go horribly wrong.
You will notice functions that expect char arrays that are not necessarily treated as strings will always require a buffer length parameter to be passed.
For example if you want to process char data where a zero byte is a valid value you can't use '\0' as a terminator character.
You could do what some of the MS Windows APIs do where you (the caller) pass a pointer and the size of the memory you allocated. If the size isn't enough, you're told how many bytes to allocate. If it was enough, the memory is used and the result is the number of bytes used.
Thus the decision about how to efficiently use memory is left to the caller. They can allocate a fixed 255 bytes (common when working with paths in Windows) and use the result from the function call to know whether more bytes are needed (not the case with paths due to MAX_PATH being 255 without bypassing Win32 API) or whether most of the bytes can be ignored...
The caller could also pass zero as the memory size and be told exactly how much needs to be allocated - not as efficient processing-wise, but could be more efficient space-wise.
You can certainly preallocate to an upperbound, and use all or something less.
Just make sure you actually use all or something less.
Making two passes is also fine.
You asked the right questions about the tradeoffs.
How do you decide?
Use two passes, initially, because:
1. you'll know you aren't wasting memory.
2. you're going to profile to find out where
you need to optimize for speed anyway.
3. upperbounds are hard to get right before
you've written and tested and modified and
used and updated the code in response to new
requirements for a while.
4. simplest thing that could possibly work.
You might tighten up the code a little, too.
Shorter is usually better. And the more the
code takes advantage of known truths, the more
comfortable I am that it does what it says.
char* copyWithoutDuplicateChains(const char* str)
{
if (str == NULL) return NULL;
const char* s = str;
char prev = *s; // [prev][s+1]...
unsigned int outlen = 1; // first character counted
// Determine length necessary by mimicking processing
while (*s)
{ while (*++s == prev); // skip duplicates
++outlen; // new character encountered
prev = *s; // restart chain
}
// Construct output
char* outstr = (char*)malloc(outlen);
s = str;
*outstr++ = *s; // first character copied
while (*s)
{ while (*++s == prev); // skip duplicates
*outstr++ = *s; // copy new character
}
// done
return outstr;
}

How many chars can be in a char array?

#define HUGE_NUMBER ???
char string[HUGE_NUMBER];
do_something_with_the_string(string);
I was wondering what would be the maximum number that I could add to a char array without risking any potential memory problems, buffer overflows or the like. I wanted to get user input into it, and possibly the maximum possible.
See this response by Jack Klein (see original post):
The original C standard (ANSI 1989/ISO
1990) required that a compiler
successfully translate at least one
program containing at least one
example of a set of environmental
limits. One of those limits was being
able to create an object of at least
32,767 bytes.
This minimum limit was raised in the
1999 update to the C standard to be at
least 65,535 bytes.
No C implementation is required to
provide for objects greater than that
size, which means that they don't need
to allow for an array of ints greater
than (int)(65535 / sizeof(int)).
In very practical terms, on modern
computers, it is not possible to say
in advance how large an array can be
created. It can depend on things like
the amount of physical memory
installed in the computer, the amount
of virtual memory provided by the OS,
the number of other tasks, drivers,
and programs already running and how
much memory that are using. So your
program may be able to use more or
less memory running today than it
could use yesterday or it will be able
to use tomorrow.
Many platforms place their strictest
limits on automatic objects, that is
those defined inside of a function
without the use of the 'static'
keyword. On some platforms you can
create larger arrays if they are
static or by dynamic allocation.
Now, to provide a slightly more tailored answer, DO NOT DECLARE HUGE ARRAYS TO AVOID BUFFER OVERFLOWS. That's close to the worst practice one can think of in C. Rather, spend some time writing good code, and carefully make sure that no buffer overflow will occur. Also, if you do not know the size of your array in advance, look at malloc, it might come in handy :P
It depends on where char string[HUGE_NUMBER]; is placed.
Is it inside a function? Then the array will be on the stack, and if and how fast your OS can grow stacks depends on the OS. So here is the general rule: dont place huge arrays on the stack.
Is it ouside a function then it is global (process-memory), if the OS cannot allocate that much memory when it tries to load your program, your program will crash and your program will have no chance to notice that (so the following is better:)
Large arrays should be malloc'ed. With malloc, the OS will return a null-pointer if the malloc failed, depending on the OS and its paging-scheme and memory-mapping-scheme this will either fail when 1) there is no continuous region of free memory large enough for the array or 2) the OS cannot map enough regions of free physical memory to memory that appears to your process as continous memory.
So, with large arrays do this:
char* largeArray = malloc(HUGE_NUMBER);
if(!largeArray) { do error recovery and display msg to user }
Declaring arbitrarily huge arrays to avoid buffer overflows is bad practice. If you really don't know in advance how large a buffer needs to be, use malloc or realloc to dynamically allocate and extend the buffer as necessary, possibly using a smaller, fixed-sized buffer as an intermediary.
Example:
#define PAGE_SIZE 1024 // 1K buffer; you can make this larger or smaller
/**
* Read up to the next newline character from the specified stream.
* Dynamically allocate and extend a buffer as necessary to hold
* the line contents.
*
* The final size of the generated buffer is written to bufferSize.
*
* Returns NULL if the buffer cannot be allocated or if extending it
* fails.
*/
char *getNextLine(FILE *stream, size_t *bufferSize)
{
char input[PAGE_SIZE]; // allocate
int done = 0;
char *targetBuffer = NULL;
*bufferSize = 0;
while (!done)
{
if(fgets(input, sizeof input, stream) != NULL)
{
char *tmp;
char *newline = strchr(input, '\n');
if (newline != NULL)
{
done = 1;
*newline = 0;
}
tmp = realloc(targetBuffer, sizeof *tmp * (*bufferSize + strlen(input)));
if (tmp)
{
targetBuffer = tmp;
*bufferSize += strlen(input);
strcat(targetBuffer, input);
}
else
{
free(targetBuffer);
targetBuffer = NULL;
*bufferSize = 0;
fprintf(stderr, "Unable to allocate or extend input buffer\n");
}
}
}
If the array is going to be allocated on the stack, then you are limited by the stack size (typically 1MB on Windows, some of it will be used so you have even less). Otherwise I imagine the limit would be quite large.
However, making the array really big is not a solution to buffer overflow issues. Don't do it. Use functions that have a mechanism for limiting the amount of buffer they use to make sure you don't overstep your buffer, and make the size something more reasonable (1K for example).
You can use malloc() to get larger portions of memory than normally an array could handle.
Well, a buffer overflow wouldn't be caused by too large a value for HUGE_NUMBER so much as too small compared to what was written to it (write to index HUGE_NUMBER or higher, and you've overflown the buffer).
Aside from that it will depend upon the machine. There are certainly systems that could handle several millions in the heap, and a million or so on the stack (depending on other pressures), but there are also certainly some that couldn't handle more than a few hundred (small embedded devices would be an obvious example). While 65,535 is a standard-specified minimum, a really small device could specify that the standard was deliberately departed from for this reason.
In real terms, on a large machine, long before you actually run out of memory, you are needlessly putting pressure on the memory in a way that would affect performance. You would be better off dynamically sizing an array to an appropriate size.

C Memory Management

I've always heard that in C you have to really watch how you manage memory. And I'm still beginning to learn C, but thus far, I have not had to do any memory managing related activities at all.. I always imagined having to release variables and do all sorts of ugly things. But this doesn't seem to be the case.
Can someone show me (with code examples) an example of when you would have to do some "memory management" ?
There are two places where variables can be put in memory. When you create a variable like this:
int a;
char c;
char d[16];
The variables are created in the "stack". Stack variables are automatically freed when they go out of scope (that is, when the code can't reach them anymore). You might hear them called "automatic" variables, but that has fallen out of fashion.
Many beginner examples will use only stack variables.
The stack is nice because it's automatic, but it also has two drawbacks: (1) The compiler needs to know in advance how big the variables are, and (2) the stack space is somewhat limited. For example: in Windows, under default settings for the Microsoft linker, the stack is set to 1 MB, and not all of it is available for your variables.
If you don't know at compile time how big your array is, or if you need a big array or struct, you need "plan B".
Plan B is called the "heap". You can usually create variables as big as the Operating System will let you, but you have to do it yourself. Earlier postings showed you one way you can do it, although there are other ways:
int size;
// ...
// Set size to some value, based on information available at run-time. Then:
// ...
char *p = (char *)malloc(size);
(Note that variables in the heap are not manipulated directly, but via pointers)
Once you create a heap variable, the problem is that the compiler can't tell when you're done with it, so you lose the automatic releasing. That's where the "manual releasing" you were referring to comes in. Your code is now responsible to decide when the variable is not needed anymore, and release it so the memory can be taken for other purposes. For the case above, with:
free(p);
What makes this second option "nasty business" is that it's not always easy to know when the variable is not needed anymore. Forgetting to release a variable when you don't need it will cause your program to consume more memory that it needs to. This situation is called a "leak". The "leaked" memory cannot be used for anything until your program ends and the OS recovers all of its resources. Even nastier problems are possible if you release a heap variable by mistake before you are actually done with it.
In C and C++, you are responsible to clean up your heap variables like shown above. However, there are languages and environments such as Java and .NET languages like C# that use a different approach, where the heap gets cleaned up on its own. This second method, called "garbage collection", is much easier on the developer but you pay a penalty in overhead and performance. It's a balance.
(I have glossed over many details to give a simpler, but hopefully more leveled answer)
Here's an example. Suppose you have a strdup() function that duplicates a string:
char *strdup(char *src)
{
char * dest;
dest = malloc(strlen(src) + 1);
if (dest == NULL)
abort();
strcpy(dest, src);
return dest;
}
And you call it like this:
main()
{
char *s;
s = strdup("hello");
printf("%s\n", s);
s = strdup("world");
printf("%s\n", s);
}
You can see that the program works, but you have allocated memory (via malloc) without freeing it up. You have lost your pointer to the first memory block when you called strdup the second time.
This is no big deal for this small amount of memory, but consider the case:
for (i = 0; i < 1000000000; ++i) /* billion times */
s = strdup("hello world"); /* 11 bytes */
You have now used up 11 gig of memory (possibly more, depending on your memory manager) and if you have not crashed your process is probably running pretty slowly.
To fix, you need to call free() for everything that is obtained with malloc() after you finish using it:
s = strdup("hello");
free(s); /* now not leaking memory! */
s = strdup("world");
...
Hope this example helps!
You have to do "memory management" when you want to use memory on the heap rather than the stack. If you don't know how large to make an array until runtime, then you have to use the heap. For example, you might want to store something in a string, but don't know how large its contents will be until the program is run. In that case you'd write something like this:
char *string = malloc(stringlength); // stringlength is the number of bytes to allocate
// Do something with the string...
free(string); // Free the allocated memory
I think the most concise way to answer the question in to consider the role of the pointer in C. The pointer is a lightweight yet powerful mechanism that gives you immense freedom at the cost of immense capacity to shoot yourself in the foot.
In C the responsibility of ensuring your pointers point to memory you own is yours and yours alone. This requires an organized and disciplined approach, unless you forsake pointers, which makes it hard to write effective C.
The posted answers to date concentrate on automatic (stack) and heap variable allocations. Using stack allocation does make for automatically managed and convenient memory, but in some circumstances (large buffers, recursive algorithms) it can lead to the horrendous problem of stack overflow. Knowing exactly how much memory you can allocate on the stack is very dependent on the system. In some embedded scenarios a few dozen bytes might be your limit, in some desktop scenarios you can safely use megabytes.
Heap allocation is less inherent to the language. It is basically a set of library calls that grants you ownership of a block of memory of given size until you are ready to return ('free') it. It sounds simple, but is associated with untold programmer grief. The problems are simple (freeing the same memory twice, or not at all [memory leaks], not allocating enough memory [buffer overflow], etc) but difficult to avoid and debug. A hightly disciplined approach is absolutely mandatory in practive but of course the language doesn't actually mandate it.
I'd like to mention another type of memory allocation that's been ignored by other posts. It's possible to statically allocate variables by declaring them outside any function. I think in general this type of allocation gets a bad rap because it's used by global variables. However there's nothing that says the only way to use memory allocated this way is as an undisciplined global variable in a mess of spaghetti code. The static allocation method can be used simply to avoid some of the pitfalls of the heap and automatic allocation methods. Some C programmers are surprised to learn that large and sophisticated C embedded and games programs have been constructed with no use of heap allocation at all.
There are some great answers here about how to allocate and free memory, and in my opinion the more challenging side of using C is ensuring that the only memory you use is memory you've allocated - if this isn't done correctly what you end up with is the cousin of this site - a buffer overflow - and you may be overwriting memory that's being used by another application, with very unpredictable results.
An example:
int main() {
char* myString = (char*)malloc(5*sizeof(char));
myString = "abcd";
}
At this point you've allocated 5 bytes for myString and filled it with "abcd\0" (strings end in a null - \0).
If your string allocation was
myString = "abcde";
You would be assigning "abcde" in the 5 bytes you've had allocated to your program, and the trailing null character would be put at the end of this - a part of memory that hasn't been allocated for your use and could be free, but could equally be being used by another application - This is the critical part of memory management, where a mistake will have unpredictable (and sometimes unrepeatable) consequences.
A thing to remember is to always initialize your pointers to NULL, since an uninitialized pointer may contain a pseudorandom valid memory address which can make pointer errors go ahead silently. By enforcing a pointer to be initialized with NULL, you can always catch if you are using this pointer without initializing it. The reason is that operating systems "wire" the virtual address 0x00000000 to general protection exceptions to trap null pointer usage.
Also you might want to use dynamic memory allocation when you need to define a huge array, say int[10000]. You can't just put it in stack because then, hm... you'll get a stack overflow.
Another good example would be an implementation of a data structure, say linked list or binary tree. I don't have a sample code to paste here but you can google it easily.
(I'm writing because I feel the answers so far aren't quite on the mark.)
The reason you have to memory management worth mentioning is when you have a problem / solution that requires you to create complex structures. (If your programs crash if you allocate to much space on the stack at once, that's a bug.) Typically, the first data structure you'll need to learn is some kind of list. Here's a single linked one, off the top of my head:
typedef struct listelem { struct listelem *next; void *data;} listelem;
listelem * create(void * data)
{
listelem *p = calloc(1, sizeof(listelem));
if(p) p->data = data;
return p;
}
listelem * delete(listelem * p)
{
listelem next = p->next;
free(p);
return next;
}
void deleteall(listelem * p)
{
while(p) p = delete(p);
}
void foreach(listelem * p, void (*fun)(void *data) )
{
for( ; p != NULL; p = p->next) fun(p->data);
}
listelem * merge(listelem *p, listelem *q)
{
while(p != NULL && p->next != NULL) p = p->next;
if(p) {
p->next = q;
return p;
} else
return q;
}
Naturally, you'd like a few other functions, but basically, this is what you need memory management for. I should point out that there are a number tricks that are possible with "manual" memory management, e.g.,
Using the fact that malloc is guaranteed (by the language standard) to return a pointer divisible by 4,
allocating extra space for some sinister purpose of your own,
creating memory pools..
Get a good debugger... Good luck!
#Euro Micelli
One negative to add is that pointers to the stack are no longer valid when the function returns, so you cannot return a pointer to a stack variable from a function. This is a common error and a major reason why you can't get by with just stack variables. If your function needs to return a pointer, then you have to malloc and deal with memory management.
#Ted Percival:
...you don't need to cast malloc()'s return value.
You are correct, of course. I believe that has always been true, although I don't have a copy of K&R to check.
I don't like a lot of the implicit conversions in C, so I tend to use casts to make "magic" more visible. Sometimes it helps readability, sometimes it doesn't, and sometimes it causes a silent bug to be caught by the compiler. Still, I don't have a strong opinion about this, one way or another.
This is especially likely if your compiler understands C++-style comments.
Yeah... you caught me there. I spend a lot more time in C++ than C. Thanks for noticing that.
In C, you actually have two different choices. One, you can let the system manage the memory for you. Alternatively, you can do that by yourself. Generally, you would want to stick to the former as long as possible. However, auto-managed memory in C is extremely limited and you will need to manually manage the memory in many cases, such as:
a. You want the variable to outlive the functions, and you don't want to have global variable. ex:
struct pair{
int val;
struct pair *next;
}
struct pair* new_pair(int val){
struct pair* np = malloc(sizeof(struct pair));
np->val = val;
np->next = NULL;
return np;
}
b. you want to have dynamically allocated memory. Most common example is array without fixed length:
int *my_special_array;
my_special_array = malloc(sizeof(int) * number_of_element);
for(i=0; i
c. You want to do something REALLY dirty. For example, I would want a struct to represent many kind of data and I don't like union (union looks soooo messy):
struct data{
int data_type;
long data_in_mem;
};
struct animal{/*something*/};
struct person{/*some other thing*/};
struct animal* read_animal();
struct person* read_person();
/*In main*/
struct data sample;
sampe.data_type = input_type;
switch(input_type){
case DATA_PERSON:
sample.data_in_mem = read_person();
break;
case DATA_ANIMAL:
sample.data_in_mem = read_animal();
default:
printf("Oh hoh! I warn you, that again and I will seg fault your OS");
}
See, a long value is enough to hold ANYTHING. Just remember to free it, or you WILL regret. This is among my favorite tricks to have fun in C :D.
However, generally, you would want to stay away from your favorite tricks (T___T). You WILL break your OS, sooner or later, if you use them too often. As long as you don't use *alloc and free, it is safe to say that you are still virgin, and that the code still looks nice.
Sure. If you create an object that exists outside of the scope you use it in. Here is a contrived example (bear in mind my syntax will be off; my C is rusty, but this example will still illustrate the concept):
class MyClass
{
SomeOtherClass *myObject;
public MyClass()
{
//The object is created when the class is constructed
myObject = (SomeOtherClass*)malloc(sizeof(myObject));
}
public ~MyClass()
{
//The class is destructed
//If you don't free the object here, you leak memory
free(myObject);
}
public void SomeMemberFunction()
{
//Some use of the object
myObject->SomeOperation();
}
};
In this example, I'm using an object of type SomeOtherClass during the lifetime of MyClass. The SomeOtherClass object is used in several functions, so I've dynamically allocated the memory: the SomeOtherClass object is created when MyClass is created, used several times over the life of the object, and then freed once MyClass is freed.
Obviously if this were real code, there would be no reason (aside from possibly stack memory consumption) to create myObject in this way, but this type of object creation/destruction becomes useful when you have a lot of objects, and want to finely control when they are created and destroyed (so that your application doesn't suck up 1GB of RAM for its entire lifetime, for example), and in a Windowed environment, this is pretty much mandatory, as objects that you create (buttons, say), need to exist well outside of any particular function's (or even class') scope.

Resources