Surprisingly simple/stupid/basic question, but I have no idea: Suppose I want to return the user of my function a C-string, whose length I do not know at the beginning of the function. I can place only an upper bound on the length at the outset, and, depending on processing, the size may shrink.
The question is, is there anything wrong with allocating enough heap space (the upper bound) and then terminating the string well short of that during processing? i.e. If I stick a '\0' into the middle of the allocated memory, does (a.) free() still work properly, and (b.) does the space after the '\0' become inconsequential? Once '\0' is added, does the memory just get returned, or is it sitting there hogging space until free() is called? Is it generally bad programming style to leave this hanging space there, in order to save some upfront programming time computing the necessary space before calling malloc?
To give this some context, let's say I want to remove consecutive duplicates, like this:
input "Hello oOOOo !!" --> output "Helo oOo !"
... and some code below showing how I'm pre-computing the size resulting from my operation, effectively performing processing twice to get the heap size right.
char* RemoveChains(const char* str)
{
if (str == NULL) {
return NULL;
}
if (strlen(str) == 0) {
char* outstr = (char*)malloc(1);
*outstr = '\0';
return outstr;
}
const char* original = str; // for reuse
char prev = *str++; // [prev][str][str+1]...
unsigned int outlen = 1; // first char auto-counted
// Determine length necessary by mimicking processing
while (*str) {
if (*str != prev) { // new char encountered
++outlen;
prev = *str; // restart chain
}
++str; // step pointer along input
}
// Declare new string to be perfect size
char* outstr = (char*)malloc(outlen + 1);
outstr[outlen] = '\0';
outstr[0] = original[0];
outlen = 1;
// Construct output
prev = *original++;
while (*original) {
if (*original != prev) {
outstr[outlen++] = *original;
prev = *original;
}
++original;
}
return outstr;
}
If I stick a '\0' into the middle of the allocated memory, does
(a.) free() still work properly, and
Yes.
(b.) does the space after the '\0' become inconsequential? Once '\0' is added, does the memory just get returned, or is it sitting there hogging space until free() is called?
Depends. Often, when you allocate large amounts of heap space, the system first allocates virtual address space - as you write to the pages some actual physical memory is assigned to back it (and that may later get swapped out to disk when your OS has virtual memory support). Famously, this distinction between wasteful allocation of virtual address space and actual physical/swap memory allows sparse arrays to be reasonably memory efficient on such OSs.
Now, the granularity of this virtual addressing and paging is in memory page sizes - that might be 4k, 8k, 16k...? Most OSs have a function you can call to find out the page size. So, if you're doing a lot of small allocations then rounding up to page sizes is wasteful, and if you have a limited address space relative to the amount of memory you really need to use then depending on virtual addressing in the way described above won't scale (for example, 4GB RAM with 32-bit addressing). On the other hand, if you have a 64-bit process running with say 32GB of RAM, and are doing relatively few such string allocations, you have an enormous amount of virtual address space to play with and the rounding up to page size won't amount to much.
But - note the difference between writing throughout the buffer then terminating it at some earlier point (in which case the once-written-to memory will have backing memory and could end up in swap) versus having a big buffer in which you only ever write to the first bit then terminate (in which case backing memory is only allocated for the used space rounded up to page size).
It's also worth pointing out that on many operating systems heap memory may not be returned to the Operating System until the process terminates: instead, the malloc/free library notifies the OS when it needs to grow the heap (e.g. using sbrk() on UNIX or VirtualAlloc() on Windows). In that sense, free() memory is free for your process to re-use, but not free for other processes to use. Some Operating Systems do optimise this - for example, using a distinct and independently releasble memory region for very large allocations.
Is it generally bad programming style to leave this hanging space there, in order to save some upfront programming time computing the necessary space before calling malloc?
Again, it depends on how many such allocations you're dealing with. If there are a great many relative to your virtual address space / RAM - you want to explicitly let the memory library know not all the originally requested memory is actually needed using realloc(), or you could even use strdup() to allocate a new block more tightly based on actual needs (then free() the original) - depending on your malloc/free library implementation that might work out better or worse, but very few applications would be significantly affected by any difference.
Sometimes your code may be in a library where you can't guess how many string instances the calling application will be managing - in such cases it's better to provide slower behaviour that never gets too bad... so lean towards shrinking the memory blocks to fit the string data (a set number of additional operations so doesn't affect big-O efficiency) rather than having an unknown proportion of the original string buffer wasted (in a pathological case - zero or one character used after arbitrarily large allocations). As a performance optimisation you might only bother returning memory if unusued space is >= the used space - tune to taste, or make it caller-configurable.
You comment on another answer:
So it comes down to judging whether the realloc will take longer, or the preprocessing size determination?
If performance is your top priority, then yes - you'd want to profile. If you're not CPU bound, then as a general rule take the "preprocessing" hit and do a right-sized allocation - there's just less fragmentation and mess. Countering that, if you have to write a special preprocessing mode for some function - that's an extra "surface" for errors and code to maintain. (This trade-off decision is commonly needed when implementing your own asprintf() from snprintf(), but there at least you can trust snprintf() to act as documented and don't personally have to maintain it).
Once '\0' is added, does the memory just get returned, or is it
sitting there hogging space until free() is called?
There's nothing magical about \0. You have to call realloc if you want to "shrink" the allocated memory. Otherwise the memory will just sit there until you call free.
If I stick a '\0' into the middle of the allocated memory, does (a.)
free() still work properly
Whatever you do in that memory free will always work properly if you pass it the exact same pointer returned by malloc. Of course if you write outside it all bets are off.
\0 is just one more character from malloc and free perspective, they don't care what data you put in the memory. So free will still work whether you add \0 in the middle or don't add \0 at all. The extra space allocated will still be there, it won't be returned back to the process as soon as you add \0 to the memory. I personally would prefer to allocate only the required amount of memory instead of allocating at some upper bound as that will just wasting the resource.
As soon as you get memory from heap by calling malloc(), the memory is yours to use. Inserting \0 is like inserting any other character. This memory will remain in your possession until you free it or until OS claims it back.
The \0is a pure convention to interpret character arrays as stings - it is independent of the memory management. I.e., if you want to get your money back, you should call realloc. The string does not care about memory (what is a source of many security problems).
malloc just allocates a chunk of memory .. Its upto you to use however you want and call free from the initial pointer position... Inserting '\0' in the middle has no consequence...
To be specific malloc doesnt know what type of memory you want (It returns onle a void pointer) ..
Let us assume you wish to allocate 10 bytes of memory starting 0x10 to 0x19 ..
char * ptr = (char *)malloc(sizeof(char) * 10);
Inserting a null at 5th position (0x14) does not free the memory 0x15 onwards...
However a free from 0x10 frees the entire chunk of 10 bytes..
free() will still work with a NUL byte in memory
the space will remain wasted until free() is called, or unless you subsequently shrink the allocation
Generally, memory is memory is memory. It doesn't care what you write into it. BUT it has a race, or if you prefer a flavor (malloc, new, VirtualAlloc, HeapAlloc, etc). This means that the party that allocates a piece of memory must also provide the means to deallocate it. If your API comes in a DLL, then it should provide a free function of some sort.
This of course puts a burden on the caller right?
So why not put the WHOLE burden on the caller?
The BEST way to deal with dynamically allocated memory is to NOT allocate it yourself. Have the caller allocate it and pass it on to you. He knows what flavor he allocated, and he is responsible to free it whenever he is done using it.
How does the caller know how much to allocate?
Like many Windows APIs have your function return the required amount of bytes when called e.g. with a NULL pointer, then do the job when provided with a non-NULL pointer (using IsBadWritePtr if it is suitable for your case to double-check accessibility).
This can also be much much more efficient. Memory allocations COST a lot. Too many memory allocations cause heap fragmentation and then the allocations cost even more. That's why in kernel mode we use the so called "look-aside lists". To minimize the number of memory allocations done, we reuse the blocks we have already allocated and "freed", using services that the NT Kernel provides to driver writers.
If you pass on the responsibility for memory allocation to your caller, then he might be passing you cheap memory from the stack (_alloca), or passing you the same memory over and over again without any additional allocations. You don't care of course, but you DO allow your caller to be in charge of optimal memory handling.
To elaborate on the use of the NULL terminator in C:
You cannot allocate a "C string" you can allocate a char array and store a string in it, but malloc and free just see it as an array of the requested length.
A C string is not a data type but a convention for using a char array where the null character '\0' is treated as the string terminator.
This is a way to pass strings around without having to pass a length value as a separate argument. Some other programming languages have explicit string types that store a length along with the character data to allow passing strings in a single parameter.
Functions that document their arguments as "C strings" are passed char arrays but have no way of knowing how big the array is without the null terminator so if it is not there things will go horribly wrong.
You will notice functions that expect char arrays that are not necessarily treated as strings will always require a buffer length parameter to be passed.
For example if you want to process char data where a zero byte is a valid value you can't use '\0' as a terminator character.
You could do what some of the MS Windows APIs do where you (the caller) pass a pointer and the size of the memory you allocated. If the size isn't enough, you're told how many bytes to allocate. If it was enough, the memory is used and the result is the number of bytes used.
Thus the decision about how to efficiently use memory is left to the caller. They can allocate a fixed 255 bytes (common when working with paths in Windows) and use the result from the function call to know whether more bytes are needed (not the case with paths due to MAX_PATH being 255 without bypassing Win32 API) or whether most of the bytes can be ignored...
The caller could also pass zero as the memory size and be told exactly how much needs to be allocated - not as efficient processing-wise, but could be more efficient space-wise.
You can certainly preallocate to an upperbound, and use all or something less.
Just make sure you actually use all or something less.
Making two passes is also fine.
You asked the right questions about the tradeoffs.
How do you decide?
Use two passes, initially, because:
1. you'll know you aren't wasting memory.
2. you're going to profile to find out where
you need to optimize for speed anyway.
3. upperbounds are hard to get right before
you've written and tested and modified and
used and updated the code in response to new
requirements for a while.
4. simplest thing that could possibly work.
You might tighten up the code a little, too.
Shorter is usually better. And the more the
code takes advantage of known truths, the more
comfortable I am that it does what it says.
char* copyWithoutDuplicateChains(const char* str)
{
if (str == NULL) return NULL;
const char* s = str;
char prev = *s; // [prev][s+1]...
unsigned int outlen = 1; // first character counted
// Determine length necessary by mimicking processing
while (*s)
{ while (*++s == prev); // skip duplicates
++outlen; // new character encountered
prev = *s; // restart chain
}
// Construct output
char* outstr = (char*)malloc(outlen);
s = str;
*outstr++ = *s; // first character copied
while (*s)
{ while (*++s == prev); // skip duplicates
*outstr++ = *s; // copy new character
}
// done
return outstr;
}
Related
I'm running into a problem creating an array big enough to store words from a large text document (think books).
Usually, I would just do:
char wordList[1000000][30];
But, as expected the program crashes as soon as it tries to initialize the array. So, I tried a few different things, such as:
char *wordList[30]
int k=0;
while(k<1000000){
wordList[k]= malloc(sizeof(char)*30);
k++;
}
This, too, didn't work. So I'm wondering if there is an easier way. I know its possible. For the first option, my research as lead my to believe the the variable is initialized on the stack (which has small memory) and segfaults.
I'm not sure why the second one fails. Any suggestions? I've searched everywhere I could to find an answer, but most of the suggestions are in java or c++ where you just call new, or arraylist etc.
wordList is an array of 30 char* pointers. You are accessing way beyond the limit of this array. Specifically, you are accessing up to million spaces, but the array only has 30 spaces. This will cause undefined behaviour.
You need to instead make sure wordList has enough space for 1000000 pointers.
This should be instead:
char *wordList[1000000];
This allows flexibility on the length of the words. The only fixed size here is the array size.
If you use:
wordList[k]= malloc(sizeof(char)*30);
Moreover, this will run into issues if the words are more than 29 characters, excluding the \0 character at the end. Although, their are not many words longer than 29 characters. Words as long as:
supercalifragilisticexpialidocious
Are hard to come by.
Furthermore, it depends on how you are reading these words from your text document. If you parse the words, and instead use a temporary buffer to store them, then you can do this:
wordList[k]= malloc(strlen(word)+1); /* +1 for null-terminator */
Which will allocate memory for any sized word you copy into wordList[k]. This will be more efficient for smaller words like "the" and "or", instead of allocating 30 spaces for any word.
Note: Allocating a million pointers on the heap beforehand is also very wasteful, this process should be done on an as needed basis. It might even be better to use char **wordList, to allow more flexibility with how many words you allocate.
For example, you could allocate a starting size:
size_t start_size = 1000;
char **wordList = malloc(start_size * sizeof(*wordlist));
Then if more words are found, you can realloc() more space as needed.
realloc() resizes block of memory it points to, and returns a pointer.
An example would be:
if (start_size == word_count) {
start_size *= 2;
wordList = realloc(wordList, start_size * sizeof(*wordList));
if (wordList == NULL) {
/* handle exit */
Which would return a pointer which holds start_size * 2 spaces.
You should also check the return of malloc() and realloc(), as they can return NULL if unsuccessful. At the end of your program, you should also free() the pointers allocated from malloc().
In practice, a good rule of thumb when programming in C is that "big" data should be allocated in heap memory.
(I am guessing that you are coding for an ordinary laptop or desktop machine running some common operating system; I'm actually thinking of a desktop Linux computer, like the machine I am answering here; but you could adapt my answer to a desktop running some Windows, to a tablet running some Android, to an Apple computer running some MacOSX)
Notice that your current wordList is 30 megabytes (that is sizeof(wordList)==30000000). That is big! But in some cases not big enough (probably not enough for the whole Saint James bible, and certainly not for the entire laws, decrees, and juridictions of US or of France, or for the archive of a century-old newspaper). You can easily find textual data of more than several dozens of megabytes today, such as all the messages on StackOverflow.
You may need to understand more about current operating systems to understand all of my answer; I recommend reading Operating Systems: Three Easy Pieces and, if your computer runs Linux and you want to code for Linux, Advanced Linux Programming.
You don't want to allocate a big array (or any kind of big data) as local variables (or with alloca(3) ...) on your call stack, because the call stack is limited (typically to one megabyte, or a few of them). On some special computers (think of expensive servers running some specially configured Linux) you could raise that (machine call stack) limit, but perhaps not easily, to several dozens of megabytes. Expecting a gigabyte call stack is not reasonable.
You probably don't want to have huge global or static data (those allocated at compile time in your data segment) of fixed size. If you did that, your program might still lack of memory (because you under-estimated that fixed size) or could not even start on smaller computers (e.g. if your data segment had 20Gbytes, your executable might start on my desktop with 32Gbytes, but would fail to start -at execve(2) time- on your laptop with only 16Gbytes).
The remaining option is usual practice: allocate all "big" data in heap memory by indirectly using primitives growing your virtual address space. In standard C, you'll extensively use malloc and friends (e.g. calloc) - with free to release memory. FWIW, the underlying primitives (to grow the virtual address space) on Linux include mmap(2) and related system calls (and may be called by the malloc implementation on your system). But the standard C dynamic memory allocation techniques (that is malloc & free) are hiding these gory (implementation specific) details in the C standard library (so your code using malloc & free could become portable with efforts from your part).
Read some coding rules, e.g. the GNU ones, related to memory usage, and robust programs, notably:
Avoid arbitrary limits on the length or number of any data structure, including file names, lines, files, and symbols, by allocating all data structures dynamically
(emphasis is mine)
Practically speaking, your wordList (that is a poor name, it is not a list but a vector or a table) should probably be a dynamically allocated array of pointers to dynamically allocated strings. You could declare it as char**wordList; and you want to keep its allocated size and used length (perhaps in two other global variables, size_t allocatedSize, usedLength; ...). You might prefer to use a struct ending with a flexible array member
Don't forget to check against failure of malloc. Perhaps you want to initialize your data with something like:
allocatedSize=1000;
usedLength=0;
wordList= calloc(allocatedSize, sizeof(char*));
if (!wordList) { perror("initial calloc wordlist"); exit(EXIT_FAILURE); };
Here is a routine to add a new word to your wordList; that routine does not check if the word is indeed new (maybe you want to use some other data structure, like some hash-table, or some self balancing binary search tree); if you want to keep only unique words, read some Introduction to Algorithms. Otherwise, you could use:
void add_new_word(const char*w) {
if (usedLength >= allocatedSize) {
size_t newsize = 4*usedLength/3+10;
(heuristically, we don't want to re-allocate wordList too often; hence the "geometrical" growth above)
char**newlist = calloc(newsize*sizeof(char*));
if (!newlist) { perror("calloc newlist"); exit(FAILURE); };
memcpy (newlist, wordList, usedLength*sizeof(char*));
free (wordList);
wordList = newlist;
allocatedSize = newsize;
};
// here we are sure that wordList is not full,
// so usedLength < allocatedSize
char *dw = strdup(w);
if (!dw) { perror("strdup failure"); exit(EXIT_FAILURE); };
we are using the very common strdup(3) function to copy some string into a heap allocated one. If your system don't have that, it is really easy to write, using strlen, malloc, strcpy ...
wordList[usedLength++] = dw;
} // end of add_new_word
After a call to add_new_word you know that a new word has been added at index usedLength-1 of wordList.
Notice that my add_new_word is checking against failure of malloc (including the one called from strdup) and satisfy the "robustness" criteria: all data is heap allocated!
BTW, some computers are (IMHO wrongly) enabling memory overcommitment. This is a system administration issue. I dislike that feature (because when it is enabled, malloc would never fail, but programs would crash badly when memory resources are exhausted).
FWIW, here is the routine to release memory, to be called near end of program (or registered thru atexit(3))
void destroy_word_list(void) {
if (!wordList) return;
for (size_t ix=0; ix<usedLength; ix++) free(wordList[ix]);
free (wordList);
usedLength = 0;
allocatedSize = 0;
wordList = NULL;
} // end of destroy_word_list
You may want to use valgrind e.g. to debug memory leaks.
I have I am doing this problem on SPOJ. http://www.spoj.com/problems/NHAY/. It requires taking input dynamically. In the code below even though I am not allocating memory to char *needle using malloc() - I am taking l = 1 - yet I am able to take input of any length and also it is printing out the entire string. Sometimes it gives runtime error. Why is this when I have not allocated enough memory for the string?
#include<stdio.h>
#include<malloc.h>
#include<ctype.h>
#include<stdlib.h>
int main()
{
long long l;
int i;
char *needle;
while(1){
scanf("%lld",&l);
needle =(char *)malloc(sizeof(char)*l);
scanf("%s",needle);
i=0;
while(needle[i]!='\0'){
printf("%c",needle[i]);
i++;
}
free(needle);
}
}
I also read on stackoverflow that a string is a char * so I should declare char *needle. How can I use this fact in the code? If I take l = 1 then no matter, what the length of the input string it should contain characters only up to the memory allocated for the char * pointer, i.e 1 byte. How can I do that?
Your code is producing an intentional buffer overflow by having sscanf copying a string bigger than the allocated space into the memory allocated by malloc. This "works" because in most cases, the buffer that is allocated is somewhere in the middle of a page so copying more data into the buffer "only" overwrites adjacent data. C (and C++) don't do any array bounds checking on plain C array and thus the error is uncaught.
In the cases where you end up with a runtime error, you most likely copied part of the string into unmapped and unallocated memory, which trigger an access violation.
Memory is usually allocated from the underlying OS in pages of a fixed size. For example, on x86 systems, pages are usually 4k in size. If the mapped address you are writing to is far enough away from the beginning and end of the page, the whole string will fit within the boundaries of the page. If you get close enough to the upper boundary, the code may attempt to write past the boundary, triggering the access violation.
[With some assumptions about the underlying system]
The reason it works for now is that the C library manages pools of memory allocated in pages from the operating system. The operating system only returns pages. The C library returns arbitrary amounts of data.
For your first allocation, you are getting read/write pages allocated by the operating system and managed by the pool. You are going off the edge of the data allocated by the library but are within the page returned by the operating system.
DOing what you are doing will corrupt the structure of the pool and a more extensive program using dynamic memory will eventually crash.
C language do not have default bound check. At the best it will crash while debugging, sometimes it will work as expected. Otherwise you will end up overwriting other memory blocks.
It will not always work. It is Undefined Behaviour.
I don't understand how dynamically allocated strings in C work. Below, I have an example where I think I have created a pointer to a string and allocated it 0 memory, but I'm still able to give it characters. I'm clearly doing something wrong, but what?
#include <stdlib.h>
#include <stdio.h>
int main(int argc, char *argv[])
{
char *str = malloc(0);
int i;
str[i++] = 'a';
str[i++] = 'b';
str[i++] = '\0';
printf("%s\n", str);
return 0;
}
What you're doing is undefined behavior. It might appear to work now, but is not required to work, and may break if anything changes.
malloc normally returns a block of memory of the given size that you can use. In your case, it just so happens that there's valid memory outside of that block that you're touching. That memory is not supposed to be touched; malloc might use that memory for internal housekeeping, it might give that memory as the result of some malloc call, or something else entirely. Whatever it is, it isn't yours, and touching it produces undefined behavior.
Section 7.20.3 of the current C standard states in part:
"If the size of the space requested is zero, the behavior is
implementation defined: either a null pointer is returned, or the
behavior is as if the size were some nonzero value, except that the
returned pointer shall not be used to access an object."
This will be implementation defined. Either it could send a NULL pointer or as mentioned something that cannot be referenced
Your are overwriting non-allocated memory. This might looks like working. But you are in trouble when you call free where the heap function tries to gives the memory block back.
Each malloc() returned chunk of memory has a header and a trailer. These structures hold at least the size of the allocated memory. Sometimes yout have additional guards. You are overwriting this heap internal structures. That's the reason why free() will complain or crash.
So you have an undefined behavior.
By doing malloc(0) you are creating a NULL pointer or a unique pointer that can be passed to free. Nothing wrong with that line. The problem lies when you perform pointer arithmetic and assign values to memory you have not allocated. Hence:
str[i++] = 'a'; // Invalid (undefined).
str[i++] = 'b'; // Invalid (undefined).
str[i++] = '\0'; // Invalid (undefined).
printf("%s\n", str); // Valid, (undefined).
It's always good to do two things:
Do not malloc 0 bytes.
Check to ensure the block of memory you malloced is valid.
... to check to see if a block of memory requested from malloc is valid, do the following:
if ( str == NULL ) exit( EXIT_FAILURE );
... after your call to malloc.
Your malloc(0) is wrong. As other people have pointed out that may or may not end up allocating a bit of memory, but regardless of what malloc actually does with 0 you should in this trivial example allocate at least 3*sizeof(char) bytes of memory.
So here we have a right nuisance. Say you allocated 20 bytes for your string, and then filled it with 19 characters and a null, thus filling the memory. So far so good. However, consider the case where you then want to add more characters to the string; you can't just out them in place because you had allocated only 20 bytes and you had already used them. All you can do is allocate a whole new buffer (say, 40 bytes), copy the original 19 characters into it, then add the new characters on the end and then free the original 20 bytes. Sounds inefficient doesn't it. And it is inefficient, it's a whole lot of work to allocate memory, and sounds like an specially large amount of work compared to other languages (eg C++) where you just concatenate strings with nothing more than str1 + str2.
Except that underneath the hood those languages are having to do exactly the same thing of allocating more memory and copying existing data. If one cares about high performance C makes it clearer where you are spending time, whereas the likes of C++, Java, C# hide the costly operations from you behind convenient-to-use classes. Those classes can be quite clever (eg allocating more memory than strictly necessary just in case), but you do have to be on the ball if you're interested in extracting the very best performance from your hardware.
This sort of problem is what lies behind the difficulties that operations like Facebook and Twitter had in growing their services. Sooner or later those convenient but inefficient class methods add up to something unsustainable.
int main()
{
char *s1, *sTemp;
s1 = (char*)malloc(sizeof(char)*7);
*(s1 + 0) = 'a';
*(s1 + 1) = 'b';
*(s1 + 2) = 'c';
*(s1 + 3) = 'd';
*(s1 + 4) = 'e';
*(s1 + 5) = 'f';
*(s1 + 6) = '\0';
sTemp = (s1 + 3);
free(sTemp); // shud delete d onwards. But it doesn't !!
return 0;
}
Hello,
In the C above code sTemp should point to the 3rd cell beyond s1 ( occupied by 'd')
So on calling free(sTemp) i expect to have something deleted from this location onwards.
( I purposely mention 'something' as the motive of my experiment initially was to find out till which location the free() ing works )
However i recieve a SIGABRT at the free().
How does free() know that this is not the start of the chunk. and correspondingly can we free up memory only from start of chunks? [ are they only the free-able pointers that free() can accept?? ]
Looking forward to replies :)
From the man pages
free() frees the memory space pointed
to by ptr, which must have been
returned by a previous call to
malloc(), calloc() or realloc().
Otherwise, or if free(ptr) has already
been called before, undefined
behaviour occurs.
Source: http://linux.die.net/man/3/free
About the actual question "how does free know...":
free knows that it is not at the start of the chunk because it maintains metadata that you don't know about. If you think about it, that's necessarily so, because otherwise it could never know how much memory to free, given only an address.
It is not specified how exactly the runtime keeps book of allocation sizes, only that you must never pass any pointer to free that did not come from a function of the malloc family.
Usually this works by malloc allocating a little more memory than needed[1] and writing some metadata in memory preceding the address that is returned. free then just looks at e.g. ptr[-8] or whatever (implementation detail!) to read this metadata.
It can then optionally do some consistency checks, and can determine the size of the memory block (one trivial consistency check that is probably always done is checking proper alignment of the pointer).
Having mentioned that, please please please, don't even think about playing with this metadata.
[1] It often does that anyway to satisfy alignment requirements and because of allocator internals (most allocators manage memory in different "buckets" of fixed size), so if you allocate, say, 4 bytes, you nominally get 4 bytes, but the runtime really allocated 8 or 16 bytes most of the time (without you knowing).
You can only free() pointers that were actually malloced (or calloced, realloced). If you try to free only a portion of memory by passing in a different pointer (even one that is "part" of another pointer), the C runtime will not be pleased (as you can see.)
I think you cannot free the memory because free doesn't know how much memory to free.Your program has information about the address malloc() returned and not for every address in this space.So you can free(s1) but not free(s1+3).Also you can handle your pointers as an array in this example:
s1[0]='a';
You can only free() memory that was previously malloced, calloced, or realloced, as dlev and Daniel have said.
The reason for this is implementation-dependent, but involves the method of keeping track of allocated memory.
Efficient memory allocation is a difficult problem because different memory allocation algorithms work well depending upon how memory is allocated: a few small chunks, half of which are freed, then slightly larger blocks being grabbed, etc.
The objective is to keep the size of the memory block used by the program at a minimum (usually this chunk will be a continuous block of virtual memory), while keeping the usage of the space within that block extremely high (few gaps between used segments of memory).
Remember that these blocks can't be moved except when being realloced, so there's always going to be some wasted space.
To minimize the waste, metadata about (at least) the size of the block is stored just before it. The memory allocator can look through the used blocks when determining how to handle a new request. If you pick a random memory location, whether part of a previously-allocated region or no, this metadata will not be present and free will be unable to determine what should be freed.
You cannot free that pointer the way you are doing, check this question and its answer: Memory Clobbering Error
#define HUGE_NUMBER ???
char string[HUGE_NUMBER];
do_something_with_the_string(string);
I was wondering what would be the maximum number that I could add to a char array without risking any potential memory problems, buffer overflows or the like. I wanted to get user input into it, and possibly the maximum possible.
See this response by Jack Klein (see original post):
The original C standard (ANSI 1989/ISO
1990) required that a compiler
successfully translate at least one
program containing at least one
example of a set of environmental
limits. One of those limits was being
able to create an object of at least
32,767 bytes.
This minimum limit was raised in the
1999 update to the C standard to be at
least 65,535 bytes.
No C implementation is required to
provide for objects greater than that
size, which means that they don't need
to allow for an array of ints greater
than (int)(65535 / sizeof(int)).
In very practical terms, on modern
computers, it is not possible to say
in advance how large an array can be
created. It can depend on things like
the amount of physical memory
installed in the computer, the amount
of virtual memory provided by the OS,
the number of other tasks, drivers,
and programs already running and how
much memory that are using. So your
program may be able to use more or
less memory running today than it
could use yesterday or it will be able
to use tomorrow.
Many platforms place their strictest
limits on automatic objects, that is
those defined inside of a function
without the use of the 'static'
keyword. On some platforms you can
create larger arrays if they are
static or by dynamic allocation.
Now, to provide a slightly more tailored answer, DO NOT DECLARE HUGE ARRAYS TO AVOID BUFFER OVERFLOWS. That's close to the worst practice one can think of in C. Rather, spend some time writing good code, and carefully make sure that no buffer overflow will occur. Also, if you do not know the size of your array in advance, look at malloc, it might come in handy :P
It depends on where char string[HUGE_NUMBER]; is placed.
Is it inside a function? Then the array will be on the stack, and if and how fast your OS can grow stacks depends on the OS. So here is the general rule: dont place huge arrays on the stack.
Is it ouside a function then it is global (process-memory), if the OS cannot allocate that much memory when it tries to load your program, your program will crash and your program will have no chance to notice that (so the following is better:)
Large arrays should be malloc'ed. With malloc, the OS will return a null-pointer if the malloc failed, depending on the OS and its paging-scheme and memory-mapping-scheme this will either fail when 1) there is no continuous region of free memory large enough for the array or 2) the OS cannot map enough regions of free physical memory to memory that appears to your process as continous memory.
So, with large arrays do this:
char* largeArray = malloc(HUGE_NUMBER);
if(!largeArray) { do error recovery and display msg to user }
Declaring arbitrarily huge arrays to avoid buffer overflows is bad practice. If you really don't know in advance how large a buffer needs to be, use malloc or realloc to dynamically allocate and extend the buffer as necessary, possibly using a smaller, fixed-sized buffer as an intermediary.
Example:
#define PAGE_SIZE 1024 // 1K buffer; you can make this larger or smaller
/**
* Read up to the next newline character from the specified stream.
* Dynamically allocate and extend a buffer as necessary to hold
* the line contents.
*
* The final size of the generated buffer is written to bufferSize.
*
* Returns NULL if the buffer cannot be allocated or if extending it
* fails.
*/
char *getNextLine(FILE *stream, size_t *bufferSize)
{
char input[PAGE_SIZE]; // allocate
int done = 0;
char *targetBuffer = NULL;
*bufferSize = 0;
while (!done)
{
if(fgets(input, sizeof input, stream) != NULL)
{
char *tmp;
char *newline = strchr(input, '\n');
if (newline != NULL)
{
done = 1;
*newline = 0;
}
tmp = realloc(targetBuffer, sizeof *tmp * (*bufferSize + strlen(input)));
if (tmp)
{
targetBuffer = tmp;
*bufferSize += strlen(input);
strcat(targetBuffer, input);
}
else
{
free(targetBuffer);
targetBuffer = NULL;
*bufferSize = 0;
fprintf(stderr, "Unable to allocate or extend input buffer\n");
}
}
}
If the array is going to be allocated on the stack, then you are limited by the stack size (typically 1MB on Windows, some of it will be used so you have even less). Otherwise I imagine the limit would be quite large.
However, making the array really big is not a solution to buffer overflow issues. Don't do it. Use functions that have a mechanism for limiting the amount of buffer they use to make sure you don't overstep your buffer, and make the size something more reasonable (1K for example).
You can use malloc() to get larger portions of memory than normally an array could handle.
Well, a buffer overflow wouldn't be caused by too large a value for HUGE_NUMBER so much as too small compared to what was written to it (write to index HUGE_NUMBER or higher, and you've overflown the buffer).
Aside from that it will depend upon the machine. There are certainly systems that could handle several millions in the heap, and a million or so on the stack (depending on other pressures), but there are also certainly some that couldn't handle more than a few hundred (small embedded devices would be an obvious example). While 65,535 is a standard-specified minimum, a really small device could specify that the standard was deliberately departed from for this reason.
In real terms, on a large machine, long before you actually run out of memory, you are needlessly putting pressure on the memory in a way that would affect performance. You would be better off dynamically sizing an array to an appropriate size.