Just two simple questions I could not find a proper answer to. I decided I would learn assembly language since its one area of my programming capabilities that lacks.
Also where functions are called recursively, how does the OS determine how large the stack should be for a thread? or does it just place the stack where there is a large amount of memory it can expand into before colliding with the heap?
Is the paging file part of the heap?
Not really. You can't access it directly; the OS may use it transparently to page out parts of your process's virtual address space.
the heap doesn't have to be a continuous block of memory?
Yes, allocations with mmap(MAP_ANONYMOUS) are considered part of the heap, and can be anywhere in your virtual address space. As Weather Vane points out, "heap" is a pretty broad / archaic term. It's not a useful concept for understanding what really happens under the hood.
e.g. you can mmap a file to make its contents part of your virtual address space. Is that part of the "heap"? I don't know and don't care.
Also where functions are called recursively, how does the OS determine how large the stack should be for a thread?
It just sets a max stack size limit when your process starts. Using recursive functions has nothing to do with it. (Although it's the most common way to use up the fixed size limit, the other being large local arrays).
Is there any malloc/realloc/free like implementation where i can specify a memory region where to manage the memory allocation?
I mean regular malloc (etc.) functions manages only the heap memory region.
What if I need to allocate some space in a shared memory segment or in a memory mapped file?
Not 100 %, As per your question you want to maintain your own memory region. so you need to go for your own my_malloc, my_realloc and my_free
Implementing your own my_malloc may help you
void* my_malloc(int size)
{
char* ptr = malloc(size+sizeof(int));
memcpy(ptr, &size, sizeof(int));
return ptr+sizeof(int);
}
This is just a small idea, full implementation will take you to the
answer.
Refer this question
use the same method to achieve my_realloc and my_free
I asked myself this question recently too, because I wanted a malloc implementation for my security programs which could safely wipe out a static memory region just before exit (which contains sensitive data like encryption keys, passwords and other such data).
First, I found this. I thought it could be very good for my purpose, but I really could not understand it's code completely. The license status was also unclear, as it is very important for one of my projects too.
I ended up writing my own.
My own implementation supports multiple heaps at same time, operating over them with pool descriptor structure, automatic memory zeroing of freed blocks, undefined behavior and OOM handlers, getting exact usable size of allocated objects and testing that pointer is still allocated, which is very sufficient for me. It's not very fast and it is more educational grade rather than professional one, but I wanted one in a hurry.
Note that it does not (yet) knows about alignment requirements, but at least it returns an address suitable for storing an 32 bit integer.
Iam using Tasking and I can store data in a specific space of memory. For example I can use:
testVar _at(0x200000);
I'm not sure if this is what you are looking for, but for example I'am using it to store data to external RAM. But as far as I know, it's only workin for global variables.
It is not very hard to implement your own my_alloc and my_free and use preferred memory range. It is simple chain of: block size, flag free/in use, and block data plus final-block marker (e.g. block size = 0). In the beginning you have one large free block and know its address. Note that my_alloc returns the address of block data and block size/flag are few bytes before.
Please help me to verify whether I am following the right coding technique/practice in specific to "Shared Library"
I have to create and deliver a 'C' "shared library" with the below requirements
Collect and store the data from an external device via Ethernet, using own protocol
Process this data
Provide the user application with an API which accepts the pointers to copy the processed data to.
User application will know the size of data in prior, so it can allocate that much memory in advance and pass the pointer to the API. API copies the data to the pointer. This data can go up to 1 GB.
Since I haven't wrote any shared libraries before, My first question here is about allocating 1GB memory (using malloc) in the shared library, Is it a right practice?
Second question: In the shared library, Is it a right practice to declare this memory(1GB) pointer as global, since I have multiple functions in the library to access the same memory for processing purposes? (I am taking care of the race conditions with mutex)
Thanks in advance
Allocating 1G of memory with malloc shouldn't be an issue in modern systems having several GBs of memory. Even if you go out of physical memory, this still will not be a performance issue until you start using that memory. Malloc simply reserves the virtual memory space, but actual memory allocation happens on the first access.
Another plus for malloc is that it doesn't initialize the memory. This will also save you some time.
Regarding the global variable. It is all about the coding style. I'd not recommend to use global variables, since they are often source of mistakes, race-conditions, hidden pitfalls.
As a general practice, you are better off allocating large blocks of memory using operating system services directly and managing them yourself, rather than relying on whatever malloc () does behind the scenes.
Off the top a couple of reasons:
1) Malloc adds memory overhead.
2) Some malloc implementations do not handle huge chunks well.
3) Large allocations like this are more likely to fail. Using system services will usually give you more diagnostic information.
In regard to your second question, generally its not a good idea but might be in some circumstances.
You seem to imply that this is memory shared across processes in addition to being a shared library. In that case, you'd need to use an system service to create the memory block.
Hope my question makes sense:
Programming in C, can I create a hash table in a shared memory segment, so any process with proper permissions has access to the keys/values in it?
If so, how can I specify at hash table creation that I want it put in the SHM?
Is there any recommended hash table implementation that allows this?
Thanks a lot
Off the top of my head I don't know any libraries that do this.
I do know that if you write your own or modify an existing library, the tricky thing that you need to do is track down and fix any code that uses pointers and replace it with code that uses base + offset.
The base would be the pointer returned by the shared memory create/open functions and it will be different in each process. The offset replaces what would be a pointer. When you need to locate a value via offset, you cast base to char*, add the offset to that and then cast it to your_type*. Something like (bucket_t*)((char*)base + offset).
That's assuming your hash implementation needs pointers at all. Some don't if they only use single-value buckets and no value lists.
The other tricky thing is that you need to manage the memory yourself. Instead of calling malloc(), you make your own function, maybe name it shm_hash_alloc(). A quick start is to just keep a pointer and increment it when allocating memory and don't bother with freeing it. Later you can use an array of pointers(offsets) to free lists of various power of two sizes or if you know your object types, a list for each type. In each free block you store the pointer to the next free block and you know the size because of what list it's on. There are even fancier methods but you probably don't need them.
I uploaded a shared memory hash table library for linux to SF (libshmht), I developed it with the performance as main feature and read / write access homegeneous access time. I think it's usefull as cache and as IPC system.
Also implements read/write locks for sharing between many processes.
http://sourceforge.net/projects/libshmht/
Hash table is just a data structure. As long as the modules accessing the shared memory know how the structure is built, they can access it. It doesn't matter which implementation you use, as long as all the modules involved know how to read it.
Think of it as a newspaper. You create your own private memory segment - that's a town local paper. Then you want to share it with all the towns around - you can, as long as the people speak the same language and can read it. There's no special language for sharing, it just has to be the one everyone understands.
Same in your case - you can use any hash table implementation, as long as all the threads use the same.
I am writing C for an MPC 555 board and need to figure out how to allocate dynamic memory without using malloc.
Typically malloc() is implemented on Unix using sbrk() or mmap(). (If you use the latter, you want to use the MAP_ANON flag.)
If you're targetting Windows, VirtualAlloc may help. (More or less functionally equivalent to anonymous mmap().)
Update: Didn't realize you weren't running under a full OS, I somehow got the impression instead that this might be a homework assignment running on top of a Unix system or something...
If you are doing embedded work and you don't have a malloc(), I think you should find some memory range that it's OK for you to write on, and write your own malloc(). Or take someone else's.
Pretty much the standard one that everybody borrows from was written by Doug Lea at SUNY Oswego. For example glibc's malloc is based on this. See: malloc.c, malloc.h.
You might want to check out Ralph Hempel's Embedded Memory Manager.
If your runtime doesn't support malloc, you can find an open source malloc and tweak it to manage a chunk of memory yourself.
malloc() is an abstraction that is use to allow C programs to allocate memory without having to understand details about how memory is actually allocated from the operating system. If you can't use malloc, then you have no choice other than to use whatever facilities for memory allocation that are provided by your operating system.
If you have no operating system, then you must have full control over the layout of memory. At that point for simple systems the easiest solution is to just make everything static and/or global, for more complex systems, you will want to reserve some portion of memory for a heap allocator and then write (or borrow) some code that use that memory to implement malloc.
An answer really depends on why you might need to dynamically allocate memory. What is the system doing that it needs to allocate memory yet cannot use a static buffer? The answer to that question will guide your requirements in managing memory. From there, you can determine which data structure you want to use to manage your memory.
For example, a friend of mine wrote a thing like a video game, which rendered video in scan-lines to the screen. That team determined that memory would be allocated for each scan-line, but there was a specific limit to how many bytes that could be for any given scene. After rendering each scan-line, all the temporary objects allocated during that rendering were freed.
To avoid the possibility of memory leaks and for performance reasons (this was in the 90's and computers were slower then), they took the following approach: They pre-allocated a buffer which was large enough to satisfy all the allocations for a scan-line, according to the scene parameters which determined the maximum size needed. At the beginning of each scan-line, a global pointer was set to the beginning of the scan line. As each object was allocated from this buffer, the global pointer value was returned, and the pointer was advanced to the next machine-word-aligned position following the allocated amount of bytes. (This alignment padding was including in the original calculation of buffer size, and in the 90's was four bytes but should now be 16 bytes on some machinery.) At the end of each scan-line, the global pointer was reset to the beginning of the buffer.
In "debug" builds, there were two scan buffers, which were protected using virtual memory protection during alternating scan lines. This method detects stale pointers being used from one scan-line to the next.
The buffer of scan-line memory may be called a "pool" or "arena" depending on whome you ask. The relevant detail is that this is a very simple data structure which manages memory for a certain task. It is not a general memory manager (or, properly, "free store implementation") such as malloc, which might be what you are asking for.
Your application may require a different data structure to keep track of your free storage. What is your application?
You should explain why you can't use malloc(), as there might be different solutions for different reasons, and there are several reasons why it might be forbidden or unavailable on small/embedded systems:
concern over memory fragmentation. In this case a set of routines that allocate fixed size memory blocks for one or more pools of memory might be the solution.
the runtime doesn't provide a malloc() - I think most modern toolsets for embedded systems do provide some way to link in a malloc() implementation, but maybe you're using one that doesn't for whatever reason. In that case, using Doug Lea's public domain malloc might be a good choice, but it might be too large for your system (I'm not familiar with the MPC 555 off the top of my head). If that's the case, a very simple, custom malloc() facility might be in order. It's not too hard to write, but make sure you unit test the hell out of uit because it's also easy to get details wrong. For example, I have a set of very small routines that use a brain dead memory allocation strategy using blocks on a free list (the allocator can be compile-time configured for first, best or last fit). I give it an array of char at initialization, and subsequent allocation calls will split free blocks as necessary. It's nowhere near as sophisticated as Lea's malloc(), but it's pretty dang small so for simple uses it'll do the trick.
many embedded projects forbid the use of dynamic memory allocation - in this case, you have to live with statically allocated structures
Write your own. Since your allocator will probably be specialized to a few types of objects, I recommend the Quick Fit scheme developed by Bill Wulf and Charles Weinstock. (I have not been able to find a free copy of this paper, but many people have access to the ACM digital library.) The paper is short, easy to read, and well suited to your problem.
If you turn out to need a more general allocator, the best guide I have found on the topic of programming on machines with fixed memory is Donald Knuth's book The Art of Computer Programming, Volume 1. If you want examples, you can find good ones in Don's epic book-length treatment of the source code of TeX, TeX: The Program.
Finally, the undergraduate textbook by Bryant and O'Hallaron is rather expensive, but it goes through the implementation of malloc in excruciating detail.
Write your own. Preallocate a big chunk of static RAM, then write some functions to grab and release chunks of it. That's the spirit of what malloc() does, except that it asks the OS to allocate and deallocate memory pages dynamically.
There are a multitude of ways of keeping track of what is allocated and what is not (bitmaps, used/free linked lists, binary trees, etc.). You should be able to find many references with a few choice Google searches.
malloc() and its related functions are the only game in town. You can, of course, roll your own memory management system in whatever way you choose.
If there are issues allocating dynamic memory from the heap, you can try allocating memory from the stack using alloca(). The usual caveats apply:
The memory is gone when you return.
The amount of memory you can allocate is dependent on the maximum size of your stack.
You might be interested in: liballoc
It's a simple, easy-to-implement malloc/free/calloc/realloc replacement which works.
If you know beforehand or can figure out the available memory regions on your device, you can also use their libbmmm to manage these large memory blocks and provide a backing-store for liballoc. They are BSD licensed and free.
FreeRTOS contains 3 examples implementations of memory allocation (including malloc()) to achieve different optimizations and use cases appropriate for small embedded systems (AVR, ARM, etc). See the FreeRTOS manual for more information.
I don't see a port for the MPC555, but it shouldn't be difficult to adapt the code to your needs.
If the library supplied with your compiler does not provide malloc, then it probably has no concept of a heap.
A heap (at least in an OS-less system) is simply an area of memory reserved for dynamic memory allocation. You can reserve such an area simply by creating a suitably sized statically allocated array and then providing an interface to provide contiguous chunks of this array on demand and to manage chunks in use and returned to the heap.
A somewhat neater method is to have the linker allocate the heap from whatever memory remains after stack and static memory allocation. That way the heap is always automatically as large as it possibly can be, allowing you to use all available memory simply. This will require modification of the application's linker script. Linker scripts are specific to the particular toolchain, and invariable somewhat arcane.
K&R included a simple implementation of malloc for example.