Hope my question makes sense:
Programming in C, can I create a hash table in a shared memory segment, so any process with proper permissions has access to the keys/values in it?
If so, how can I specify at hash table creation that I want it put in the SHM?
Is there any recommended hash table implementation that allows this?
Thanks a lot
Off the top of my head I don't know any libraries that do this.
I do know that if you write your own or modify an existing library, the tricky thing that you need to do is track down and fix any code that uses pointers and replace it with code that uses base + offset.
The base would be the pointer returned by the shared memory create/open functions and it will be different in each process. The offset replaces what would be a pointer. When you need to locate a value via offset, you cast base to char*, add the offset to that and then cast it to your_type*. Something like (bucket_t*)((char*)base + offset).
That's assuming your hash implementation needs pointers at all. Some don't if they only use single-value buckets and no value lists.
The other tricky thing is that you need to manage the memory yourself. Instead of calling malloc(), you make your own function, maybe name it shm_hash_alloc(). A quick start is to just keep a pointer and increment it when allocating memory and don't bother with freeing it. Later you can use an array of pointers(offsets) to free lists of various power of two sizes or if you know your object types, a list for each type. In each free block you store the pointer to the next free block and you know the size because of what list it's on. There are even fancier methods but you probably don't need them.
I uploaded a shared memory hash table library for linux to SF (libshmht), I developed it with the performance as main feature and read / write access homegeneous access time. I think it's usefull as cache and as IPC system.
Also implements read/write locks for sharing between many processes.
http://sourceforge.net/projects/libshmht/
Hash table is just a data structure. As long as the modules accessing the shared memory know how the structure is built, they can access it. It doesn't matter which implementation you use, as long as all the modules involved know how to read it.
Think of it as a newspaper. You create your own private memory segment - that's a town local paper. Then you want to share it with all the towns around - you can, as long as the people speak the same language and can read it. There's no special language for sharing, it just has to be the one everyone understands.
Same in your case - you can use any hash table implementation, as long as all the threads use the same.
Related
(Context: The system I am working on already maintains a form of garbage collection. I'm working on compaction.)
Most compaction algorithms follow a basic structure:
Find first object
Move object to beginning of heap
Find second object
Move second object to address right after first object
Rinse and repeat
This algorithm is followed in section 2.2 of this paper except using two pointers, denoted "from" and "to". Essentially the FROM pointer traverses the heap until it finds live objects. Then it moves said object to the TO pointer. Then TO is incremented accordingly.
The algorithm is simple, but I have yet to find much information on how these pointers determine what is a "live object". This article discusses the creation of a basic mark-and-sweep garbage collector that runs through the stack, recursively going to each reference and marking them as live. The article however requires a linked list of ALL objects ever allocated. However, this is because the author is more or less creating their own VM.
My question is, is there a way of traversing a heap in C and identifying whether the current object is a live object? Is there a similar linked list of all allocated objects already in C that I could use? Or will I require more overhead?
My question is, is there a way of traversing a heap in C and identifying whether the current object is a live object?
At a high level, the process is looking at all active pointers and determining whether or not each piece of allocated memory is accessible. (Please note that this is very complicated is C, including because a pointer could be stored in an int or other data types.) If the memory is accessible via a pointer, then it is "live" in your terms. If not, then garbage collectors would consider it safe to free that memory.
If you're asking whether or not C has a native function for determining whether or not some allocated memory can be reached, then the answer is no.
Is there a similar linked list of all allocated objects already in C that I could use? Or will I require more overhead?
Again, if you're looking for a linked list that C natively provides and you can access, then the answer is no. You'd need to implement these things.
Forgive me if you've already seen this, but there are garbage collectors that you can download if you want to see how others have done it.
TL;DR: It's impossible.
To make that work, you need to solve some non-trivial problems:
Be able to name the live objects of the heap. That means to find and follow recursively all pointers in global variables and on the stack.
Move the live objects downwards to create a compact heap
Adjust pointers in your program to reflect the new locations of the moved objects.
Regarding 1.: At runtime, the C language doesn't help you to identify where you have pointer-type global variables. And on the stack, you find a mixture of e.g. integers, function-call return addresses or data pointers. For both memory areas, you have to find a way to enumerate all potential pointer values.
To make things worse, a pointer can not only point to the beginning of your data structure, but also to some inside element. And this pointer also makes the whole object "live".
Regarding 2.: That's the easy part, using the algorithm you mentioned.
Regarding 3.: Now your objects live at new addresses, so your old pointer values are no longer correct (pointing to the old locations), and you have to adjust them. So once again, you have to follow all root references (like in 1.) and adjust all pointers that are affected by your moves. But as you can't tell for sure if e.g. 0x12345678 was meant as an numeric integer or as an (old-location) address, changing that to the new-location address might break some computation.
I am still at a conceptual stage of a project. Yet to start code implementation. A subtask is this :
2 Processes will request data from a commonly accessed DLL. This DLL would be storing this data in a buffer in memory. If I just instantiate a structure within the DLL and store data in it, then each process instance will have a seperate structure and the data won't be common. So I need to have a shared memory implementation. Now another requirement that I have is of fast lookup time within the data. I am not sure how an AVL tree can be stored within a shared memory space. Is there an implementation available on the internet for an AVL tree/Hashmap that can be stored in shared memory space ? Also, is this the right approach to the problem ? Or should I be using something else altogether ?
TIA!
Whether this is the right approach depends on various factors, such as how expensive the data is to produce, whether the processes need to communicate with each other concerning the data, and so on. The rest of this answer assumes that you really do need a lookup structure in shared memory.
You can use any data structure, provided that you can allocate storage for both your data and the data structure's internals in your shared memory space. This typically means that you won't be able to use malloc for it, since each process' heap usually remains private. You will need your own custom allocator.
Let's say you chose AVL trees. Here's a library that implements them: https://github.com/fbuihuu/libtree. It looks like in this library, the "internal" AVL node data is stored intrusively in your "objects." Intrusive means that you reserve fields to be used by the library when declaring your object struct. So, as long as you allocate space for your objects in shared memory, using your custom allocator, and also allocate space for the root tree struct there as well, the whole tree should be accessible to multiple processes. You just have to make sure that the shared memory itself is mapped to the same address range in each process.
If you used a non-intrusive AVL implementation, meaning that each node is represented by an internal struct which then points to a separate struct containing your data, the library or your implementation would have to allow you to specify the allocator for the internal struct somehow, so that you could make sure the space will be allocated in shared memory.
As for how to write the custom allocator, that really depends on your usage and the system. You need to consider if you will ever need to "resize" the shared memory region, whether the system allows you to do that, whether you will allocate only fixed-width blocks inside the region, or you need to support blocks with arbitrary length, whether it's acceptable to spread your data structures over multiple shared memory regions, how your processes can synchronize and communicate, and so on. If you go this route, you should ask a new question on the topic. Be sure to mention what system you are using (Windows?) and what your constraints are.
EDIT
Just to further discourage you from doing this unless it's necessary: if, for example, your data is expensive to produce but you don't care whether the processes build up their own independent lookup structures once the data is available to them, then you can, for example, have the DLL write the data to a simple ring buffer in shared memory, and the rest of the code take it from there. Building up two AVL trees isn't really a problem unless they are going to be very large.
Also, if you only care about concurrency, and it's not important for there to be two processes, you may be able to make them both threads of one process.
In the case of Windows, Microsoft's recommended functions return what can be different pointer values to shared memory for each process. This means that within the shared memory, offsets (from the start of shared memory) have to be used instead of pointers. For example in a linked list, there is a next offset instead of a next pointer. You may want to create macros to convert offsets to pointers, and pointers to offsets.
I'm currently experimenting with IPC via mmap on Unix.
So far, I'm able to map a sparse-file of 10MB into the RAM and access it reading and writing from two separated processes. Awesome :)
Now, I'm currently typecasting the address of the memory-segment returned by mmap to char*, so I can use it as a plain old cstring.
Now, my real question digs a bit further. I've quite a lot experience with higher levels of programming (ruby, java), but never did larger projects in C or ASM.
I want to use the mapped memory as an address-space for variable-allocation. I dont't wether this is possible or does make any sense at all. Im thinking of some kind of a hash-map-like data structure, that lives purely in the shared segment. This would allow some interesting experiments with IPC, even with other languages like ruby over FFI.
Now, a regular implementation of a hash would use pretty often something like malloc. But this woult allocate memory out of the shared space.
I hope, you understand my thoughts, although my english is not the best!
Thank you in advance
Jakob
By and large, you can treat the memory returned by mmap like memory returned by malloc. However, since the memory may be shared between multiple "unrelated" processes, with independent calls to mmap, the starting address for each may be different. Thus, any data structure you build inside the shared memory should not use direct pointers.
Instead of pointers, offsets from the initial map address should be used instead. The data structure would then compute the right pointer value by adding the offset to the starting address of the mmamp region.
The data structure would be built from the single call to mmap. If you need to grow the data structure, you have to extend the mmap region itself. The could be done with mremap or by manually munmap and mmap again after the backing file has been extended.
read about it here.
I need to implement a variation of such an interface, say we are given a large memory space to manage there should be getmem(size) and free(pointer to block) functions that has to make sure free(pointer to block) can actually free the memory if and only if all processes using that block are done using it.
What I was thinking about doing is to define a Collectable struct as pointer to block, size of it, and process using it count. then whenever a process using a Collectable struct instance for the first time it has to explicitly increment the count, and whenever the process free()'s it, the count is decremented.
The problem with this approach is that all processes must respond to that interface and make it explicitly work : whenever assigning collectable pointer to an instance the process must explicitly inc that counter, which does not satisfy me, I was thinking maybe there is a way to create a macro for this to happen implicitly in every assignment?
I'm seeking of ways to approach this problem for a while, so other approaches and ideas would be great...
EDIT : the above approach doesn't satisfy me not only because it doesn't look nice but mostly because I cant assume a running process's code would care for updating my count. I need a way to make sure its done without changing the process's code...
An early problem with reference counting is that it is relatively easy to count the initial reference by putting code in a custom malloc / free implementation, but it is quite a bit harder to determine if the initial recipient passes that address around to others.
Since C lacks the ability to override the assignment operator (to count the new reference), basically you are left with a limited number of options. The only one that can possibly override the assignment is macrodef, as it has the ability to rewrite the assignment into something that inlines the increment of the reference count value.
So you need to "expand" a macro that looks like
a = b;
into
if (b is a pointer) { // this might be optional, if lookupReference does this work
struct ref_record* ref_r = lookupReference(b);
if (ref_r) {
ref_r->count++;
} else {
// error
}
}
a = b;
The real trick will be in writing a macro that can identify the assignment, and insert the code cleanly without introducing other unwanted side-effects. Since macrodef is not a complete language, you might run into issues where the matching becomes impossible.
(jokes about seeing nails where you learn how to use a hammer have an interesting parallel here, except that when you only have a hammer, you had better learn how to make everything a nail).
Other options (perhaps more sane, perhaps not) is to keep track of all address values assigned by malloc, and then scan the program's stack and heap for matching addresses. If you match, you might have found a valid pointer, or you might have found a string with a luck encoding; however, if you don't match, you certainly can free the address; provided they aren't storing an address + offset calculated from the original address. (perhaps you can macrodef to detect such offsets, and add the offset as multiple addresses in the scan for the same block)
In the end, there isn't going to be a foolproof solution without building a referencing system, where you pass back references (pretend addresses); hiding the real addresses. The down side to such a solution is that you must use the library interface every time you want to deal with an address. This includes the "next" element in the array, etc. Not very C-like, but a pretty good approximation of what Java does with its references.
Semi-serious answer
#include "Python.h"
Python has a great reference counting memory manager. If I had to do this for real in production code, not homework, I'd consider embedding the python object system in my C program which would then make my C program scriptable in python too. See the Python C API documentation if you are interested!
Such a system in C requires some discipline on the part of the programmer but ...
You need to think in terms of ownership. All things that hold references are owners and must keep track of the objects to which it holds references, e.g. through lists. When a reference holding thing is destroyed it must loop its list of referred objects and decrement their reference counters and if zero destroy them in turn.
Functions are also owners and should keep track of referenced objects, e.g. by setting up a list at the start of the function and looping through it when returning.
So you need to determine in which situations objects should be transferred or shared with new owners and wrap the corresponding situations in macros/functions that add or remove owned objects to owning objects' lists of referenced objects (and adjust the reference counter accordingly).
Finally you need to deal with circular references somehow by checking for objects that are no longer reachable from objects/pointers on the stack. That could be done with some mark and sweep garbage collection mechanism.
I don't think you can do it automatically without overridable destructors/constructors.
You can look at HDF5 ref counting but those require explicit calls in C:
http://www.hdfgroup.org/HDF5/doc/RM/RM_H5I.html
I need to alloc a large multidimensional-array as char a[x][32][y], and x*32*y is about 6~12G. (x, y are detenmined at runtime.)
I think out a way that is to do char *a=malloc(x*32*y), and use *(a+32*y*i+y*j+k) for a[i][j][k].
However, this looks not so convient comparing to a[i][j][k].
Is there any better way ?
Added:
It is a[x][32][datlen], where datlen is detenmined at runtime and x is set considering the memory.
The whole data in the array will be new. And I have got mathines with 16 or 32GB memory to run it.
INCORRECT: You should still be able to use a[i][j][k] syntax when referencing dynamically allocated memory.
CORRECT: Use a macro to at least make the job easier
#define A(i,j,k) *(a+32*y*i+y*j+k)
A(1,2,3) would then do the right thing.
I doubt you'll find a system which will allocate you contiguous memory that large*. You're going to have to utilize a chunking strategy of some kind.
You need to ask, "What is your data access pattern?"
If it is some stride (be it 1D or 2D), use that to choose an appropriate allocation of memory for each chunk. Use a data structure to represent each stride (might just be a struct containing your character arrays).
Edit: I didn't notice your second "question" about accessing your newly found 12G contiguous chunk of memory using a[i][j][k] syntax. That isn't going to happen in any consumer grade C distribution I'm aware of.
(*) and 640k ought to be enough memory for anyone.
Since this is C you cannot wrap everything up into a handy C++ object.
But I would do something similar. Design a series of functions that allocate, manipulate and destroy this new data type of yours.
To read or write a piece of the data, call a function. Never touch the data directly. In fact, if you can use a void* handle to your data and not even put the real data types in an included header file, that's the best thing to do.
With this, you can define the functions as operating on one very large memory block, a set of large memory blocks or even an on-disk database of blocks.
Now that I wrote that, let me partly take it back. If you need more performance you might want to define all of the functions in the included header file as inline definitions. That will let your compiler remove almost all the function call overhead and optimize aggressively.
I admit that matrix_set(x, y, z, value) is not as pretty as matrix[x][y][z] = value, but it will work just as well.