I have to create for academic purpose an "On-Demand Server" that is based on the TCP protocol. The client each time sends a KEY and a VALUE that I have to store them in a global struct like the one bellow:
I use a function named "put" to store every KEY and VALUE to the struct and a "get" function that uses the key and finds the correct value:
My problem is that if a client makes a change in the struct the next one won't be able to know. I used semaphores in order to make the server wait until the child finishes and then accept the next child but I have still the same problem.
I read that I have to make the memory of this struct shared in order to make this work using mmaps but still I can't find a proper solution.
I make my first steps in server programming and every help would be great.
My problem is that if a client makes a change in the struct the next one won't be able to know.
As you noted, mmap() is part of the solution...
Shared memory needs of your application can be addressed with _shm_open()_, _shm_unlink()_, mmap etc. as illustrated in this post. And here for additional examples on shared memory
All of the essentials are summarized in the links, but this excerpt describes the basic concept:
shm_open() creates and opens a new, or opens an existing, POSIX
shared memory object. A POSIX shared memory object is in effect a
handle which can be used by unrelated processes to mmap(2) the same
region of shared memory. The shm_unlink() function performs the
converse operation, removing an object previously created by
shm_open().
I also wanted to suggest that dynamic allocation of memory may be of use in creating space for your struct on an as-needed basis. The size of members, as shown are excessively large,
char keys[4096];
char values[4096];
but in comments you state they only need to be 51 bytes. The struct then can be created as a typedef:
struct keyvalue {
char keys[51];
char values[51];
} DATA;
The create an instance of a pointer to your struct:
DATA *pData = {0};
Which can then be sized on an as needed basis using standard malloc, and or realloc
pData = malloc(initialSize*sizeof(DATA));
As size requirements change, use the following to grow memory:
DATA *tmp = {0};
tmp = realloc(pData, newSize);
if(!tmp) return -1;//ensure memory allocation successful
pData = tmp;//tmp includes previous contents of pData, and more space
... //continue using pData
Free pData when it is no longer needed.
free(pData);
To pass this struct in a function, the function might look like this:
void func1(DATA *d, int numElements)
{
...
for(i=0;i<numElements;i++)
{
strcpy(d[i]->keys, "some key");
strcpy(d[i]->values, "some value");
}
...
}
To call that function with a copy of pData, if say it had a 1000 elements:
func1(pData, 1000);
Related
I'm working with shared memory and I need several pointers for my program, but I have the restriction to use only one identifier:
int idSh;
int * mem;
char * mem2;
idSh = Shmget (IPC_PRIVATE, sizeof (char *) + sizeof (int *), IPC_CREAT | 0700);
mem = Shmat (idSh,0,0);
mem2 = Shmat (idSh,0,0);
I have to use several pointers with Shmat but I only have one segment of shared Memory with the identifier idSh.
Any solutions?
You need to rewrite your ‘pointer based’ data structures to be ‘offset based’, and make a sort of directory at the beginning of your shared memory object. The directory should identify the offsets to the root data structures, so each accessor can adapt to whichever address it is mapped at. The bad news is that it is a lot of fiddly work; the good news is it usually makes source bases better. Offset based has all sorts of advantages, such as being able to take a snapshot of the whole data structure, and write utilities that can analyze it off line.
Offset Based: if you had a node in a linked list like:
struct Node {
MyType Data;
struct Node *Link;
}
you would need to change it to something like:
struct Node {
MyType Data;
uintptr_t Link;
}
Then walking your data structure looks something like:
extern uintptr_t ShmemBase;
struct Node *Next = (struct Node *)(ShmemBase + Cur->Link);
although I bet you could do a much better job of encapsulating this, maybe even associated the “Link” field with the particular shared memory object.
I’ve done these transformations in the past, and they typically aren’t as bad as you might imagine. You do develop an appreciation for people who hide the data structure access methods properly....
Alternately, systems can often be coerced into mapping objects at a unified address for all participants; so if you hack around with your target system for a bit, you might find these magic addresses. Eventually, somebody will hate you if you go this route.
I'm creating a server and a client that are both accessing shared memory and changing the data in the shared memory.
I've used shm_open to open the shared memory, ftruncate to change the size of the shared memory, and mmap to map the shared memory object of type struct.
I want to create a function that accesses the shared memory via a file descriptor and returns a pointer to the struct.
For example:
XStruct * accessMem(int fd){ /*what to do here? */};
I'm not sure how to go about this, or what functions to use in order to access the shared memory. Any help would be great.
This is homework, so an explanation would be very helpful.
Thanks!
You will have to use (or implement a memory allocator to act on the block you received from mmap).
One simple implementation could be to use a bump allocator.
Store the value of the pointer you received from mmap. Every time you need to allocate memory for a struct, increment the pointer by sizeof (struct) and return the original pointer.
void *allocator_top; // Define a global pointer to top of the "heap" area.
.
.
.
allocator_top = mmap(...); // Do this wherever you perform the mmap
.
.
Xstruct * accessMem(){
void *temp = allocator_top;
// Need to check here if you have exceeded the amount of space mapped. If yes, you need to expand and add more pages.
allocator_top += sizeof(Xstruct);
return temp;
}
Edit : If you have multiple shared regions and want to allocate from them separately, you can take a void **allocator_top as an argument to accessMem() and pass the top of the region you want to allocate from.
This question already has answers here:
How does free know how much to free?
(11 answers)
Closed 9 years ago.
we allocate memory dynamically in C using malloc() and we receive a pointer to a location in the heap.
now we use free() to deallocate the memory, passing the same pointer value as its argumnet.
the Question now is how does free() know how much to deallocate.. considering the fact that we can always resize the memory block allocated by malloc().
is there anything related to Hash Tables here?
A typical implementation will store information just before the address returned by malloc. That information will include the information that realloc or free needs to know to do their work, but the details of what exactly is stored there depends on the implementation.
The original technique was to allocate a slightly larger block and store the size at the beginning, a part the application didn't see. The extra space holds a size and possibly links to thread the free blocks together for reuse.
There are certain issues with those tricks, however, such as poor cache and memory management behavior. Using memory right in the block tends to page things in unnecessarily and also create dirty pages which complicate sharing and copy-on-write.
So a more advanced technique is to keep a separate directory. Exotic approaches have also been developed where areas of memory use the same power-of-two sizes.
In general, the answer is: a separate data structure is allocated to keep state.
A simplist implementation is the one in the famous K&R C Bible,page 186 - 188.
The memory block we get actually is more (a struct head's or a union head's size) than we apply for.The struct may be like this:
typedef long Align;
union header
{
struct
{
union header* ptr; // next block
unsigned size; // size of this block , times of head size
}s;
Align x;
};
A figure to demonstrate it:
When we call the free function, the behaviour may be like this:
void free(void* ptr)
{
Header *bp, *p;
bp = (Header *)ptr - 1;
/* ..... */
/*return the memory to the linked list */
}
In visual studio, we have two models: release version and debug version,we could even use
the head to store debug message to make debug easier.The header in debug version is called _CrtMemBlockHeader, the definition is as below :
typedef struct _CrtMemBlockHeader
{
struct _CrtMemBlockHeader * pBlockHeaderNext;
struct _CrtMemBlockHeader * pBlockHeaderPrev;
char * szFileName;
int nLine;
size_t nDataSize;
int nBlockUse;
long lRequest;
unsigned char gap[nNoMansLandSize];
} _CrtMemBlockHeader;
Then the memory lalout is:
A memory manager uses tables to store additional data based on a pointer, sometimes right before the pointer, sometimes elsewhere. With C being very simple, the data is most likely pointer-2 or pointer-4, as int or long type. The correct details depend on the compiler.
When we use malloc ,a block will get reserve whose size will be littile more than what we have requested and in return to this malloc we get a pointer to start of this block.
AS i told you size of this block will be littile more than what exactly you needed.This extra space will be used to keep actual requested size of block,pointer to next free block and some data which checks "if you trying to access more than allocated block".
So whenever we call free using the pointer we want to deallocate, this free will search for the extra information given in the block space, Where it gets final size to deallocate.
This is my struct:
struct example {
int c=0;
char buf[10];
}
I have a struct of this type allocated in main()
...
struct example *ex=malloc(sizeof(*ex));
...
and passed to a thread by
...
pthread_create(&id[i], NULL, t_func, ex);
...
pthread_create is called into a child of main.
t_func is:
void *t_func(void* args) {
struct example *extmp = args;
....
How can I create a local struct example in every thread and copy into it the struct and the values of args? I need it such that if I modify a field of the local struct, that will not be seen in other threads. I've seen other similar questions, but I haven't found my answer...
In C you can assign objects of structure type:
struct example *extmp = args;
struct example local_copy = *extmp;
If you want a clean, unshared copy for each thread, the place to make the copy is before passing it to the thread. This avoids issues with needing to lock the data.
In your case, the main function should malloc another struct, then you can use memcpy or *new_data = *old_data to copy the data. Then you pass the pointer into the thread.
Now this is the important part. The main thread should never touch that copy again. The thread should call free when it is done using the data.
Yes, this means that you still have a pointer to a struct and not a struct on the thread stack. However, think about it: what does a struct on the stack get you? You have to type -> instead of . and that's about it.
Of course, my advice gets a lot more complicated and develops exceptions and curlicues if you are doing memory allocation with lock-free per-thread pools or NUMA. In those cases you probably do want to make the copy inside the thread. The main data will need to be locked and the thread will need to make its copy then unlock the data source.
In some exciting case you may need three copies: the primary copy (main memory), a temporary copy (main memory but no need to lock the primary) and the in-thread copy (now in per-thread memory, also no locks).
Ok I have a struct defined as such...
typedef struct
{
enum COMMAND command;
enum CMD_SOURCE source;
CHAR parameters[16];
} focuserCommand;
I am trying to make a function that will add an instance of this type of struct to an array of focuserCommands. The array is defined like this...
extern focuserCommand CommandBuffer[CMD_BUFFER_SIZE];
the function I am trying to write should take a pointer to a focuserCommand and add it to CommandBuffer. I am implementing CommandBuffer as a FIFO ring buffer so I know i need to move the tail forwared so that my other functions can see that the buffer contains data. CmdBuffHead and CmdBuffTail represent the read and write pointers of the buffer. Writes are added to the tail, reads are made from the head.
void AddCmdToBfr( focuserCommand * cmd )
{
// What goes here to add the struct pointed to by cmd to
// element CmdBuffTail of the buffer?
CmdBuffTail++;
}
Given that you have the variable CmdBuffTail pointing at the next element of CommandBuffer to be written to, then:
void AddCmdToBfr(focusCommand *cmd)
{
assert(CmdBuffTail >= 0 && CmdBuffTail < CMD_BUFFER_SIZE);
CommandBuffer[CmdBuffTail++] = *cmd;
if (CmdBuffTail >= CMD_BUFFER_SIZE)
CmdBuffTail = 0;
}
This preserves the invariant that you need - that the index is in the range 0..CMD_BUFFER_SIZE-1.
You may also need a separate index (CmdBuffHead perhaps) to identify where to read from. If so, you would probably need to alter that value in AddCmdToBfr() if CmdBuffTail if catches up with it.
After fixing a goof copying the incoming command, it occurs to me that you need to think carefully about the memory management. In this case, your focususerCommand structure is simple enough (unless the typedef CHAR hides a pointer - in which case, slap wrists; don't hide pointers) that you can simply copy it with impunity. If you can't copy the structure simply (because it contains pointers to allocated memory), then you need to ensure you understand who owns the data to make sure that (a) any allocated memory is released, and (b) any allocated memory is released just once. This would mean, for example, that instead of just copying the new command over the old, you would have to first release the memory allocated to the old entry. Think in terms of C++ copy constructors and destructors - in C.