I have a server which is multiprocess and multithreaded. The child processes while handling request updates some statistics. This statistics data is a struct updated by all the child processes. Each of these child processes is again multithreaded. The number of child processes is dynamic(increases or decreases) based on the number of requests.
To synchronize the writes to this stat struct, I use mmap. Here is how the map is initialized.
fd = open(mapfile, O_CREAT|O_RDWR, 0666);
write(fd, dummy, MMAP_FILE_SIZE); //dummy is all zeros
void *addr = mmap(0, sizeof(stat_t), PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
close(fd);
if (addr == (void *)-1) {
mapped = 0;
}
else {
gStat = (stat_t*)addr;
}
// here gStat struct is initialized
Also in the code where I manipulate the stats, I use a lock to synchronize across multiple threads within the process.
Now, the issue I am having is that under heavy load, the writes don't seem to synchronize. The statistics are not updated correctly. Under normal load, the stats get incremented correctly.
According to my understanding, If MAP_SHARED is specified, write references change the underlying object and the mapping type is retained across fork(). What am I missing?
Related
How to create shared memory after fork or in child process?
I want to first make a global pointer in shared memory, then in child process create multiple node dynamically and add the node to this global pointer.
#include <string.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
struct shm_t {
int data;
struct shm_t *next;
};
void main() {
struct shm_t *shm = (struct shm_t *)mmap(NULL, sizeof(*shm), PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_SHARED, 0, 0);
shm->data = 1;
shm->next = NULL;
int pid = fork();
if (pid == 0) {
printf("Child \n");
// How to create a shared shm_t here, and assign to *shm->next?
} else {
printf("Parent \n");
}
}
In order for two processes to share memory, they have to have something in common.
Your first allocation is shared because you allocated it with MAP_SHARED, and the child process inherited it from the parent. Once the fork already finished, there is no way for that to happen automatically (just one example: which process does the child share this with? The parent is just a random process with no special significance, as far Linux is concerned).
The way to do this properly is be sharing a file descriptor. The number of pitfalls here, however, is staggering.
Let's start with the solution first, and then understand why it won't work:
// Before the fork
int shared_fd = open("/dev/shm/somefile", O_CREAT|O_TRUNC|O_EXCL|O_RDWR, 0600);
// Check for errors
unlink("/dev/shm/somefile");
// check for errors
// Now you can fork
If you now use mmap from this file, it is possible to share it between the processes (i.e. - one process' writes will be visible to the other process).
Like I said, the number of pitfalls here is huge:
The file is created empty, and you cannot mmap past the end of a file. In order for this to work, you will need to increase the file using ftruncate. Here is a problem, though. The calls to ftruncate will need to be synchronized, or you will have the lost update problem, where both processes think they are increasing the file to the same size together.
Even after you do that, you still cannot store pointers inside the memory. The reason is that the memory is shared, in the sense that one process' writes are immediately visible to the other. There is no guarantee, however, that they will be mapped to the same addresses.
I'm not sure what you want to do here (if you want to share a linked list between two processes, you have huge synchronization issues to handle here as well). If you can provide a-priori limit on how much memory you're going to need, just pre-allocate it with the original mmap.
In all likelihood, however, what you're trying to achieve here isn't possible in the way you're trying to achieve it (not in any sane way).
I am attempting to create a code that simulates child care center. In this center one adult can care for up to three children. This condition has to be fulfilled all the time. Adults and children are processes generated randomly and amount of children and adults is set in program arguments. Child can enter only if there is enough adults inside and adult can leave only if there is enough other adults to care for the children. If not, passive waiting should be implemented, until the condition allows child/adult to leave/enter.
#include <time.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <signal.h>
#include <string.h>
#include <semaphore.h>
#include <sys/mman.h>
#include <sys/ipc.h>
#include <sys/shm.h>
void load_init_values();
void handler(int, int, char*);
pid_t adults, children;
int adult_max_t, child_max_t, adc, chc, amt, cmt, shm_a_id, shm_c_id;
int *adults_inside, *children_inside;
sem_t *adults_sem, *children_sem, *entry;
int main(int argc, char *argv[])
{
srand(time(NULL));
setbuf(stdout,NULL);
adc=atoi(argv[1]);
chc=atoi(argv[2]);
adult_max_t=atoi(argv[3]);
child_max_t=atoi(argv[4]);
amt=atoi(argv[5]);
cmt=atoi(argv[6]);
int pid=0;
load_init_values();
adults = fork();
if (adults == 0)
{
for(int i=0; i<=adc-1; i++)
{
int adult_t = (random() % (adult_max_t + 1));
usleep(adult_t*1000);
adults = fork();
// Adult process is created here
if(adults == 0)
{
handler(getpid(), amt, "adult");
}
else
{
}
}
}
else
{
children = fork();
if (children == 0)
{
for(int i=0; i<=chc-1; i++)
{
int child_t = (random() % (child_max_t + 1));
usleep(child_t*1000);
children = fork();
// Child process is created here
if(children == 0)
{
handler(getpid(), cmt, "child");
break;
}
else
{
}
}
}
else
{
}
}
return 0;
}
void handler(int pid,int maxtime, char* type)
{
sem_wait(entry);
printf("%s %i%s\n",type,pid," attempting to enter...");
if(type == "child")
{
int child_leave_t = (random() % (maxtime + 1));
if((*adults_inside) != 0)
{
if(((*children_inside)+1)/(*adults_inside) <= 3)
{
(*children_inside)++;
printf("%s %i%s\n",type,pid," entered!");
usleep(child_leave_t*1000);
printf("%s %i%s\n",type,pid," left!");
(*children_inside)--;
}
else
{
printf("%s %i%s\n",type,pid," can not enter. Waiting...");
}
}
else
{
printf("%s %i%s\n",type,pid," can not enter. Waiting...");
}
}
else if(type == "adult")
{
(*adults_inside)++;
printf("%s %i%s\n",type,pid," entered.");
}
sem_post(entry);
}
void load_init_values()
{
adults_sem = mmap(NULL, sizeof(sem_t), PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_SHARED, 0, 0);
children_sem = mmap(NULL, sizeof(sem_t), PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_SHARED, 0, 0);
entry = mmap(NULL, sizeof(sem_t), PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_SHARED, 0, 0);
shm_a_id = shmget(IPC_PRIVATE, sizeof(int), IPC_CREAT | IPC_EXCL | 0666);
shm_c_id = shmget(IPC_PRIVATE, sizeof(int), IPC_CREAT | IPC_EXCL | 0666);
adults_inside = (int *) shmat(shm_a_id, NULL, 0);
children_inside = (int *) shmat(shm_c_id, NULL, 0);
sem_init(adults_sem,1,1);
sem_init(children_sem,1,1);
sem_init(entry,1,1);
}
This code only simulates generating of processes. There is one shared semaphore entry that allows only one process at the time to request entering. Shared memory variables adults_inside and children_inside keep track of the inner state.
My problem is basically located in handler function. After condition disallowing child to enter is triggered, I can not figure out how to implement passive wait. I was thinking about using pause() system call and store the waiting processes in queue, but is seem quite inefficient. What approach should I choose?
You will need to implement this in terms of some form of IPC. You mentioned using Linux, but I will assume POSIX-with-unnamed-semaphores (i.e. not OS X) since you aren't yet using anything Linux-specific. Others have mentioned this could be simpler if you used threads. But maybe you have some reason for using multiple processes, so I'll just assume that.
As specified, the code does not appear to allow adults to exit, which makes things a bit simpler. You already know how many children are allowed at any point in time, as that is directly proportional to the number of adults present at any given point in time.
To figure out how to solve the problem, let's consider how such a thing might be handled in real life. We can imagine that there is some kind of "gatekeeper" at the daycare. This "gatekeeper" is represented in the code by the sum of the state: the semaphore and the shared memory variables representing the number of adults and children present at any point in time. When no children are allowed to enter, the gatekeeper blocks entry and the children must form a line. I assume that the intent is that children are allowed to enter in a first-come-first-serve basis; this means you'll need to have some kind of FIFO to represent the queue. When a child leaves, the gatekeeper must be able to notify the first child in line that they are eligible to enter.
So this code is missing two things:
Some kind of FIFO representing the ordering of children waiting to enter
Some kind of notification mechanism that a child can wait for a notification on, and that the gatekeeper can trigger to "wake up" the child.
Now, the question is what data we store in this queue and how we do the notification. There are several options, but I'll discuss the two most obvious.
Making the child wait could be as simple as having the gatekeeper place the child PID at the tail of the FIFO and sending that PID SIGSTOP using kill(2). This may happen several times. Once a child leaves, the gatekeeper dequeues from the head of the FIFO and sends the pid SIGCONT.
As currently architected, the "gatekeeper" in your program is more of an abstract concept. A clearer implementation might implement a gatekeeper as a management process.
But since no such process exists, we need to imagine something like the child sees a "please wait" sign at the door and waits. The process representing the child can make itself wait by placing itself at the tail of the FIFO, and using the raise(3) library function, and sending itself SIGSTOP. Then, when any child leaves, it reads from the front of the FIFO and sends that pid SIGCONT using kill(2).
This implementation is relatively straightforward and the only additional resources required are to somehow represent the queue in shared memory.
An alternative approach would be to give each child its own file descriptor(s). This could be either a pipe(2), or a bidirectional file descriptor like a PF_LOCAL socket(2). Leaving the file descriptors in blocking mode, when a child is not allowed to enter, it puts (maybe the write-side of, if a pipe) its file descriptor at the tail of the FIFO, and blocks read(2)ing one byte from the read-side (which would be the same fd if not a pipe).
When a child exits, it pulls the entry from the front of the FIFO and write(2)s one byte to the file descriptor there. This will wake the child process that is blocked in read(2), and it will continue on its merry way into the daycare.
As previously stated, condition variables have also been suggested. I usually avoid them; they are easy to misuse, and you're already serializing the entry process.
In both the signal and the file descriptor case, a ring buffer of integers suffices -- so that's all the state you need to store in the FIFO.
The FIFO requires some careful consideration. Since multiple processes will be reading and manipulating it, it must also be in shared memory. Whether the FIFO is implemented as a ring buffer or some other way, you'll probably want to define some limits on the length of your queue. If there are too many children in line, maybe arriving children just "go home." Also, you'll need to make sure you handle the case of an empty FIFO gracefully on entry/exit, and make sure that transitioning from 1 waiter to 0 works as intended. Since you're serializing entry / exit with a semaphore, this should be straightforward.
2 semaphores precisely model the actual problem
While it is tempting to combine stats into synchronization, the minimum you need to synchronize for this child care is really only:
the number of vacancies for children are > 0 for a child to enter, otherwise they wait.
exiting adults take their 3 vacancies atomically with respect to each other or wait. (One adult refusing to take more responsibility on the way out is not explicit in your model, but prevents an exit starvation similar to writer starvation in rwlock implementations.)
But they must be mapped precisely
When semaphores hit 0 they force a wait, so to model this with the 2 semaphores you began to setup, their usage needs to match a few more specifics:
sem_init(adults_exiting_sem,1,1); /* Allow 1 adult to be decrementing */
sem_init(children_spots_sem,1,0); /* Allow no child without an adult */
Then the handler code can rely on the correct model of the semaphores to force waiting:
void handler(int pid,int maxtime, char* type)
{
int leave_t = (random() % (maxtime + 1));
if(type == "child")
{
printf("%s %i%s\n",type,pid," attempting to enter...");
sem_wait(children_spots_sem);
printf("%s %i%s\n",type,pid," entered!");
sleep(leave_t);
sem_post(children_spots_sem);
}
else if(type == "adult")
{
/* probably an inline function */
sem_post(children_spots_sem);
sem_post(children_spots_sem);
sem_post(children_spots_sem);
printf("%s %i%s\n",type,pid," entered.");
sleep(leave_t);
printf("%s %i%s\n",type,pid," attempting to leave...");
/* adult exit funnel */
sem_wait(adults_exiting_sem);
/* probably an inline function */
sem_wait(children_spots_sem);
sem_wait(children_spots_sem);
sem_wait(children_spots_sem);
sem_post(adults_exiting_sem);
}
printf("%s %i%s\n",type,pid," left!");
}
And there is always more to want
Naturally, you may want to further extend the model by:
use sem_timedwait to model parents giving up on dropping off their children.
reintroducing stats either with additional synchronization or just log for post analysis.
So I'm just wondering if I had a simple task to do in concurrency, how would I do this with multiple processes using fork() and exec() from a parent process, while locking some aspects of the parent process' memory (so that they don't overwrite each other), but making it available to those processes later?
I know I can do this with POSIX threads with their mutex locks, but what's the process equivalent to that? Is there a way to "lock" shared memory amongst threads? And then would I have to "wait()" for the other threads to finish those locked areas of memory before the other threads could access it?
If you're using the pthreads implementation of mutexes, you would still use them to synchronize between processes... you would place them in shared memory. Initializing a pthread mutex in shared memory addresses this.
You can also use a simple pipe to synchronize access -- pre-fill the pipe with a token and require a successful read of the token to permit resource access. Then write the token back into the pipe in order to release the resource.
First: if you call exec and it succeeds then your process image will be overwritten. You will loose any shared memory and you will need to set it up with your favourite shared memory paradigm (e.g. posix shared memory shm_open).
If you fork then any memory that was mapped shared will remain shared. Means you can place your favourite mutex (e.g. pthread_mutex_t, sem_t) into it and use it with the standard functions that go with it.
void * shared_memory = mmap(
NULL // anywhere
, sysconf(_SC_PAGESIZE) // mmap only works in chunks of pages
// typically 0x1000
, PROT_READ | PROT_WRITE // read-write
, MAP_SHARED // shared
| MAP_ANONYMOUS // anonymous, non-file backed
#ifdef MAP_HASSEMAPHORE
| MAP_HASSEMAPHORE // OS X requires this flag in case you
// intend to have semaphores in that segment
#endif
, -1 // no file backing
, 0
);
if (shared_memory == MAP_FAILED) {
perror("mmap");
abort();
}
// we use that memory to place a mutex tehre
pthread_mutex_t * mutex = shared_memory;
pthread_mutex_init(mutex, NULL);
pid_t pid = fork();
if (pid < 0) {
perror("fork");
abort();
}
if (!pid) {
// child goes here
// use the mutex here
} else {
// parent goes here
// use the mutex here
}
Update: I was not able to get this working and took a different approach. The problem was, as #nos pointed out, not with this code but elsewhere. ArrayList allocates memory in its implementation, and that is not part of the shared memory we've allocated. Thank you for the responses.
Original question:
I have a single fork. My goal is to have the child populate a data structure(db - it is an ArrayList - external implementation) and then be able to read that data structure from the parent. Currently, the result is a seg fault because the data structure is empty in the parent process.
Here is the code:
static ArrayList *db;
pid_t child_pid, pid;
int child_status;
//set up shared memory structure
db = mmap(NULL, 20000, PROT_READ | PROT_WRITE,
MAP_SHARED | MAP_ANONYMOUS, -1, 0);
child_pid = fork();
if(child_pid == 0) {
//Some operations to initialize and populate db happen here. This works fine.
//printAll(db, 1); //THIS would work. But I want to do this from the parent
exit(0);
} else {
//wait for child process to finish
do {
pid = wait(&child_status);
} while(pid != child_pid);
printAll(db, 1);//THIS IS WHERE IT SEGFAULTS, because db is not initialized - so it was never properly retrieved from mmap
munmap(db, 20000); //fixed this
}
munmap(db, 20000);
printAll(db, 1);//T//THIS IS WHERE IT SEGFAULTS,
That is not particularly surprising, you unmapped the memory on the line just above it.
I have a strange problem with using wait conditions and mutexes from pthreads on Linux to do synchronization between processes. Note that this is not between threads in one process only.
My use case is that there's a producer that creates resources (images in my case), saves them to a shared memory area, updates some information about the resource and then signals a waiting consumer. The shared memory and the meta data part works fine, so I'll leave that out, the problem is that the signalling does not work reliably. The use case is simple in that it doesn't matter if the consumer misses one image or two, the producer basically just overwrites one old image if the consumer hasn't had time to read it yet. So the wait condition only needs to handle waking up the consumer, I don't need there to be any resource counts or other data.
Both the producer and consumer have a struct like this:
struct EventData {
pthread_mutex_t mutexHandle;
pthread_cond_t conditionHandle;
};
A thread in the consumer process sits and waits for something to happen:
pthread_mutex_lock( &eventData->mutexHandle );
pthread_cond_wait( &eventData->conditionHandle, &eventData->mutexHandle );
thread_mutex_unlock( &eventData->mutexHandle );
The producing process does this when it has created an image, saved it into shared memory and is ready to let the consumer grab the image:
pthread_mutex_lock( &eventData->mutexHandle );
pthread_cond_signal( &eventData->conditionHandle );
// also tried:
//pthread_cond_broadcast( &eventData->conditionHandle );
pthread_mutex_unlock( &eventData->mutexHandle );
This looks quite ok to me and it works to some extent. The producer can signal the consumer about 100-1000 times without any issues, the consumer wakes up, grabs the image and shows it, the result being a video that I can see moving. At some point, usually around a few hundred frames, the consumer will freeze in pthread_cond_wait() and never return. The producer still happily creates images, calls pthread_cond_signal() and continues without problems. The consumer has not totally frozen, only the thread that does the pthread_cond_wait(), the rest of the application runs without issue.
So, something causes the signal to get lost when it moves from one thread to another thread in another process. It takes normally 5-20 seconds before the consumer freezes, and the amount of times the waking up works also varies between 100 and 1000 (based on values seen so far).
As mutexes and wait conditions are not trivial to share between processes by default, I've used this setup to create the primitives:
EventData * eventData;
int fd = open( tmpnam(NULL), O_RDWR | O_CREAT | O_EXCL, 0666);
if (fd < 0) {
// failed to open file for event
}
if ( ftruncate(fd, sizeof (eventData )) < 0 ) {
// failed to truncate file
}
// setup attributes to allow sharing between processes
pthread_condattr_init( &conditionAttribute );
pthread_condattr_setpshared( &conditionAttribute, PTHREAD_PROCESS_SHARED );
pthread_mutexattr_init( &mutexAttribute );
pthread_mutexattr_setpshared( &mutexAttribute, PTHREAD_PROCESS_SHARED );
// map memory for the event struct
eventData = (EventData *) mmap(NULL, sizeof(EventData), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
close (fd);
// finally initialize the memory
pthread_mutex_init( &eventData->mutexHandle, &mutexAttribute );
pthread_cond_init( &eventData->conditionHandle, &conditionAttribute );
The above is done by the party that creates the mutex and wait condition. The name of the file, i.e. tmpnam(NULL) is in reality saved and passed to the other process for opening:
int fd = open( nameOfEventFile, O_RDWR, 0666 );
if (fd < 0) {
// failed to open file for event
}
eventData = (EventData *) mmap( NULL, sizeof(EventData), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0 );
close( fd );
I see no errors here and would like some hint as to what could go wrong, especially as it works some random time.
And as soon as I had written 95% of the question the error poked me in the eye... I still decied to put it up here along with the fix in case someone else stumbles upon something similar. The part where the mutex and wait conditions are created looks like this:
EventData * eventData;
int fd = open( tmpnam(NULL), O_RDWR | O_CREAT | O_EXCL, 0666);
if (fd < 0) {
// failed to open file for event
}
if ( ftruncate(fd, sizeof (eventData )) < 0 ) {
// failed to truncate file
}
If you look carefully you see that the ftruncate() truncates to the size of the eventData pointer, not the size of struct EventData. So, the one character fix needed here is:
if ( ftruncate(fd, sizeof (EventData )) < 0 ) {
// failed to truncate file
}
Stupid bug, indeed.