Thread synchronization issues of APUE - c

I'm reading APUE and I am confused with thread synchronization of chapter 11. Below is a code snippet.
#define NHASH 29
#define HASH(fp) (((unsigned long)fp)%NHASH)
struct foo *fh[NHASH];
pthread_mutex_t hashlock = PTHREAD_MUTEX_INITIALIZER;
struct foo {
int f_count;
pthread_mutex_t f_lock;
struct foo *f_next; /* protected by hashlock */
int f_id;
/* ... more stuff here ... */
};
struct foo *
foo_alloc(void) /* allocate the object */
{
struct foo *fp;
int idx;
if ((fp = malloc(sizeof(struct foo))) != NULL) {
fp->f_count = 1;
if (pthread_mutex_init(&fp->f_lock, NULL) != 0) {
free(fp);
return(NULL);
}
idx = HASH(fp);
pthread_mutex_lock(&hashlock);
fp->f_next = fh[idx];
fh[idx] = fp;
pthread_mutex_lock(&fp->f_lock);
pthread_mutex_unlock(&hashlock);
/* ... continue initialization ... */
pthread_mutex_unlock(&fp->f_lock);
}
return(fp);
}
My doubts are:
Why place pthread_mutex_lock(&fp->f_lock) before the pthread_mutex_unlock(&hashlock)? Could I place it afterward instead?
Since fp is local variable, could pthread_mutex_lock(&fp->f_lock) and pthread_mutex_unlock(&fp->f_lock) be removed all together?

No, because the actions after the pthread_mutex_lock(&hashlock) expose the newly created structure to other threads by adding it to the the fh list. While the hashlock is held, no-one can access the variable; as soon as the hashlock is released, it becomes accessible to other threads via the hash, but locking the fp_>f_lock mutex prevents anyone messing with fp.
Not with the code as written. If the whole structure was initialized except for the hashing, then you could do without locking the fp->f_lock mutex; just before completing, you'd lock the hashlock, hook the newly allocated item into the hash tables, and then release the hashlock, and you would be safe. If you need any exclusive access after the structure is added into the hash table, you have to acquire its mutex. The way it is written, that's a non-waiting acquisition of the mutex; there is no other process that could have access to the variable.
if ((fp = malloc(sizeof(struct foo))) != NULL) {
fp->f_count = 1;
if (pthread_mutex_init(&fp->f_lock, NULL) != 0) {
free(fp);
return(NULL);
}
idx = HASH(fp);
/* ... complete initialization except for adding to hash table ... */
pthread_mutex_lock(&hashlock);
fp->f_next = fh[idx];
fh[idx] = fp;
pthread_mutex_unlock(&hashlock);
}
So, there is logic behind what is done; it is correct.

I guess that there's a second thread that's looping through the created objects, doing something with them. In this case:
No, because the loop thread might access the newly created object before being initialized at all.
No, because the loop thread might access the newly created object being half initialized.

Related

Queue of Threads in pthread C - web server repsonse pipelining

I have a working HTTP Apache-like web server implemented in C, and my problem is that I don't know how to initialize the queue (and therefore how to enqueue threads into it), mostly because I'm not sure how to check if there is a previous thread to join before proceeding with the current one.
The server can exploit pipeline requests to increase its response speed, using threads in a
more sophisticated way: the web server can generate a new thread for each request for a new
resource, and simultaneously prepare responses; however, since the resources must be returned
to the client in the same order in which the requests were received by the server (FIFO), it will
take a coordination phase between the various response threads.
This coordination phase is achieved by implementing a sort of "waiting room for the doctor"
in which each patient, when entering, asks who was the last to arrive, keeps track of it and
enters the doctor's office only when the person in front of him leaves. In this way, everyone has
a partial view of the queue (cares for only one person) but this partial view allows a correct
implementation of a FIFO queue.
Here is the description of what do I have to do:
Likewise, each new thread will have to store the identifier of the thread that handles the previous
request and wait for its termination using the system call pthread_join (). The first thread,
obviously, will not have to wait for anyone and the last thread will have to be waited by the main
thread that handles the requests on that connection before closing the connection itself and
returning to wait for new connection requests.
I am having trouble initializing properly the to_join data structure, mostly because I don't understand how to compute the index i of the thread to join.- how can I differenciate the first and last thread in an array of pointers?
Here is the code (I could only modify in between the TO BE DONE START and TO BE DONE END comments):
#include "incApache.h"
pthread_mutex_t accept_mutex = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_t mime_mutex = PTHREAD_MUTEX_INITIALIZER;
int client_sockets[MAX_CONNECTIONS]; /* for each connection, its socket FD */
int no_response_threads[MAX_CONNECTIONS]; /* for each connection, how many response threads */
pthread_t thread_ids[MAX_THREADS];
int connection_no[MAX_THREADS]; /* connection_no[i] >= 0 means that i-th thread belongs to connection connection_no[i] */
pthread_t *to_join[MAX_THREADS]; /* for each thread, the pointer to the previous (response) thread, if any */
int no_free_threads = MAX_THREADS - 2 * MAX_CONNECTIONS; /* each connection has one thread listening and one reserved for replies */
struct response_params thread_params[MAX_THREADS - MAX_CONNECTIONS]; /* params for the response threads (the first MAX_CONNECTIONS threads are waiting/parsing requests) */
pthread_mutex_t threads_mutex = PTHREAD_MUTEX_INITIALIZER; /* protects the access to thread-related data structures */
pthread_t thread_ids[MAX_CONNECTIONS];
int connection_no[MAX_CONNECTIONS];
void *client_connection_thread(void *vp) {
int client_fd;
struct sockaddr_storage client_addr;
socklen_t addr_size;
pthread_mutex_lock(&threads_mutex);
int connection_no = *((int *) vp);
/*** properly initialize the thread queue to_join ***/
/*** TO BE DONE 3.1 START ***/
//to_join[0] = thread_ids[new_thread_idx];
//pthread_t *first; Am I perhaps supposed to initialize the to_join data structure as a queue with two pointers
//pthread_t *last; indicating the first and last element? How can I do it on an array of pointers?
/*** TO BE DONE 3.1 END ***/
pthread_mutex_unlock(&threads_mutex);
#endif
for (;;) {
addr_size = sizeof(client_addr);
pthread_mutex_lock(&accept_mutex);
if ((client_fd = accept(listen_fd, (struct sockaddr *) &client_addr, &addr_size)) == -1)
fail_errno("Cannot accept client connection");
pthread_mutex_unlock(&accept_mutex);
client_sockets[connection_no] = client_fd;
char str[INET_ADDRSTRLEN];
struct sockaddr_in *ipv4 = (struct sockaddr_in *) &client_addr;
printf("Accepted connection from %s\n", inet_ntop(AF_INET, &(ipv4->sin_addr), str, INET_ADDRSTRLEN));
manage_http_requests(client_fd
, connection_no);
}
}
#pragma clang diagnostic pop
void send_resp_thread(int out_socket, int response_code, int cookie,
int is_http1_0, int connection_idx, int new_thread_idx,
char *filename, struct stat *stat_p)
{
struct response_params *params = thread_params + (new_thread_idx - MAX_CONNECTIONS);
debug(" ... send_resp_thread(): idx=%lu\n", (unsigned long)(params - thread_params));
params->code = response_code;
params->cookie = cookie;
params->is_http1_0 = is_http1_0;
params->filename = filename ? my_strdup(filename) : NULL;
params->p_stat = stat_p;
pthread_mutex_lock(&threads_mutex);
connection_no[new_thread_idx] = connection_idx;
debug(" ... send_resp_thread(): parameters set, conn_no=%d\n", connection_idx);
/*** enqueue the current thread in the "to_join" data structure ***/
/*** TO BE DONE 3.1 START ***/
//Again, should I use a standard enqueue implementation? But then how would I keep track of the last node ot arrive?
/*** TO BE DONE 3.1 END ***/
if (pthread_create(thread_ids + new_thread_idx, NULL, response_thread, connection_no + new_thread_idx))
fail_errno("Could not create response thread");
pthread_mutex_unlock(&threads_mutex);
debug(" ... send_resp_thread(): new thread created\n");
}
void *response_thread(void *vp)
{
size_t thread_no = ((int *) vp) - connection_no;
int connection_idx = *((int *) vp);
debug(" ... response_thread() thread_no=%lu, conn_no=%d\n", (unsigned long) thread_no, connection_idx);
const size_t i = thread_no - MAX_CONNECTIONS;
send_response(client_sockets[connection_idx],
thread_params[i].code,
thread_params[i].cookie,
thread_params[i].is_http1_0,
(int)thread_no,
thread_params[i].filename,
thread_params[i].p_stat);
debug(" ... response_thread() freeing filename and stat\n");
free(thread_params[i].filename);
free(thread_params[i].p_stat);
return NULL;
}
I am having trouble initializing properly the to_join data structure,
mostly because I don't understand how to compute the index i of the
thread to join.- how can I differenciate the first and last thread in
an array of pointers?
Assignment is different from initialization, and operating on one element is different from operating on the whole array. As far as I can determine, you're not actually to initialize to_join in that function (so the comment is misleading). Instead, you're only to assign an appropriate value to a single element.
That analysis follows from my interpretation of the names, scope, and documentation comments of the various global variables and from the name, signature, and initial lines of the function in question:
it appears that the various arrays hold data pertaining to multiple threads of multiple connections, as the role of one of the file-scope connection_no arrays is to associate threads with connections.
it appears that the function is meant to be the thread-start function for connection-associated threads.
no thread started at a time when any other connection-associated threads are running should do anything other than set data pertaining to itself, lest it clobber data on which other threads and connections rely.
Now, as for the actual question -- how do you determine which thread the new one should join? You can't. At least, not relying only on the template code presented in the question, unmodified.*
Hypothetically, if you could access the version of the connection_no array that associates threads with connections then you could use it to find the indexes of all threads associated with the current connection. You could then get their thread IDs from the corresponding thread_ids array (noting that there is another name collision here), and their join targets from the join_to array. The first thread for the connection is the one that does not join to another, and the last is the one that is not joined by any other. That analysis is not altogether straightforward, but there are no real tricks to it. Details are left as the exercise they are meant to be.
But even if the file-scope name collisions were resolved, you could not perform the above analysis because the file-scope connection_no array is shadowed by a local variable of the same name inside the whole area where you are permitted to insert code.*
Note also that you appear to need to choose a thread index for the new thread, which in general will not be 0. It looks like you need to scan the thread_ids or connection_no array to find an available index.
*Unless you cheat. I take the intent to be for you to insert code (only) into the body of the client_connection_thread function, but you could, in fact, split that function into two or more by inserting code into the designated area. If the second file-scope declarations of connection_no and thread_ids were assumed to be ignored or missing in practice, then splitting up the function could provide a workaround for the shadowing issue. For example:
/*** properly initialize the thread queue to_join ***/
/*** TO BE DONE 3.1 START ***/
return client_connection_thread_helper1(connection_no);
} // end of function
// parameter 'con' is the number of this thread's connection
void *client_connection_thread_helper1(int con) {
int my_index;
// ... Find an available thread index (TODO: what if there isn't one?) ...
thread_ids[my_index] = pthread_self();
connection_no[my_index] = con; // connection_no is not shadowed in this scope
pthread_t *last = NULL;
// ... Find the last (other) thread associated with connection 'con', if any ...
// You can determine the first, too, but that does not appear to be required.
to_join[my_index] = last;
return client_connection_thread_helper2(con);
}
// A second additional function is required for the remaining bits of
// client_connection_thread(), because they need the local connection_no
void *client_connection_thread_helper2(int connection_no) {
int client_fd;
struct sockaddr_storage client_addr;
socklen_t addr_size;
/*** TO BE DONE 3.1 END ***/
pthread_mutex_unlock(&threads_mutex);
I suppose it is possible that figuring out the need and implementation for such function-splitting was intended to be part of the exercise, but that would be a dirty trick, and overall it seems more likely that the exercise is just poorly formed.

Linux Kernel Where to lock and unlock semaphores?

Within the Linux Kernel (specifically for device drivers), how would I know what variables to lock and when they need locking? In particular, why does the locking in the following code only happen after dev has been set, even though dev points to a global variable scull_devices?
struct scull_qset {
void **data; /* pointer to an array of pointers which each point to a quantum buffer */
struct scull_qset *next;
};
struct scull_dev {
struct scull_qset *data; /* Pointer to first quantum set */
int quantum; /* the current quantum size */
int qset; /* the current array size */
unsigned long size; /* amount of data stored here */
unsigned int access_key; /* used by sculluid and scullpriv */
struct semaphore sem; /* mutual exclusion semaphore */
struct cdev cdev; /* Char device structure initialized in scull_init_module */
};
struct scull_dev *scull_devices; /* allocated dynamically in scull_init_module */
int scull_open(struct inode *inode, struct file *filp)
{
struct scull_dev *dev; /* device information */
dev = container_of(inode->i_cdev, struct scull_dev, cdev);
filp->private_data = dev; /* for other methods */
/* now trim to 0 the length of the device if open was write-only */
if ( (filp->f_flags & O_ACCMODE) == O_WRONLY) {
if (down_interruptible(&dev->sem))
return -ERESTARTSYS;
scull_trim(dev); /* empty out the scull device */
up(&dev->sem);
}
return 0; /* success */
}
If the code for scull_init_module is needed for a more complete picture, here it is:
int scull_major = SCULL_MAJOR;
int scull_minor = 0;
int scull_quantum = SCULL_QUANTUM;
int scull_qset = SCULL_QSET;
int scull_nr_devs = SCULL_NR_DEVS;
int scull_init_module(void)
{
int result, i;
dev_t dev = 0;
/* assigns major and minor numbers (left out for brevity sake) */
/*
* allocate the devices -- we can't have them static, as the number
* can be specified at load time
*/
scull_devices = kmalloc(scull_nr_devs * sizeof(struct scull_dev), GFP_KERNEL);
if (!scull_devices) {
result = -ENOMEM;
goto fail;
}
memset(scull_devices, 0, scull_nr_devs * sizeof(struct scull_dev));
/* Initialize each device. */
for (i = 0; i < scull_nr_devs; i++) {
scull_devices[i].quantum = scull_quantum;
scull_devices[i].qset = scull_qset;
init_MUTEX(&scull_devices[i].sem);
scull_setup_cdev(&scull_devices[i], i);
}
/* some other stuff left out for brevity sake */
return 0; /* succeed */
fail: /* isn't this a little redundant? */
scull_cleanup_module();
return result;
}
/*
* Set up the char_dev structure for this device.
*/
static void scull_setup_cdev(struct scull_dev *dev, int index)
{
int err, devno = MKDEV(scull_major, scull_minor + index);
cdev_init(&dev->cdev, &scull_fops);
dev->cdev.owner = THIS_MODULE;
dev->cdev.ops = &scull_fops;
err = cdev_add (&dev->cdev, devno, 1);
/* Fail gracefully if need be */
if (err)
printk(KERN_NOTICE "Error %d adding scull%d", err, index);
}
Locking in the example has nothing to do with the global scull_devices variable, but the locking is used to protect attributes of one scull_dev.
E.g. assume there exists a read() operation which copies size bytes from data while the mentioned scroll_trim() operation frees data.
So, when process #1 calls open() and process #2 tries to read() from an already opened device at the same time, the read() operation can access freed data and oopses.
That is why you need to protect data against races. Semaphores are one way; mutexes another one which is often more appropriate. Spinlocks and atomic variables might work too.
lock - it is way to protect critical section
critical section - in your driver code, if multiple instances are accessing same area, that is critical section.
multiple instances - it could be thread, regular ioctl cmd(from user space), and softirq and irq. It depends on your driver implementation.
Based on "context", you should use different lock too.
thread context which can sleep -> semaphore/mutex
non-sleeping context -> spinlock
softirq, tasklet -> spin_lock_bh
irq -> spin_lock_irq, spin_lock_irqsave
It is completely based on your requirements.
Let's take an example. If you are working on network driver, your netdev has stats and packet buffer and those needs to be protected by lock since it can be updated by multiple instances like net_rx_softirq, net_tx_softirq, ioctl/netlink from userspace request an so on.
In this case, based on your resource's context, you need to use different lock/mutex and sometimes you need more than 1 lock.

shm_open: Differences between Mac and Linux

I have a queue in shared memory. It does work on Linux (kernel 4.3.4), but not on Mac OS X. Are there any differences between how Mac OS X handles shared memory and how linux does, which may explain this?
I get the shared memory via:
int sh_fd = shm_open(shmName, O_RDWR | O_CREAT,
S_IROTH | S_IWOTH // others hav read/write permission
| S_IRUSR | S_IWUSR // I have read/write permission
);
// bring the shared memory to the desired size
ftruncate(sh_fd, getpagesize());
The queue is very simple as well. Here is the basic struct:
typedef struct {
// this is to check whether the queue is initialized.
// on linux, this will be 0 initially
bool isInitialized;
// mutex to protect concurrent access
pthread_mutex_t access;
// condition for the reader, readers should wait here
pthread_cond_t reader;
// condition for the writer, writers should wait here
pthread_cond_t writer;
// whether the queue can still be used.
bool isOpen;
// maximum capacity of the queue.
int32_t capacity;
// current position of the reader and number of items.
int32_t readPos, items;
// entries in the queue. The array actually is longer, which means it uses the space behind the struct.
entry entries[1];
} shared_queue;
Basically everyone who wants access acquires the mutex, readPos indicates where the next value should be read (incrementing readPos afterwards), (readPos+items) % capacity is where new items go. The only somewhat fancy trick is the isInitialized byte. ftruncate fills the shared memory with zeros if it had length 0 before, so I rely on isInitiualized to be zero on a fresh shared memory page and write a 1 there as soon as I initialize the struct.
As I said, it works on Linux, so I don't think it is a simple implementation bug. Is there any subtle difference between shm_open on Mac vs. Linux which I may not be aware of? The bug I see looks like the reader tries to read from an empty queue, so, maybe the pthread mutex/condition does not work on shared memory in a Mac?
The problem is that PTHREAD_PROCESS_SHARED is not supported on mac.
http://alesteska.blogspot.de/2012/08/pthreadprocessshared-not-supported-on.html
You must set PTHREAD_PROCESS_SHARED on both the mutex and condition variables.
So for a mutex:
pthread_mutexattr_t mutex_attr;
pthread_mutex_t the_mutex;
pthread_mutexattr_init(&mutex_attr);
pthread_mutexattr_setpshared(&mutex_attr, PTHREAD_PROCESS_SHARED);
pthread_mutexattr(&the_mutex, &mutex_attr);
Basically the same steps for the condition variables, but replace mutexattr with condattr.
If the the pthread_*attr_setpshared functions don't exist or return an error, then it may not be supported on your platform.
To be on the safe side, you might want to set PTHREAD_MUTEX_ROBUST if supported. This will prevent deadlock over the mutex (though not guarantee queue consistency) if a process exits while holding the lock.
EDIT: As an added caution, having a boolean "is initialized" flag is an insufficient plan on its own. You need more than that to really guarantee only one process can initialize the structure. At the very least you need to do:
// O_EXCL means this fails if not the first one here
fd = shm_open(name, otherFlags | O_CREAT | O_EXCL );
if( fd != -1 )
{
// initialize here
// Notify everybody the mutex has been initialized.
}
else
{
fd = shm_open(name, otherFlags ); // NO O_CREAT
// magically somehow wait until queue is initialized.
}
Are you sure really need to roll your own queue? Will POSIX message queues (see mq_open man page) do the job? If not, what about one of many messaging middleware solutions out there?
Update 2016-Feb-10: Possible mkfifo based solution
One alternative to implementing your own queue in shared memory is to use an OS provided named FIFO using mkfifo. A key difference between a FIFO and a named pipe is that you are allowed to have multiple simultaneous readers and writers.
A "catch" to this, is that the reader sees end-of-file when the last writer exits, so if you want readers to go indefinitely, you may need to open a dummy write handle.
FIFOs are super easy to use on the command line, like so:
reader.sh
mkfifo my_queue
cat my_queue
write.sh
echo "hello world" > my_queue
Or slightly more effort in C:
reader.c
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <errno.h>
int main(int argc, char**argv)
{
FILE * fifo;
FILE * wfifo;
int res;
char buf[1024];
char * linePtr;
/* Try to create the queue. This may belong on reader or writer side
* depending on your setup. */
if( 0 != mkfifo("work_queue", S_IRUSR | S_IWUSR ) )
{
if( errno != EEXIST )
{
perror("mkfifo:");
return -1;
}
}
/* Get a read handle to the queue */
fifo = fopen("work_queue", "r");
/* Get a write handle to the queue */
wfifo = fopen("work_queue", "w");
if( !fifo )
{
perror("fopen: " );
return -1;
}
while(1)
{
/* pull a single message from the queue at a time */
linePtr = fgets(buf, sizeof(buf), fifo);
if( linePtr )
{
fprintf(stdout, "new command=%s\n", linePtr);
}
else
{
break;
}
}
return 0;
}
writer.c
#include <stdio.h>
#include <unistd.h>
int main(int argc, char**argv)
{
FILE * pipe = fopen("work_queue", "w");
unsigned int job = 0;
int my_pid = getpid();
while(1)
{
/* Write one 'entry' to the queue */
fprintf(pipe, "job %u from %d\n", ++job, my_pid);
}
}

Fail to allocate semaphore/netconn Tiva C TM4C1294

I am using tm4c1294+lwip1.4.1+FreeRTOS.
As netconn_alloc() is called for socket communication, it allocates an unused semaphore. the number of semaphore is defined as SYS_SEM_MAX, so it can not be over SYS_SEM_MAX.
However, as semaphores are allocated continuously it reaches SYS_SEM_MAX and stop working since I guess sys_sem_free() does not deallocate it properly
Here is function that creates a semaphore implemented in sys_arch.c
err_t
sys_sem_new(sys_sem_t *sem, u8_t count)
{
void *temp;
u32_t i;
/* Find a semaphore that is not in use. */
for(i = 0; i < SYS_SEM_MAX; i++) {
if(sems[i].queue == 0) {
break;
}
}
if(i == SYS_SEM_MAX) {
#if SYS_STATS
STATS_INC(sys.sem.err);
#endif /* SYS_STATS */
return ERR_MEM;
}
/* Create a single-entry queue to act as a semaphore. */
#if RTOS_FREERTOS
sem->queue = xQueueCreate(1, sizeof(void *));
if(sem->queue == NULL) {
#endif /* RTOS_FREERTOS */
#if SYS_STATS
STATS_INC(sys.sem.err);
#endif /* SYS_STATS */
return ERR_MEM;
}
/* Acquired the semaphore if necessary. */
if(count == 0) {
temp = 0;
xQueueSend(sem->queue, &temp, 0);
}
/* Update the semaphore statistics. */
#if SYS_STATS
STATS_INC(sys.sem.used);
#if LWIP_STATS
if(lwip_stats.sys.sem.max < lwip_stats.sys.sem.used) {
lwip_stats.sys.sem.max = lwip_stats.sys.sem.used;
}
#endif
#endif /* SYS_STATS */
/* Save the queue handle. */
sems[i].queue = sem->queue;
/* Return this semaphore. */
return (ERR_OK);
}
Here is another function that frees semaphore implemented in sys_arch.c
void
sys_sem_free(sys_sem_t *sem)
{
/* Delete Sem , By Jin */
vQueueDelete(sem->queue);
/* Clear the queue handle. */
sem->queue = 0;
/* Update the semaphore statistics. */
#if SYS_STATS
STATS_DEC(sys.sem.used);
#endif /* SYS_STATS */
}
Whenever netconn_free() is called sys_sem_free() deallocates the semaphore, but it does not free the semaphore assigned in sem[] array.
I added vQueueDelete(sem->queue); that was suggested by someone, but still all same.
Not only functions creates/frees semaphore but also functions handling mbox are same as functions above, so functions handling mbox could be wrong as well.
Someone already reported this issue to TI, but it seems they have not solved the problems yet.
Therefore, I may need to implement my own functions handling semaphore/mbox in sys_arch.c, but I don't have any clues so far.
Can anyone give me any ideas? or anything?
Thanks,
Jin
The sys_arch.txt file in /doc is somewhat helpful. Apparently, looking at that document, and at what lwip 1.3.2 used to do, it looks like the ports/tiva-tm4c129/sys_arch.c is incorrect and incomplete.
sys_sem_free() should indeed be doing the vQueueDelete() as you discovered. It should not be doing "sem->queue = 0". If you look at netconn_free() over in src/api/api_msg.c, you can see it calls sys_sem_free() and then sys_sem_set_invalid(). The queue handle will be needed in the second function, and should not be clobbered in the first.
sys_sem_set_invalid() should make a sweep over the sems[] array and if it finds a match to the sem->queue, it should zero out that copy in sems[]. Once that is done, it should set sem->queue to 0.
This, I think, best matches what's in Dunkel's sys_arch.txt document, and fixed the resource leak on my system.
I concur that the mailboxes are in the same shape. Fwiw, I went ahead and modified those in similar fashion to that just described for the sems.
btw, the lwip files I was working with were from the TivaWare_C_Series 2.1.0.12573 third_party folder.

Initializing shared memory safely

I have several processes communicating with each through POSIX shared memory on OS X.
My issue is these processes could spawn in any order, and try to initialize the shared memory segment at the same time.
I tried using advisory locks with fcntl and flock but both fail telling me I'm passing an invalid file descriptor (I'm positive the file descriptor is not invalid). So clearly that's out of the picture.
Are there any alternatives to this? Or is there any details about using locks with shared memory that I'm not aware of?
Edit:
My attempt at using locks looks like this:
// Some declarations...
struct Queue {
int index[ENTRIES_PER_QUEUE];
sem_t lock;
sem_t readWait;
sem_t writeSem;
struct Entry slots[ENTRIES_PER_QUEUE];
};
struct ipc_t {
int fd;
char name[512];
struct Queue* queue;
};
ipc_t ipc_create(const char* name, int owner) {
int isInited = 1;
struct Queue* queue;
struct flock lock = {
.l_type = F_WRLCK,
.l_whence = SEEK_SET,
.l_start = 0,
.l_len = 0
};
ipc_t conn = malloc(sizeof(struct ipc_t));
sprintf(conn->name, "/arqvenger_%s", name);
conn->fd = shm_open(conn->name, O_CREAT | O_RDWR, 0666);
if (conn->fd == -1) {
free(conn);
perror("shm_open failed");
return NULL;
}
if (fcntl(conn->fd, F_SETLKW, &lock) == -1) {
perror("Tanked...");
}
// Do stuff with the lock & release it
The output I get is:
Tanked...: Bad file descriptor
A common technique is to first call shm_open with O_CREAT|O_EXCL. This will succeed for only one process that then has to do the setup. The others then would have to do the open as before and wait a bit, probably polling, that the setup is finished.
Edit: To show how this could work as discussed in the comments.
struct head {
unsigned volatile flag;
pthread_mutex_t mut;
};
void * addr = 0;
/* try shm_open with exclusive, and then */
if (/* we create the segment */) {
addr = mmap(something);
struct head* h = addr;
pthread_mutex_init(&h->mut, aSharedAttr);
pthread_mutex_lock(&h->mut);
h->flag = 1;
/* do the rest of the initialization, and then */
pthread_mutex_unlock(&h->mut);
} else {
/* retry shm_open without exclusive, and then */
addr = mmap(something);
struct head* h = addr;
/* initialy flag is guaranteed to be 0 */
/* this will break out of the loop whence the new value is written to flag */
while (!h->flag) sched_yield();
pthread_mutex_lock(&h->mut);
pthread_mutex_unlock(&h->mut);
}
I have been able to reproduce the problem. Then I found a sad notice in the standard:
When the file descriptor fildes refers to a shared memory object, the
behavior of fcntl() shall be the same as for a regular file except the
effect of the following values for the argument cmd shall be
unspecified: F_SETFL, F_GETLK, F_SETLK, and F_SETLKW.
So it's likely it's not yet supported. For such a simple setup I would use a semaphore for signaling "the memory is ready".
EDIT
It seems I need to mention semaphores can be created using "O_EXCL" (so there are no races).

Resources