I have several processes communicating with each through POSIX shared memory on OS X.
My issue is these processes could spawn in any order, and try to initialize the shared memory segment at the same time.
I tried using advisory locks with fcntl and flock but both fail telling me I'm passing an invalid file descriptor (I'm positive the file descriptor is not invalid). So clearly that's out of the picture.
Are there any alternatives to this? Or is there any details about using locks with shared memory that I'm not aware of?
Edit:
My attempt at using locks looks like this:
// Some declarations...
struct Queue {
int index[ENTRIES_PER_QUEUE];
sem_t lock;
sem_t readWait;
sem_t writeSem;
struct Entry slots[ENTRIES_PER_QUEUE];
};
struct ipc_t {
int fd;
char name[512];
struct Queue* queue;
};
ipc_t ipc_create(const char* name, int owner) {
int isInited = 1;
struct Queue* queue;
struct flock lock = {
.l_type = F_WRLCK,
.l_whence = SEEK_SET,
.l_start = 0,
.l_len = 0
};
ipc_t conn = malloc(sizeof(struct ipc_t));
sprintf(conn->name, "/arqvenger_%s", name);
conn->fd = shm_open(conn->name, O_CREAT | O_RDWR, 0666);
if (conn->fd == -1) {
free(conn);
perror("shm_open failed");
return NULL;
}
if (fcntl(conn->fd, F_SETLKW, &lock) == -1) {
perror("Tanked...");
}
// Do stuff with the lock & release it
The output I get is:
Tanked...: Bad file descriptor
A common technique is to first call shm_open with O_CREAT|O_EXCL. This will succeed for only one process that then has to do the setup. The others then would have to do the open as before and wait a bit, probably polling, that the setup is finished.
Edit: To show how this could work as discussed in the comments.
struct head {
unsigned volatile flag;
pthread_mutex_t mut;
};
void * addr = 0;
/* try shm_open with exclusive, and then */
if (/* we create the segment */) {
addr = mmap(something);
struct head* h = addr;
pthread_mutex_init(&h->mut, aSharedAttr);
pthread_mutex_lock(&h->mut);
h->flag = 1;
/* do the rest of the initialization, and then */
pthread_mutex_unlock(&h->mut);
} else {
/* retry shm_open without exclusive, and then */
addr = mmap(something);
struct head* h = addr;
/* initialy flag is guaranteed to be 0 */
/* this will break out of the loop whence the new value is written to flag */
while (!h->flag) sched_yield();
pthread_mutex_lock(&h->mut);
pthread_mutex_unlock(&h->mut);
}
I have been able to reproduce the problem. Then I found a sad notice in the standard:
When the file descriptor fildes refers to a shared memory object, the
behavior of fcntl() shall be the same as for a regular file except the
effect of the following values for the argument cmd shall be
unspecified: F_SETFL, F_GETLK, F_SETLK, and F_SETLKW.
So it's likely it's not yet supported. For such a simple setup I would use a semaphore for signaling "the memory is ready".
EDIT
It seems I need to mention semaphores can be created using "O_EXCL" (so there are no races).
Related
I have two processes that need to communicate with each other. One is the sender and the other is the receiver.
The receiver should read a word from the user and save it in the shared memory. The sender should then read this word and use it to save an answer to it in the shared memory. Now the receiver should output this answer.
Small minimal example:
//Process 1
//
do
{
strcpy(shared_mem->message, getCommand());
sem_post(&semaphore_getMessage);
sem_wait(&semaphore_getAnswer);
printf("%s",shared_mem->Answer);
} while (getCommand()!=0);
//Process 2
//
sem_wait(&semaphore_getMessage);
do
{
strcpy(shared_mem->Answer, getAnswer(shared_mem->Message));
sem_post(&semaphore_getAnswer);
sem_wait(&semaphore_getMessage);
} while (strcmp(shared_mem->Message, "Exit\n") != 0);
Unfortunately, I sometimes get a deadlock (i.e. my programme is permanently stuck in a state). I just don't know how... Does anyone have an idea? Language is C
The Semaphores are defined in an struct:
typedef struct
{
sem_t semaphore_getMessage;
sem_t semaphore_getAnswer;
} semaphores;
as are the filedescriptors and the mapping of the memory:
typedef struct
{
sempahores *mapped_region_locks;
} mappings;
typedef struct
{
int locks;
} filedescriptor;
The locks are then mapped to shared space:
filedescriptor->locks = shm_open(NAME_SEMAPHORES, FLAGS_READWRITE_ACCES, MODE_RW);
ftruncate(filedescriptor->locks, sizeof(filedescriptor));
mappings->mapped_region_locks = mmap(NULL, sizeof(semaphores),
PROT_READ | PROT_WRITE, MAP_SHARED, filedescriptor->locks, 0);
sem_init(&mapped_region_locks->semaphore_getMessage, 0, 1);
//in the first code snippet "mapped_region_locks->semaphore_getMessage" is referred to as "semaphore_getMessage" as it is easier to read.
Within the Linux Kernel (specifically for device drivers), how would I know what variables to lock and when they need locking? In particular, why does the locking in the following code only happen after dev has been set, even though dev points to a global variable scull_devices?
struct scull_qset {
void **data; /* pointer to an array of pointers which each point to a quantum buffer */
struct scull_qset *next;
};
struct scull_dev {
struct scull_qset *data; /* Pointer to first quantum set */
int quantum; /* the current quantum size */
int qset; /* the current array size */
unsigned long size; /* amount of data stored here */
unsigned int access_key; /* used by sculluid and scullpriv */
struct semaphore sem; /* mutual exclusion semaphore */
struct cdev cdev; /* Char device structure initialized in scull_init_module */
};
struct scull_dev *scull_devices; /* allocated dynamically in scull_init_module */
int scull_open(struct inode *inode, struct file *filp)
{
struct scull_dev *dev; /* device information */
dev = container_of(inode->i_cdev, struct scull_dev, cdev);
filp->private_data = dev; /* for other methods */
/* now trim to 0 the length of the device if open was write-only */
if ( (filp->f_flags & O_ACCMODE) == O_WRONLY) {
if (down_interruptible(&dev->sem))
return -ERESTARTSYS;
scull_trim(dev); /* empty out the scull device */
up(&dev->sem);
}
return 0; /* success */
}
If the code for scull_init_module is needed for a more complete picture, here it is:
int scull_major = SCULL_MAJOR;
int scull_minor = 0;
int scull_quantum = SCULL_QUANTUM;
int scull_qset = SCULL_QSET;
int scull_nr_devs = SCULL_NR_DEVS;
int scull_init_module(void)
{
int result, i;
dev_t dev = 0;
/* assigns major and minor numbers (left out for brevity sake) */
/*
* allocate the devices -- we can't have them static, as the number
* can be specified at load time
*/
scull_devices = kmalloc(scull_nr_devs * sizeof(struct scull_dev), GFP_KERNEL);
if (!scull_devices) {
result = -ENOMEM;
goto fail;
}
memset(scull_devices, 0, scull_nr_devs * sizeof(struct scull_dev));
/* Initialize each device. */
for (i = 0; i < scull_nr_devs; i++) {
scull_devices[i].quantum = scull_quantum;
scull_devices[i].qset = scull_qset;
init_MUTEX(&scull_devices[i].sem);
scull_setup_cdev(&scull_devices[i], i);
}
/* some other stuff left out for brevity sake */
return 0; /* succeed */
fail: /* isn't this a little redundant? */
scull_cleanup_module();
return result;
}
/*
* Set up the char_dev structure for this device.
*/
static void scull_setup_cdev(struct scull_dev *dev, int index)
{
int err, devno = MKDEV(scull_major, scull_minor + index);
cdev_init(&dev->cdev, &scull_fops);
dev->cdev.owner = THIS_MODULE;
dev->cdev.ops = &scull_fops;
err = cdev_add (&dev->cdev, devno, 1);
/* Fail gracefully if need be */
if (err)
printk(KERN_NOTICE "Error %d adding scull%d", err, index);
}
Locking in the example has nothing to do with the global scull_devices variable, but the locking is used to protect attributes of one scull_dev.
E.g. assume there exists a read() operation which copies size bytes from data while the mentioned scroll_trim() operation frees data.
So, when process #1 calls open() and process #2 tries to read() from an already opened device at the same time, the read() operation can access freed data and oopses.
That is why you need to protect data against races. Semaphores are one way; mutexes another one which is often more appropriate. Spinlocks and atomic variables might work too.
lock - it is way to protect critical section
critical section - in your driver code, if multiple instances are accessing same area, that is critical section.
multiple instances - it could be thread, regular ioctl cmd(from user space), and softirq and irq. It depends on your driver implementation.
Based on "context", you should use different lock too.
thread context which can sleep -> semaphore/mutex
non-sleeping context -> spinlock
softirq, tasklet -> spin_lock_bh
irq -> spin_lock_irq, spin_lock_irqsave
It is completely based on your requirements.
Let's take an example. If you are working on network driver, your netdev has stats and packet buffer and those needs to be protected by lock since it can be updated by multiple instances like net_rx_softirq, net_tx_softirq, ioctl/netlink from userspace request an so on.
In this case, based on your resource's context, you need to use different lock/mutex and sometimes you need more than 1 lock.
I have a queue in shared memory. It does work on Linux (kernel 4.3.4), but not on Mac OS X. Are there any differences between how Mac OS X handles shared memory and how linux does, which may explain this?
I get the shared memory via:
int sh_fd = shm_open(shmName, O_RDWR | O_CREAT,
S_IROTH | S_IWOTH // others hav read/write permission
| S_IRUSR | S_IWUSR // I have read/write permission
);
// bring the shared memory to the desired size
ftruncate(sh_fd, getpagesize());
The queue is very simple as well. Here is the basic struct:
typedef struct {
// this is to check whether the queue is initialized.
// on linux, this will be 0 initially
bool isInitialized;
// mutex to protect concurrent access
pthread_mutex_t access;
// condition for the reader, readers should wait here
pthread_cond_t reader;
// condition for the writer, writers should wait here
pthread_cond_t writer;
// whether the queue can still be used.
bool isOpen;
// maximum capacity of the queue.
int32_t capacity;
// current position of the reader and number of items.
int32_t readPos, items;
// entries in the queue. The array actually is longer, which means it uses the space behind the struct.
entry entries[1];
} shared_queue;
Basically everyone who wants access acquires the mutex, readPos indicates where the next value should be read (incrementing readPos afterwards), (readPos+items) % capacity is where new items go. The only somewhat fancy trick is the isInitialized byte. ftruncate fills the shared memory with zeros if it had length 0 before, so I rely on isInitiualized to be zero on a fresh shared memory page and write a 1 there as soon as I initialize the struct.
As I said, it works on Linux, so I don't think it is a simple implementation bug. Is there any subtle difference between shm_open on Mac vs. Linux which I may not be aware of? The bug I see looks like the reader tries to read from an empty queue, so, maybe the pthread mutex/condition does not work on shared memory in a Mac?
The problem is that PTHREAD_PROCESS_SHARED is not supported on mac.
http://alesteska.blogspot.de/2012/08/pthreadprocessshared-not-supported-on.html
You must set PTHREAD_PROCESS_SHARED on both the mutex and condition variables.
So for a mutex:
pthread_mutexattr_t mutex_attr;
pthread_mutex_t the_mutex;
pthread_mutexattr_init(&mutex_attr);
pthread_mutexattr_setpshared(&mutex_attr, PTHREAD_PROCESS_SHARED);
pthread_mutexattr(&the_mutex, &mutex_attr);
Basically the same steps for the condition variables, but replace mutexattr with condattr.
If the the pthread_*attr_setpshared functions don't exist or return an error, then it may not be supported on your platform.
To be on the safe side, you might want to set PTHREAD_MUTEX_ROBUST if supported. This will prevent deadlock over the mutex (though not guarantee queue consistency) if a process exits while holding the lock.
EDIT: As an added caution, having a boolean "is initialized" flag is an insufficient plan on its own. You need more than that to really guarantee only one process can initialize the structure. At the very least you need to do:
// O_EXCL means this fails if not the first one here
fd = shm_open(name, otherFlags | O_CREAT | O_EXCL );
if( fd != -1 )
{
// initialize here
// Notify everybody the mutex has been initialized.
}
else
{
fd = shm_open(name, otherFlags ); // NO O_CREAT
// magically somehow wait until queue is initialized.
}
Are you sure really need to roll your own queue? Will POSIX message queues (see mq_open man page) do the job? If not, what about one of many messaging middleware solutions out there?
Update 2016-Feb-10: Possible mkfifo based solution
One alternative to implementing your own queue in shared memory is to use an OS provided named FIFO using mkfifo. A key difference between a FIFO and a named pipe is that you are allowed to have multiple simultaneous readers and writers.
A "catch" to this, is that the reader sees end-of-file when the last writer exits, so if you want readers to go indefinitely, you may need to open a dummy write handle.
FIFOs are super easy to use on the command line, like so:
reader.sh
mkfifo my_queue
cat my_queue
write.sh
echo "hello world" > my_queue
Or slightly more effort in C:
reader.c
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <errno.h>
int main(int argc, char**argv)
{
FILE * fifo;
FILE * wfifo;
int res;
char buf[1024];
char * linePtr;
/* Try to create the queue. This may belong on reader or writer side
* depending on your setup. */
if( 0 != mkfifo("work_queue", S_IRUSR | S_IWUSR ) )
{
if( errno != EEXIST )
{
perror("mkfifo:");
return -1;
}
}
/* Get a read handle to the queue */
fifo = fopen("work_queue", "r");
/* Get a write handle to the queue */
wfifo = fopen("work_queue", "w");
if( !fifo )
{
perror("fopen: " );
return -1;
}
while(1)
{
/* pull a single message from the queue at a time */
linePtr = fgets(buf, sizeof(buf), fifo);
if( linePtr )
{
fprintf(stdout, "new command=%s\n", linePtr);
}
else
{
break;
}
}
return 0;
}
writer.c
#include <stdio.h>
#include <unistd.h>
int main(int argc, char**argv)
{
FILE * pipe = fopen("work_queue", "w");
unsigned int job = 0;
int my_pid = getpid();
while(1)
{
/* Write one 'entry' to the queue */
fprintf(pipe, "job %u from %d\n", ++job, my_pid);
}
}
I'm reading APUE and I am confused with thread synchronization of chapter 11. Below is a code snippet.
#define NHASH 29
#define HASH(fp) (((unsigned long)fp)%NHASH)
struct foo *fh[NHASH];
pthread_mutex_t hashlock = PTHREAD_MUTEX_INITIALIZER;
struct foo {
int f_count;
pthread_mutex_t f_lock;
struct foo *f_next; /* protected by hashlock */
int f_id;
/* ... more stuff here ... */
};
struct foo *
foo_alloc(void) /* allocate the object */
{
struct foo *fp;
int idx;
if ((fp = malloc(sizeof(struct foo))) != NULL) {
fp->f_count = 1;
if (pthread_mutex_init(&fp->f_lock, NULL) != 0) {
free(fp);
return(NULL);
}
idx = HASH(fp);
pthread_mutex_lock(&hashlock);
fp->f_next = fh[idx];
fh[idx] = fp;
pthread_mutex_lock(&fp->f_lock);
pthread_mutex_unlock(&hashlock);
/* ... continue initialization ... */
pthread_mutex_unlock(&fp->f_lock);
}
return(fp);
}
My doubts are:
Why place pthread_mutex_lock(&fp->f_lock) before the pthread_mutex_unlock(&hashlock)? Could I place it afterward instead?
Since fp is local variable, could pthread_mutex_lock(&fp->f_lock) and pthread_mutex_unlock(&fp->f_lock) be removed all together?
No, because the actions after the pthread_mutex_lock(&hashlock) expose the newly created structure to other threads by adding it to the the fh list. While the hashlock is held, no-one can access the variable; as soon as the hashlock is released, it becomes accessible to other threads via the hash, but locking the fp_>f_lock mutex prevents anyone messing with fp.
Not with the code as written. If the whole structure was initialized except for the hashing, then you could do without locking the fp->f_lock mutex; just before completing, you'd lock the hashlock, hook the newly allocated item into the hash tables, and then release the hashlock, and you would be safe. If you need any exclusive access after the structure is added into the hash table, you have to acquire its mutex. The way it is written, that's a non-waiting acquisition of the mutex; there is no other process that could have access to the variable.
if ((fp = malloc(sizeof(struct foo))) != NULL) {
fp->f_count = 1;
if (pthread_mutex_init(&fp->f_lock, NULL) != 0) {
free(fp);
return(NULL);
}
idx = HASH(fp);
/* ... complete initialization except for adding to hash table ... */
pthread_mutex_lock(&hashlock);
fp->f_next = fh[idx];
fh[idx] = fp;
pthread_mutex_unlock(&hashlock);
}
So, there is logic behind what is done; it is correct.
I guess that there's a second thread that's looping through the created objects, doing something with them. In this case:
No, because the loop thread might access the newly created object before being initialized at all.
No, because the loop thread might access the newly created object being half initialized.
How can I distinguish between "listener" file descriptors and "client" file descriptors?
Here's what I saw in the manpage example:
if(events[n].data.fd == listener) {
...
} else {
...
}
'But what if I don't have access to listener?
Sorry if this is a vague question. I'm not quite sure how to word it.
Assuming you are writing a server, you should either keep the listening socket descriptor around in some variable (listener in the manual page), or setup a small structure for each socket you give to epoll_ctl(2) and point to it with data.ptr member of the struct epoll_event (don't forget to de-allocate that structure when socket is closed).
Something like this:
struct socket_ctl
{
int fd; /* socket descriptor */
int flags; /* my info about the socket, say (flags&1) != 0 means server */
/* whatever else you want to have here, like pointers to buffers, etc. */
};
...
struct socket_ctl* pctl = malloc( sizeof( struct socket_ctl ));
/* check for NULL */
pctl->fd = fd;
pctl->flags = 1; /* or better some enum or define */
struct epoll_event ev;
ev.events = EPOLLIN|...;
ev.data.ptr = pctl;
...
if (( events[n].data.ptr->flags & 1 ) != 0 )
{
/* this is server socket */
}
As you can see it's much more work then just having access to the server socket descriptor, but it has a nice property of keeping all information related to one socket in one place.