I have a thread I made to accept incoming connections with:
pthread_t thread;
pthread_create(&thread, NULL, worker_func, NULL);
pthread_detach(thread);
Then worker_func does:
<snip>
fd = accept(GlobalFD, NULL, NULL);
<snip>
However, If I try to close GlobalFD in my original thread, accept will still block (I want it to fail if GlobalFD is closed). I've looked online and at other questions and can't seem to find the answer to my particular problem.
Any ideas?
Thanks
Different threads of the same program share memory, including file descriptor tables. If one thread closes an FD then it is closed for all other threads, too. This is one of the differences between using multiple threads and using multiple processes. Therefore, do not allow one thread to close a file descriptor that another is relying upon to remain open.
More generally, however, you must take great care about modifying shared data. Generally speaking, you must synchronize access via a semaphore, a condition variable, or some other construct or action with significance for synchronization. Program behavior is otherwise not well defined. In particular, you cannot expect that closing a file descriptor will cause an active I/O function running in a different thread to terminate. Send the target thread a signal instead.
In general, closing a file-descriptor in one thread isn't guaranteed to cause a function that's waiting on that file-descriptor in another thread to return.
Your options are 1) install a signal handler and use pthread_kill() (don't forget to check the return-code of the blocked function) and 2) create a "termination file-descriptor" that is also passed to the select() or poll() function and either close it or write to it in the other thread.
Good luck.
Related
I'm experimenting with a fictional server/client application where the client side launches request threads by a (possibly very large) period of time, with small in-between delays. Each request thread writes on the 'public' fifo (known by all client and server threads) the contents of the request, and receives the server answer in a 'private' fifo that is created by the server with a name that is implicitly known (in my case, it's 'tmp/processId.threadId').
The public fifo is opened once in the main (request thread spawner) thread so that all request threads may write to it.
Since I don't care about the return value of my request threads and I can't make sure how many request threads I create (so that I store their ids and join them later), I opted to create the threads in a detached state, exit the main thread when the specified timeout expires and let the already spawned threads live on their own.
All of this is fine, however, I'm not closing the public fifo anywhere after all spawned request threads finish: after all, I did exit the main thread without waiting. Is this a small kind of disaster, in which case I absolutely need to count the active threads (perhaps with a condition variable) and close the fifo when it's 0? Should I just accept that the file is not explicitly getting closed, and let the OS do it?
All of this is fine, however, I'm not closing the public fifo anywhere
after all spawned request threads finish: after all, I did exit the
main thread without waiting. Is this a small kind of disaster, in
which case I absolutely need to count the active threads (perhaps with
a condition variable) and close the fifo when it's 0? Should I just
accept that the file is not explicitly getting closed, and let the OS
do it?
Supposing that you genuinely mean a FIFO, such as might be created via mkfifo(), no, it's not a particular issue that the process does not explicitly close it. If any open handles on it remain when the process terminates, they will be closed. Depending on the nature of the termination, it might be that pending data are not flushed, but that is of no consequence if the FIFO is used only for communication among the threads of one process.
But it possibly is an issue that the process does not remove the FIFO. A FIFO has filesystem persistence. Once you create one, it lives until it no longer has any links to the filesystem and is no longer open in any process (like any other file). Merely closing it does not cause it to be removed. Aside from leaving clutter on your filesystem, this might cause issues for concurrent or future runs of the program.
If indeed you are using your FIFOs only for communication among the threads of a single process, then you would probably be better served by pipes.
I managed to solve this issue setting up a cleanup rotine with atexit, which is called when the process terminates, ie. all threads finish their work.
I'm working on a kevent/kqueue emulation library for Linux. I'm a new maintainer on this project, and unfortunately, the previous maintainer is not involved much anymore (so I can't pick their brains about this).
Under FreeBSD and macOS when you close() the file descriptor provided by kqeueue() you free any resources and events associated with it.
It seems like the existing code doesn't provide a similar interface. Before I add a function to the API (or revive an old one) to explicitly free kqueue resources, I was wondering if there was any way to associate triggers with a file descriptor in linux, so that when it's closed we can cleanup anything associated with the FD.
The file descriptor itself could be any type, i.e. one provided by eventfd, or epoll or anything else that creates file descriptors.
When the last write file descriptor from a pipe() call is closed epoll()/poll() waiters will see an [E]POLLHUP event on any read file descriptors still open. Presumably the same is true of any fd that represents a connection rather than state.
The solution to this is fairly simple, if a little annoying to implement. It relies on an fcntl called F_SETSIG, to specify the signal used to communicate FD state changes, and an fcntl called F_SETOWN_EX to specify what thread the signal should be delivered to.
When the application starts it spawns a separate monitoring thread. This thread is used to receive FD generated signals.
In our particular use case, the monitoring thread must be started implicitly the first time a monitored FD is created, and destroyed without an explicit join. This is because we're emulating a FreeBSD API (kqueue), which does not have explicit init and deinit functions.
The monitoring thread:
Listens on a the signal we passed to F_SETSIG.
Gets its thread ID, and stores it in a global.
Informs the application that the monitoring thread has started (and the global is filled) using pthread_cond_broadcast.
Calls pthread_detach to ensure it's cleaned up correctly without another thread needing to do an explicit pthread_join.
Calls sigwaitinfo to wait on delivery of a signal.
The application thread(s):
Uses pthread_once to start the monitoring thread the first time a FD is created, then waits for the monitoring thread to start fully.
Uses F_SETSIG to specify the signal sent when the FD is open/closed, and F_SETOWN_EX to direct those signals to the monitoring thread.
When a monitored FD is closed the sigwaitinfo call in the monitoring thread returns. In our case we're using a pipe to represent the kqueue, so we need to map the FD we received the signal for, to the one associated with the resources (kqueues) we need to free. Once this mapping is done, we may (see below for more information) cleanup the resources associated with the FD pair, and call sigwaitinfo again to wait for more signals.
One of the other key pieces to this strategy, is that the resources associated with the FDs are reference counted. This is because the signals are not synchronously delivered, so an FD can be closed, and a new FD can be created with the same number, before the signal indicating the original FD was closed, was delivered and acted on. This would obviously cause big issues with active resources being freed.
To solve this we maintain a mutex synchronised FD to resource mapping array. Each element in this array contains a reference count for a particular FD.
In the case where the signal is not delivered before the FD is reused when creating a new pipe/resource pair the reference count for that particular FD will be > 0. When this occurs we immediately free the resource, and reinitialise it, increasing the reference count.
When the signal indicating the FD was closed is delivered, the reference count is decremented (but not to zero), and the resource is not freed.
Alternatively if the signal is delivered before the FD was reused, then the monitoring thread will decrement the reference count to zero, and immediately free the associated resources.
If this description is a bit confusing you can look over our real world implementation using any of the links above.
Note: Our implementation isn't exactly as was described above (notably we don't check the reference count of an FD when creating a new FD/resource mapping). I think this is because we rely on the fact that closing one of the pipe doesn't necessarily result in the other end being closed, so the open end's FD isn't available for reuse immediately. Unfortunately the developer who wrote the code isn't available for querying.
This is not any error in code, but to get an idea how to accomplish the following criteria. I have seen similar questions here and here, but did not get any particular way to do it.
Consider the following parts of code:
....
pthread_create(&Receiver,NULL,ReceiveFromServer,(void*)&ClientSoc);
sending =1;
while (sending)
{
SHOW_PROMPT;
gets(message);
memcpy(Packet.data,message,MAX_DATA_LENGTH);
if (SetCommand(&Packet,message) == CMD_EXT)
sending = 0;
send(ClientSoc,&Packet,PACKET_SIZE,0);
}
close(ClientSoc);
pthread_join(Receiver,NULL);
return 0;
....
And the ReceiveFromServer function:
void* ReceiveFromServer(void* ClientSoc)
{
int receiving =1;
int Status;
strPacket Packet;
while(receiving)
{
if(recv(*(int*)ClientSoc,&Packet,PACKET_SIZE,0)>0)
{
ParseReply(Packet);
SHOW_PROMPT;
}
if(GET_COMMAND(Packet.Header) == CMD_EXT)
receiving = 0;
}
return NULL;
}
Assume everything is declared or defined correctly.
So the thread terminates depending upon the received data from server, but the sender still loops as it do not know that the receiver is terminated. The sender only comes out of loop depending upon the user input (a particular word such as 'exit').How to notify the parent that the thread is terminating? I tried to make sending global and change it from inside the thread, but it didn't work.
Close the socket in the 'ReceiveFromServer' thread when you get the CMD_EXT. The sending thread should get the message soon enough - the send() call will return with an error.
In the main loop of the parent, replace:
while (sending)
with:
while (sending && pthread_kill(Receiver, 0) != ESRCH)
This will check the existence of the Receiver thread. See How do you query a pthread to see if it is still running? for more info.
Note, however, that How do I determine if a pthread is alive? emphasizes that you cannot do this for a detached thread. And http://man7.org/linux/man-pages/man3/pthread_kill.3.html states:
POSIX.1-2008 recommends that if an implementation detects the use of a thread ID after the end of its lifetime, pthread_kill() should return the error ESRCH. The glibc implementation returns this error in the cases where an invalid thread ID can be detected. But note also that POSIX says that an attempt to use a thread ID whose lifetime has ended produces undefined behavior, and an attempt to use an invalid thread ID in a call to pthread_kill() can, for example, cause a segmentation fault.
Closing the socket in the ReceiveFromServer thread when you get the CMD_EXT, would probably also work, as Are parallel calls to send/recv on the same socket valid? seems to indicate POSIX sockets are thread safe. But R. strongly recommends against it in Is closesocket thread safe?. And the RHEL5 Linux man 2 close states:
It is probably unwise to close file descriptors while they may be in use by system calls in other threads in the same process. Since a file descriptor may be re-used, there are some obscure race conditions that may cause unintended side effects.
When dealing with sockets, you have to be sure that there is no recv(2) still blocking on it on another thread, otherwise it might block forever, since no more messages will be sent via the socket. Be sure to use shutdown(2) to shut down all parts the connection before closing the socket.
Also, you'll probably also want to replace gets(message) with a non-blocking equivalent, to handle the case where the peer asynchronously sends a CMD_EXT while you are waiting for human input.
And of course you'll need to add socket error-handling, but that is beyond the scope of this question.
Maybe you could try to use pthread_tryjoin_np (I never used it) in the parent loop to detect if the thread has finished. It is only available for linux as I understood.
I have a thread that is essentially just for listening on a socket. I have the thread blocking on accept() currently.
How do I tell the thread to finish any current transaction and stop listening, rather than staying blocked on accept?
I don't really want to do non-blocking if I don't have to...
Use the select(2) call to check which fd are ready to read.
The file descriptors from call can be read with out it blocking. eg accept() on the returned fd will immediately create a new connection.
Basically you have two options, the first one is to use interrupts: i.e
http://www.cs.cf.ac.uk/Dave/C/node32.html (see the signal handler section, it also supply a th_kill example).
From accept man page:
accept() shall fail if:
EINTR
The system call was interrupted by a signal that was caught before a valid connection arrived.
Another option is to use Non blocking sockets and select(): i.e.:
http://publib.boulder.ibm.com/infocenter/iseries/v5r3/index.jsp?topic=%2Frzab6%2Frzab6xnonblock.htm
Anyhow, usually in multi-threaded servers there's one thread which accepts new connections and spawns other threads for each connections. Since accept()ing and than recv()ing, can delay new connections requests... (Unless you're working with one client, and then accept()ing and recieving might be OK)
Use pthread_cancel on the thread. You'll need to make sure you've installed appropriate cancellation handlers (pthread_cleanup_push) to avoid resource leaks, and you should disable cancellation except for the duration of the accept call to avoid race conditions where the cancellation request might get acted upon later by a different function than accept.
Note that, due to bugs in glibc's implementation of cancellation, this approach could lead to lost connections and file descriptor leaks. This is because glibc/NPTL provides no guarantee that accept did not already finish execution and allocate a new file descriptor for the new connection before the cancellation request is acted upon. It should be a fairly rare occurrence but it's still an issue to consider...
See: http://sourceware.org/bugzilla/show_bug.cgi?id=12683
and for a discussion of the issue: Implementing cancellable syscalls in userspace
From Wake up thread blocked on accept() call
I just used the shutdown() system call and it seems to work...
I'm running a multi-threaded C program (process?) , making use of semaphores & pthreads. The threads keep interacting, blocking, waking & printing prompts on stdout continuously, without any human intervention. I want to be able to exit this process (gracefully after printing a message & putting down all threads, not via a crude CTRL+C SIGINT) by pressing a keyboard character like #.
What are my options for getting such an input from the user?
What more relevant information could I provide that will help to solve this problem?
Edit:
All your answers sound interesting, but my primary question remains. How do I get user input, when I don't know which thread is currently executing? Also, semaphore blocking using sem_wait() breaks if signalled via SIGINT, which may cause a deadlock.
There is no difference in reading standard input from threads except if more than one thread is trying to read it at the same time. Most likely your threads are not all calling functions to read standard input all the time, though.
If you regularly need to read input from the user you might want to have one thread that just reads this input and then sets flags or posts events to other threads based on this input.
If the kill character is the only thing you want or if this is just going to be used for debugging then what you probably want to do is occasionally poll for new data on standard input. You can do this either by setting up standard input as non-blocking and try to read from it occasionally. If reads return 0 characters read then no keys were pressed. This method has some problems, though. I've never used stdio.h functions on a FILE * after having set the underlying file descriptor (an int) to non-blocking, but suspect that they may act odd. You could avoid the use of the stdio functions and use read to avoid this. There is still an issue I read about once where the block/non-block flag could be changed by another process if you forked and exec-ed a new program that had access to a version of that file descriptor. I'm not sure if this is a problem on all systems. Nonblocking mode can be set or cleared with a 'fcntl' call.
But you could use one of the polling functions with a very small (0) timeout to see if there is data ready. The poll system call is probably the simplest, but there is also select. Various operating systems have other polling functions.
#include <poll.h>
...
/* return 0 if no data is available on stdin.
> 0 if there is data ready
< 0 if there is an error
*/
int poll_stdin(void) {
struct pollfd pfd = { .fd = 0, .events = POLLIN };
/* Since we only ask for POLLIN we assume that that was the only thing that
* the kernel would have put in pfd.revents */
return = poll(&pfd, 1, 0);
}
You can call this function within one of your threads until and as long as it retuns 0 you just keep on going. When it returns a positive number then you need to read a character from stdin to see what that was. Note that if you are using the stdio functions on stdin elsewhere there could actually be other characters already buffered up in front of the new character. poll tells you that the operating system has something new for you, not what C's stdio has.
If you are regularly reading from standard input in other threads then things just get messy. I'm assuming you aren't doing that (because if you are and it works correctly you probably wouldn't be asking this question).
You would have a thread listening for keyboard input, and then it would join() the other threads when receiving # as input.
Another way is to trap SIGINT and use it to handle the shutdown of your application.
The way I would do it is to keep a global int "should_die" or something, whose range is 0 or 1, and another global int "died," which keeps track of the number of threads terminated. should_die and died are both initially zero. You'll also need two semaphores to provide mutex around the globals.
At a certain point, a thread checks the should_die variable (after acquiring the mutex, of course). If it should die, it acquires the died_mutex, ups the died count, releases the died_mutex, and dies.
The main initial thread periodically wakes up, checks that the number of threads that have died is less than the number of threads, and goes back to sleep. The main thread dies when all the other threads have checked in.
If the main thread doesn't spawn all the threads itself, a small modification would be to have "threads_alive" instead of "died". threads_alive is incremented when a thread forks, and decremented when the thread dies.
In general, terminating a multithreaded operation cleanly is a pain in the butt, and besides special cases where you can use things like the semaphore barrier design pattern, this is the best I've heard of. I'd love to hear it if you find a better, cleaner one.
~anjruu
In general, I have threads waiting on a set of events and one of those events is the termination event.
In the main thread, when I have triggered the termination event, I then wait on all the threads having exited.
SIGINT is actually not that difficult to handle and is often used for graceful termination. You need a signal handler and a way to tell all the threads that it's time to stop. One global flag that threads check in their loops and the signal handler sets might do. Same approach works for "on user command" termination, though you need a way to get the input from the terminal - either poll in a dedicated thread, or again, set the terminal to generate a signal for you.
The tricky part is to unblock waiting threads. You have to carefully design the notification protocol of who tells who to stop and what they need to do - put dummy message into a queue, set a flag and signal a cv, etc.