how to restart socket properly in a multithread c/c++ program - c

Background: My code structure: I have a master socket on main thread, then each time a new client is coming, the threadpool will be notified and let one pre allocated thread take the task.
Inside this thread, I will pass a slave socket to it, and let it using accept call to listen to the client.
Scenario: In my thread pool, thread A is listening to a client right now, now I want to stop all the pre-allocated thread and close all the connection to the client, the main thread is trying to close the connection using close the connection to the client, and trying to terminate thread A using pthread_join.
main() {
// create threadpool
// logic to create mastersocket
startServer(masterSock)
IwantToCloseServer() // this function is not directly called in main, but simulated by a terminal signal , like kill -quit pid.
}
int startServer(int msock) {
int ssock; // slaveSocket
struct sockaddr_in client_addr; // the address of the client...
unsigned int client_addr_len = sizeof(client_addr); // ... and its length
while (!stopCondition) {
// Accept connection:
ssock = ::accept((int)msock, (struct sockaddr*)&client_addr, &client_addr_len); // the return value is a socket
// I was trying to replace this line of code to poll(), but it's not does the same thing as before
if (ssock < 0) {
if (errno == EINTR) continue;
perror("accept");
running =0;
return 0;
// exit(0);
} else {
// push task to thread pool to deal with logic
}
// main thread continues with the loop...
}
return 1;
}
IwantToCloseServer(slaveSocket) {
// when i want to close() or shutdown() function to close connections, these 2 function always return -1, because the thread is blocked on accept call
// logic try to terminate all the preallocated threads, the pthread_join function is stuck because the thread is blocked on accept
}
Problem: The thread A is keeping blocking on the ::accept function , the close and shutdown function return -1, they won’t close the connection , and the pthread_join is not keep going because thread A is blocked on accept.
Things I tried:
I have try to change my while loop related accept function, for example, set a flag stopCondition,
while(!stopConditon) {
ssock = ::accept((int)msock, (struct sockaddr*)&client_addr, &client_addr_len);
}
However, when the main thread change stopCondtion, the thread A is blocked inside the accept function.
It won’t go inside the while loop, so this solution won’t affect the accept function, it’s not working
I have also tried to send a signal to this blocked Thread A, using
pthread_cancel or pthread_kill(Thread A, 9)
However, if I do this, the whole process gets killed.
3.try to use poll() to replace the line, where the accept functions at, with a timeout
however, the program doesn't behave like before, the program can't listen to client anymore.
How do I terminate thread A (which is blocked on accept function call right now), so that I can clean this pre allocated thread and restart my server ?
btw i can not use library like boost in my current program. And this is under linux system not winsocket

to check periodically stopConditon in your while(!stopConditon) { first call accept/pool with a timeout to know if there is something new about msock, then depending on the result call accept etc else do nothing
I was trying to replace this line of code to poll()
try to use poll() to replace the line, where the accept functions at, with a timeout
you cannot replace accept by poll, you have to call accept / pool first and of course check the result then may be call accept
Out of that
while(!stopConditon) {
if(!stopCondtion) {
is redundant and can be replaced by
while(!stopConditon) {

Related

one thread to exit them all

I have a main program that generates a few threads (using a while loop with accept() to get clients), and one that all it has to do is "listen to the keyboard" and when the user enters the word exit it will close the entire program.
first, the main program create the listening thread, then it enters a while loop that accept the clients. even if the listening thread get the exit input the loop is still stuck on accept.
i don't have to use a seperate thread to listen to the keyboard but i could'nt find a none blocking way that would work.
the listening thread:
DWORD WINAPI ListenService(LPVOID lpParam)
{
char buffer[5];
if (EOF == scanf("%s", buffer))
{
printf("faile get word from keyboard\n");
}
if (buffer[4] != '\0')
strcat(buffer, "\0");
if (STRINGS_ARE_EQUAL(buffer, "exit"))
{
return 999;
}
return -1;
}
in the main code:
ThreadListen = CreateThread(NULL,0,ListenService,NULL,0,&(ThreadId));
while(1)
{
SOCKET AcceptSocket = accept(MainSocket, NULL, NULL);
if (AcceptSocket == INVALID_SOCKET)
{
printf("Accepting connection with client failed, error %ld\n", WSAGetLastError());
CleanupWorkerThreads();
WSACleanup();
}
printf("Client Connected.\n");
}
There are many different ways you can handle this.
You can abort a blocked accept() by simply closing the listening socket.
Or, you can use select() with a short timeout to detect when a new client is waiting before then calling accept(). You can check your exit condition in between calls to select(). Just be aware that there is a small race condition where a client may disconnect between the time select() and accect() are called, so accept() may still block, if there are no more clients waiting.
Or, you can get rid of your threads and just use non-blocking sockets in a single thread, checking your exit condition periodically in between socket operations.
Or, you can use asynchronous sockets, using WSACreateEvent(), WSAEventSelect(), and WSAWaitForMultipleEvents() to detect socket activity. Then you can create an addition event to wait on for when the exit condition happens.
Or, you can use an I/O Completion Port to handle socket activity, and then you can post a custom exit packet into the IOCP queue using PostQueuedCompletionStatus() to "wake up" any waiting threads.

Which is the correct way of cancelling main() thread from signal_handler routine?

I am writing a multithreaded server application which echoes back whatever client sends. I am spawning one thread per new client. I have used a while(1) loop to handle successive clients indefinitely (not exactly indefinite, I have forcefully limited it to 64, after which it rejects any new connection). Now I need to handle Ctrl+C signal in my server application.
One problem is that its not recommended to use signal() in multithreaded application.
However, even if I use it to catch the SIGINT signal (from Ctrl+C) using the method already discussed at SO this is what I need to do:
1) The server should no longer accept more clients.
2) The present connections to server should continue, except when client choses to disconnect.
My signal handler function is:
void Ctrl_C_handler(int sig)
{
if(sig == SIGINT) // Ctrl+C
{
printf("Ctrl+C detected by server !!\n");
printf("No more connections will be accepted!!");
pthread_cancel(main_thread_id);
}
}
The methods I propose to use inside main() to cancel the thread are:
// Method 1
pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);
while(1)
{
pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL);
pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, NULL);
// ..
// client handling
// ..
}
// Method 2
while(1)
{
pthread_testcancel(); // Create Cancellation point
// ..
// client handling
// ..
}
// Method 3
while(1)
{
pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL);
pthread_testcancel();
pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, NULL);
// ..
// client handling
// ..
}
Reason for why I am using pthread_cancel():
According to this link the thread will be finally terminated using pthread_exit() (Requirement 1 fulfilled). Since the NOTES section of pthread_exit() explains that it allows other threads to continue execution (Requirement 2 fulfilled).
Now if I am right till here,
Which method is best for cancellation of thread.
I chose method 3 because:
(Demerits) Method 1: Asynchronous thread cancel is not safe (as I saw in an answer in SO) if the thread allocates memory, and few other reasons
(Demerits) Method 2: According to this link there are so many other functions which can also work as cancellation points, some of those I will be using in my code.
(Merits) Method 3: Similar to method 2 but safer because other cancellation points will not be active since thread is cancel disabled.
Please tell me if my way is correct or not? And its wrong, is there any other better method to handle Ctrl+C similarly other than using pthread_cancel()?
I'd suggest avoiding cancellation.
Instead, register a SIGINT handler that close()s the server's listening socket. (Use sigaction for this.) The main thread can quit gracefully after the socket is closed, leaving any client threads running.
E.g. in pseudocode:
static int listen_fd = -1 // server socket
void handle_SIGINT(int s): // Requirement 1
if listen_fd >= 0:
close(listen_fd)
listen_fd = -1
int main(...):
listen_fd = new_listening_socket(...)
block_signal(SIGINT)
trap_signal(SIGINT, handle_SIGINT) // again, calls sigaction
while 1:
unblock_signal(SIGINT)
new_client = accept(listen_fd)
block_signal(SIGINT)
if new_client < 0: // Requirement 1, also general error handling
break
spawn_worker_thread(new_client) // probably attr PTHREAD_CREATE_DETACHED
pthread_exit(NULL) // Requirement 2
return 0 // not reached
You can of course include more nuanced error handling, cleanup routines, etc., as needed.

Making unknown number of child threads inside parent thread using win32 in C?

How can I make unknown number of child threads inside parent thread and wait for each of them one by one using win32 in C?
The parent thread life time is infinite and it is wait for request and if received any request then make a new child thread for that request e.g. like servers .
I am searching the web but I cant find anything .
Any tutorial and information appreciated .
Thanks a lot , good luck .
note :
1 . For example imagine a simple server : When a user send a request to that server the server make a new thread for that user and wait for that thread to terminated but if another user send another request the server make another thread which is completely separate from the old one and then the server must wait for the new thread separate from the old one to terminate .
2 . The main thread scan an global array with the size of constant n in the infinite loop and if find the specific value in each of array's block then run the new thread to do some operation on that block's information and then after the thread become terminate update that block's information . The parent thread life time is infinite because it has a infinite loop .
You'ld create each thread with CreateThread and store the thread handle in a dynamic list or array. If you want the main thread to recognize when a thread terminated, then you can call WaitForMultipleObjects, providing it an array with all thread handles. The return value of WaitForMultipleObjects will tell you which thread handles was signalled, so which thread terminated. Don't forget to CloseHandle the thread handle at the end.
If you just want to spawn threads and the main thread does not need to know when the threads terminate, then you can just create the threads with CreateThread and close the thread handle. The thread's resources will be freed when the thread terminates.
In your main thread you will also need to check if you receive a client request. If you have an Interface to the client where you can wait on an event, then just add the event to the event array passed to WaitForMultipleObjects. If you do not have an event like in your case 2, then you might consider calling WaitForMultipleObjects with a timeout so WaitForMultipleObjects either returns when a thread terminated or when the timeout occured. In both cases your main loop keeps running and you can check if you need to spawn another thread.
Here is some pseudo code when using an event for the client requests:
initialize empty list of thread data (thread handles and other data for each thread);
for(;;) {
create an array a big enough for the request event and for all thread handles;
a[0] = request event handle;
a[1..n] = thread handles from the list;
DWORD ret = WaitForMultiObjects(n+1, a, FALSE, INFINITE);
if(ret == WAIT_OBJECT_0) {
create thread and store it's handle and other data in the list;
}
else if(WAIT_OBJECT_0 + 1 <= ret && ret <= WAIT_OBJECT_0 + n) {
thread (ret - WAIT_OBJECT_0 - 1) terminated, do your cleanup and don't forget CloseHandle();
}
else
error occured, should not happen;
}
If you don't have an event for the client requests, then you need to poll:
initialize empty list of thread data (thread handles and other data for each thread);
for(;;) {
create an array a big enough for the request event and for all thread handles;
a[0..n-1] = thread handles from the list;
DWORD ret = WaitForMultiObjects(n, a, FALSE, your desired timeout);
if(ret != WAIT_TIMEOUT)
; // timeout occured, no thread terminated yet, nothing to do here
else if(WAIT_OBJECT_0 <= ret && ret < WAIT_OBJECT_0 + n) {
thread (ret - WAIT_OBJECT_0) terminated, do your cleanup and don't forget CloseHandle();
}
else
error occured, should not happen;
// Always check for client requests, not only in case of WAIT_TIMEOUT.
// Otherwise you might run into the situation that every time you call WaitForMultiObjects a thread ended just before the timeout occured and you only recognize it after a lot of loop runs when the last thread terminated.
if(there is a client request) {
create thread and store it's handle and other data in the list;
}
}
If you do not need to store extra data for each thread, then you can just store the thread handles and maybe already store them in a big array. Thus you could eliminate the step to build an array from the thread handle list.
Maybe yo should use the WaitForMultipleObjects function.
http://msdn.microsoft.com/en-us/library/windows/desktop/ms687025(v=vs.85).aspx

c - WSAWaitForMultipleObjects blocking any thread but last

i have a problem with a multi-thread SMTP/POP3 server. The server starts a pool of threads to handle incoming connections. The main thread create the sockets and the the threads, passing the sockets as parameters in a proper structure. The loop function for the threads is the following:
SOCKET SMTP_ListenSocket = (SOCKET) data->SMTPconn;
SOCKET POP3_ListenSocket = (SOCKET) data->POP3conn;
static struct sockaddr_in ClntAddr;
unsigned int clntLen = sizeof(ClntAddr);
hEvents[0] = CreateEvent(NULL, FALSE, FALSE, NULL);
hEvents[1] = CreateEvent(NULL, FALSE, FALSE, NULL);
hEvents[2] = exitEvent; //HANDLE FOR A MANUAL RESET EVENT
WSAEventSelect(SMTP_ListenSocket, hEvents[0], FD_ACCEPT);
WSAEventSelect(POP3_ListenSocket, hEvents[1], FD_ACCEPT);
while(1){
DWORD res = WaitForMultipleObjects(3, hEvents, FALSE, INFINITE);
switch(res){
case WAIT_OBJECT_0: {
ClientSocket = my_accept(SMTP_ListenSocket,(struct sockaddr *) &ClntAddr,&clntLen);
/* ... */
my_shutdown(ClientSocket,2);
my_closesocket(ClientSocket);
ClientSocket = INVALID_SOCKET;
break;
}
case WAIT_OBJECT_0 + 1: {
ClientSocket = my_accept(POP3_ListenSocket,(struct sockaddr *) &ClntAddr,&clntLen);
/* ... */
my_shutdown(ClientSocket,2);
my_closesocket(ClientSocket);
ClientSocket = INVALID_SOCKET;
break;
}
case WAIT_OBJECT_0 + 2:
{
exitHandler(0);
break;
}
}//end switch
}//end while
When the pool contains only one thread there's no problem. When the pool consist of more threads, only one thread accepts the incoming connections
Do you have the pooled threads all calling this same code? If so, then don't use WaitForMultipleObjects() (or WSAWaitForMultipleEvents()) like this. This kind of model only works reliably if one thread is polling connections. If you have multiple threads polling at the same time, then you have race conditions.
Instead, you should use AcceptEx() with Overlapped I/O or Completion Ports instead. The thread that creates the sockets can call AcceptEx() on each socket to queue a new operation on each one, then the pooled threads can use GetQueuedCompletionStatus() or GetOverlappedResult() to dequeue a pending connection without worrying about trampling on other threads. Once a connection is accepted, the receiving thread can process it as needed and then call AcceptEx() to queue a new operation for that socket.
Each thread here is setting a new WSAEventSelect prior to entering the wait. This overwrites any existing event selects. This means that, once a thread (call it thread A) accepts a connection, there is no event associated with the socket.
To solve this, you should call WSAEventSelect again within your switch, immediately after the accept(). This will restore the event binding immediately before going into any potentially lengthy processing.
Note that it's possible that two threads may be awoken for the same event, if the timing works out just right. You can hack around that by going back to your wait loop if the accept fails, but this is a bit unsatisfying.
So, instead of rolling your own version, use IO completion ports here. I/O completion ports have a number of additional features, and avoid potential race conditions in which two threads might pick up the same event. They also take steps to reduce context switches when your code is not CPU bound.

Wake up thread blocked on accept() call

Sockets on Linux question
I have a worker thread that is blocked on an accept() call. It simply waits for an incoming network connection, handles it, and then returns to listening for the next connection.
When it is time for the program to exit, how do I signal this network worker thread (from the main thread) to return from the accept() call while still being able to gracefully exit its loop and handle its cleanup code.
Some things I tried:
pthread_kill to send a signal. Feels kludgy to do this, plus it doesn't reliably allow the thread to do it's shutdown logic. Also makes the program terminate as well. I'd like to avoid signals if at all possible.
pthread_cancel. Same as above. It's a harsh kill on the thread. That, and the thread may be doing something else.
Closing the listen socket from the main thread in order to make accept() abort. This doesn't reliably work.
Some constraints:
If the solution involves making the listen socket non-blocking, that is fine. But I don't want to accept a solution that involves the thread waking up via a select call every few seconds to check the exit condition.
The thread condition to exit may not be tied to the process exiting.
Essentially, the logic I am going for looks like this.
void* WorkerThread(void* args)
{
DoSomeImportantInitialization(); // initialize listen socket and some thread specific stuff
while (HasExitConditionBeenSet()==false)
{
listensize = sizeof(listenaddr);
int sock = accept(listensocket, &listenaddr, &listensize);
// check if exit condition has been set using thread safe semantics
if (HasExitConditionBeenSet())
{
break;
}
if (sock < 0)
{
printf("accept returned %d (errno==%d)\n", sock, errno);
}
else
{
HandleNewNetworkCondition(sock, &listenaddr);
}
}
DoSomeImportantCleanup(); // close listen socket, close connections, cleanup etc..
return NULL;
}
void SignalHandler(int sig)
{
printf("Caught CTRL-C\n");
}
void NotifyWorkerThreadToExit(pthread_t thread_handle)
{
// signal thread to exit
}
int main()
{
void* ptr_ret= NULL;
pthread_t workerthread_handle = 0;
pthread_create(&workerthread, NULL, WorkerThread, NULL);
signal(SIGINT, SignalHandler);
sleep((unsigned int)-1); // sleep until the user hits ctrl-c
printf("Returned from sleep call...\n");
SetThreadExitCondition(); // sets global variable with barrier that worker thread checks on
// this is the function I'm stalled on writing
NotifyWorkerThreadToExit(workerthread_handle);
// wait for thread to exit cleanly
pthread_join(workerthread_handle, &ptr_ret);
DoProcessCleanupStuff();
}
Close the socket using the shutdown() call. This will wake up any threads blocked on it, while keeping the file descriptor valid.
close() on a descriptor another thread B is using is inherently hazardous: another thread C may open a new file descriptor which thread B will then use instead of the closed one. dup2() a /dev/null onto it avoids that problem, but does not wake up blocked threads reliably.
Note that shutdown() only works on sockets -- for other kinds of descriptors you likely need the select+pipe-to-self or cancellation approaches.
You can use a pipe to notify the thread that you want it to exit. Then you can have a select() call which selects on both the pipe and the listening socket.
For example (compiles but not fully tested):
// NotifyPipe.h
#ifndef NOTIFYPIPE_H_INCLUDED
#define NOTIFYPIPE_H_INCLUDED
class NotifyPipe
{
int m_receiveFd;
int m_sendFd;
public:
NotifyPipe();
virtual ~NotifyPipe();
int receiverFd();
void notify();
};
#endif // NOTIFYPIPE_H_INCLUDED
// NotifyPipe.cpp
#include "NotifyPipe.h"
#include <unistd.h>
#include <assert.h>
#include <fcntl.h>
NotifyPipe::NotifyPipe()
{
int pipefd[2];
int ret = pipe(pipefd);
assert(ret == 0); // For real usage put proper check here
m_receiveFd = pipefd[0];
m_sendFd = pipefd[1];
fcntl(m_sendFd,F_SETFL,O_NONBLOCK);
}
NotifyPipe::~NotifyPipe()
{
close(m_sendFd);
close(m_receiveFd);
}
int NotifyPipe::receiverFd()
{
return m_receiveFd;
}
void NotifyPipe::notify()
{
write(m_sendFd,"1",1);
}
Then select with receiverFd(), and notify for termination using notify().
Close the listening socket and accept will return an error.
What doesn't reliably work with this? Describe the problems you're facing.
pthread_cancel to cancel a thread blocked in accept() is risky if the pthread implementation does not implement cancellation properly, that is if the thread created a socket, just before returning to your code, a pthread_cancel() is called for it, the thread is canceled, and the newly created socket is leaked. Although FreeBSD 9.0 and later does not have such a race condition problem, but you should check your OS first.

Resources