c - WSAWaitForMultipleObjects blocking any thread but last - c

i have a problem with a multi-thread SMTP/POP3 server. The server starts a pool of threads to handle incoming connections. The main thread create the sockets and the the threads, passing the sockets as parameters in a proper structure. The loop function for the threads is the following:
SOCKET SMTP_ListenSocket = (SOCKET) data->SMTPconn;
SOCKET POP3_ListenSocket = (SOCKET) data->POP3conn;
static struct sockaddr_in ClntAddr;
unsigned int clntLen = sizeof(ClntAddr);
hEvents[0] = CreateEvent(NULL, FALSE, FALSE, NULL);
hEvents[1] = CreateEvent(NULL, FALSE, FALSE, NULL);
hEvents[2] = exitEvent; //HANDLE FOR A MANUAL RESET EVENT
WSAEventSelect(SMTP_ListenSocket, hEvents[0], FD_ACCEPT);
WSAEventSelect(POP3_ListenSocket, hEvents[1], FD_ACCEPT);
while(1){
DWORD res = WaitForMultipleObjects(3, hEvents, FALSE, INFINITE);
switch(res){
case WAIT_OBJECT_0: {
ClientSocket = my_accept(SMTP_ListenSocket,(struct sockaddr *) &ClntAddr,&clntLen);
/* ... */
my_shutdown(ClientSocket,2);
my_closesocket(ClientSocket);
ClientSocket = INVALID_SOCKET;
break;
}
case WAIT_OBJECT_0 + 1: {
ClientSocket = my_accept(POP3_ListenSocket,(struct sockaddr *) &ClntAddr,&clntLen);
/* ... */
my_shutdown(ClientSocket,2);
my_closesocket(ClientSocket);
ClientSocket = INVALID_SOCKET;
break;
}
case WAIT_OBJECT_0 + 2:
{
exitHandler(0);
break;
}
}//end switch
}//end while
When the pool contains only one thread there's no problem. When the pool consist of more threads, only one thread accepts the incoming connections

Do you have the pooled threads all calling this same code? If so, then don't use WaitForMultipleObjects() (or WSAWaitForMultipleEvents()) like this. This kind of model only works reliably if one thread is polling connections. If you have multiple threads polling at the same time, then you have race conditions.
Instead, you should use AcceptEx() with Overlapped I/O or Completion Ports instead. The thread that creates the sockets can call AcceptEx() on each socket to queue a new operation on each one, then the pooled threads can use GetQueuedCompletionStatus() or GetOverlappedResult() to dequeue a pending connection without worrying about trampling on other threads. Once a connection is accepted, the receiving thread can process it as needed and then call AcceptEx() to queue a new operation for that socket.

Each thread here is setting a new WSAEventSelect prior to entering the wait. This overwrites any existing event selects. This means that, once a thread (call it thread A) accepts a connection, there is no event associated with the socket.
To solve this, you should call WSAEventSelect again within your switch, immediately after the accept(). This will restore the event binding immediately before going into any potentially lengthy processing.
Note that it's possible that two threads may be awoken for the same event, if the timing works out just right. You can hack around that by going back to your wait loop if the accept fails, but this is a bit unsatisfying.
So, instead of rolling your own version, use IO completion ports here. I/O completion ports have a number of additional features, and avoid potential race conditions in which two threads might pick up the same event. They also take steps to reduce context switches when your code is not CPU bound.

Related

one thread to exit them all

I have a main program that generates a few threads (using a while loop with accept() to get clients), and one that all it has to do is "listen to the keyboard" and when the user enters the word exit it will close the entire program.
first, the main program create the listening thread, then it enters a while loop that accept the clients. even if the listening thread get the exit input the loop is still stuck on accept.
i don't have to use a seperate thread to listen to the keyboard but i could'nt find a none blocking way that would work.
the listening thread:
DWORD WINAPI ListenService(LPVOID lpParam)
{
char buffer[5];
if (EOF == scanf("%s", buffer))
{
printf("faile get word from keyboard\n");
}
if (buffer[4] != '\0')
strcat(buffer, "\0");
if (STRINGS_ARE_EQUAL(buffer, "exit"))
{
return 999;
}
return -1;
}
in the main code:
ThreadListen = CreateThread(NULL,0,ListenService,NULL,0,&(ThreadId));
while(1)
{
SOCKET AcceptSocket = accept(MainSocket, NULL, NULL);
if (AcceptSocket == INVALID_SOCKET)
{
printf("Accepting connection with client failed, error %ld\n", WSAGetLastError());
CleanupWorkerThreads();
WSACleanup();
}
printf("Client Connected.\n");
}
There are many different ways you can handle this.
You can abort a blocked accept() by simply closing the listening socket.
Or, you can use select() with a short timeout to detect when a new client is waiting before then calling accept(). You can check your exit condition in between calls to select(). Just be aware that there is a small race condition where a client may disconnect between the time select() and accect() are called, so accept() may still block, if there are no more clients waiting.
Or, you can get rid of your threads and just use non-blocking sockets in a single thread, checking your exit condition periodically in between socket operations.
Or, you can use asynchronous sockets, using WSACreateEvent(), WSAEventSelect(), and WSAWaitForMultipleEvents() to detect socket activity. Then you can create an addition event to wait on for when the exit condition happens.
Or, you can use an I/O Completion Port to handle socket activity, and then you can post a custom exit packet into the IOCP queue using PostQueuedCompletionStatus() to "wake up" any waiting threads.

how to restart socket properly in a multithread c/c++ program

Background: My code structure: I have a master socket on main thread, then each time a new client is coming, the threadpool will be notified and let one pre allocated thread take the task.
Inside this thread, I will pass a slave socket to it, and let it using accept call to listen to the client.
Scenario: In my thread pool, thread A is listening to a client right now, now I want to stop all the pre-allocated thread and close all the connection to the client, the main thread is trying to close the connection using close the connection to the client, and trying to terminate thread A using pthread_join.
main() {
// create threadpool
// logic to create mastersocket
startServer(masterSock)
IwantToCloseServer() // this function is not directly called in main, but simulated by a terminal signal , like kill -quit pid.
}
int startServer(int msock) {
int ssock; // slaveSocket
struct sockaddr_in client_addr; // the address of the client...
unsigned int client_addr_len = sizeof(client_addr); // ... and its length
while (!stopCondition) {
// Accept connection:
ssock = ::accept((int)msock, (struct sockaddr*)&client_addr, &client_addr_len); // the return value is a socket
// I was trying to replace this line of code to poll(), but it's not does the same thing as before
if (ssock < 0) {
if (errno == EINTR) continue;
perror("accept");
running =0;
return 0;
// exit(0);
} else {
// push task to thread pool to deal with logic
}
// main thread continues with the loop...
}
return 1;
}
IwantToCloseServer(slaveSocket) {
// when i want to close() or shutdown() function to close connections, these 2 function always return -1, because the thread is blocked on accept call
// logic try to terminate all the preallocated threads, the pthread_join function is stuck because the thread is blocked on accept
}
Problem: The thread A is keeping blocking on the ::accept function , the close and shutdown function return -1, they won’t close the connection , and the pthread_join is not keep going because thread A is blocked on accept.
Things I tried:
I have try to change my while loop related accept function, for example, set a flag stopCondition,
while(!stopConditon) {
ssock = ::accept((int)msock, (struct sockaddr*)&client_addr, &client_addr_len);
}
However, when the main thread change stopCondtion, the thread A is blocked inside the accept function.
It won’t go inside the while loop, so this solution won’t affect the accept function, it’s not working
I have also tried to send a signal to this blocked Thread A, using
pthread_cancel or pthread_kill(Thread A, 9)
However, if I do this, the whole process gets killed.
3.try to use poll() to replace the line, where the accept functions at, with a timeout
however, the program doesn't behave like before, the program can't listen to client anymore.
How do I terminate thread A (which is blocked on accept function call right now), so that I can clean this pre allocated thread and restart my server ?
btw i can not use library like boost in my current program. And this is under linux system not winsocket
to check periodically stopConditon in your while(!stopConditon) { first call accept/pool with a timeout to know if there is something new about msock, then depending on the result call accept etc else do nothing
I was trying to replace this line of code to poll()
try to use poll() to replace the line, where the accept functions at, with a timeout
you cannot replace accept by poll, you have to call accept / pool first and of course check the result then may be call accept
Out of that
while(!stopConditon) {
if(!stopCondtion) {
is redundant and can be replaced by
while(!stopConditon) {

How to optimize C non-blocking accept() socket connection

I have an app (server) that uses accept() to receive TCP socket connections from mobile phones (clients). It seems to work, but when I run my app in Apple Instruments I can see this app uses about 100% CPU and the main reason 95% of it accounts for call to non-blocking accept(). I think I have added this non-blocking accept() method call in order to be able to stop server in any time by clicking GUI button "Stop Server". By this solution I can each time check whether I have a flag shouldShutDown equal to true and then I am closing the server()
Here is the code of server loop:
/**
* Function is looping infinitely and waiting
* for new incoming connections.
* It handles each connection on thread pool's worker thread
*/
result_t thread_pool_stream_server_loop(server_info_t *server_info, connection_handler_t conn_handler) {
sock_fd_t cs_fd, ps_fd;
thread_pool_t *thread_pool;
// get passive server socket
ps_fd = server_info_sock(server_info);
// initialize thread pool and set its size
thread_pool_init(&thread_pool, 5, 10, 3000 /* [ms] */);
while(1) {
if(server_info_should_shut_down(server_info)) {
thread_pool_shutdown(thread_pool);
return CLOSED;
}
if(server_info_force_shut_down(server_info)) {
thread_pool_shutdown(thread_pool);
return FORCE_CLOSED;
}
// check to accept new connections on the main thread...
cs_fd = accept_new_connection(ps_fd);
if(cs_fd == FAILURE) {
fprintf(stderr, "accept_new_connection: failed!\n");
server_info_connection_error_event(server_info, cs_fd, CONN_ERROR_ACCEPT, "accept_new_connection: failed!");
break;
} else if(cs_fd == CONTINUE) {
continue;
}
// publish client connected event
server_info_client_connected_event(server_info, cs_fd);
// revert connection socket to non-blocking
int opts = fcntl(cs_fd, F_GETFL);
opts = opts & (~O_NONBLOCK);
fcntl(cs_fd, F_SETFL, opts);
// handle new connection by thread pool's worker thread
conn_thread_runner_attr_t *connection_thread_runner_attr;
conn_thread_runner_attr_init(&connection_thread_runner_attr);
conn_thread_runner_attr_fill(connection_thread_runner_attr, conn_handler, server_info, cs_fd, NULL, NULL);
thread_pool_run(thread_pool, (runner_t) connection_thread_runner, (runner_attr_t) connection_thread_runner_attr, NULL);
// adjust thread pool actual size to number of added connection handling tasks
thread_pool_adjust_size(thread_pool);
}
thread_pool_force_free(thread_pool);
return FAILURE;
}
The code is busy waiting, for accept() to return something different then EAGAIN/EWOULDBLOCK, so what else then burning CPU cycles do you expect?
You could fix this by calling select(), with 10ms time-out, before calling accept().

How to handle 100,000 requests concurrently in C

I am expecting to get 100,000 requests at least 100 of them would be concurrent. Each time I get a request, I am creating a new thread and destroy it once it's done using pthread_exit(). Using pthread_detach I am getting 99% success rate. Is there a better way than this?
pthread_t hilo;
// infinite loop
while ((client_sock = accept(server_sock, (struct sockaddr *) &client_sockaddr, &len))) {
struct ClientSocket socks;
// some code...
pthread_create(&hilo, NULL, func, &socks);
pthread_detach(hilo);
printf("\nSocket is listening for the next request...\n");
}
I heard pthread_join would be a better way to utilize resources without reaching the thread limit, but the way I am doing it is not concurrent.
pthread_t hilo;
// infinite loop
while ((client_sock = accept(server_sock, (struct sockaddr *) &client_sockaddr, &len))) {
struct ClientSocket socks;
// some code...
pthread_create(&hilo, NULL, func, &socks);
pthread_join(hilo, NULL); // it stops the main thread
printf("\nSocket is listening for the next request...\n");
}
Any ideas would be appreciated!
Each time I get a request, I am creating a new thread and destroy it once it's done
Don't: it's not a smart thing to do, because thread creation is a heavy-weight operation.
I heard pthread_join would be a better way to utilize resources without reaching the thread limit, but the way I am doing it is not concurrent.
Correct. Whenever you see someone doing pthread_create immediately followed by pthread_join, you can tell that they have no idea what they are doing. The exact same result could be achieved (much more efficiently) by simply calling func directly.
Any ideas would be appreciated!
You need a producer-consumer queue. The listening thread will accept connections and enqueue them for other threads in a (size-limited) thread pool. The other threads will dequeue one item of work, and perform func on that item. Then go back for next item, and so on.
Here is an example producer/consumer implementation. But you can find many others with "producer consumer queue" search.

Signal to print message every 20 seconds (C Multithreaded program)

I am trying to print a message every 20 seconds in a multithreaded program. I have a server that has two threads. One thread waits for incoming connections and makes a new thread for the client when it connects.
I have looked at this: C: SIGALRM - alarm to display message every second but with my code, I'm not sure where I would put the loop. I am not allowed to make a new thread or use sleep() or any variation of sleep().
Code for server acceptor thread:
while((csock = accept(sock, (struct sockaddr *) &theClient, (socklen_t *) &cl)))
{
pthread_t newClient;
new_sock = malloc(sizeof(socket_t));
*new_sock = csock;
pthread_create(&newClient, NULL, getInput, (void *) new_sock)
}
The other thread is just handling the client's input. I tried putting the loop inside the above loop but then it never accepts new connections.
How would I go about doing this? Any help would be appreciated.
Your problem seems to be currently that accept() is blocking the thread until a new connection comes in; therefore you can't print anything.
You could use non-blocking accept() in a loop to check for connections, and in the same loop wait until 20 seconds has passed. Note that this is very inefficient as this loop doesn't stop; it uses 100% of 1 cpu core.
// sock is listening
fcntl(sock,F_SETFD,O_NONBLOCK);
time_t tm = time(); // Unix timestamp, in seconds.
while(true) {
csock = accept(sock, (sockaddr*)&theClient, (socklen_t*)&cl);
if (csock==-1) {
if (errno==EAGAIN || errno==EWOULDBLOCK); // No connection at the
// moment, we need to try
// again later.
else break; // Some other error occurred
}
else { // If it is connected
pthread_t newClient;
new_sock = malloc(sizeof(socket_t));
*new_sock = csock;
pthread_create(&newClient,NULL,getInput,(void*)new_sock);
}
if (time()-tm>=20) { // At least 20 seconds have elapsed since
// last message.
printf("Hello World!!\n");
tm = time(); // Start waiting for another 20 seconds.
}
}
Using select() to wait for a new connection would be far more efficient - you can also set a timeout which expires so that you can print your message
Edit: You don't want to use a signal, because, as it says in the article you linked, you can't use printf from inside a signal handler.
If you use the signal handler to set a flag, you won't be able to read the flag unless you use non-blocking accept() (because otherwise accept() could block for a minute but nothing prints).

Resources