I have an app (server) that uses accept() to receive TCP socket connections from mobile phones (clients). It seems to work, but when I run my app in Apple Instruments I can see this app uses about 100% CPU and the main reason 95% of it accounts for call to non-blocking accept(). I think I have added this non-blocking accept() method call in order to be able to stop server in any time by clicking GUI button "Stop Server". By this solution I can each time check whether I have a flag shouldShutDown equal to true and then I am closing the server()
Here is the code of server loop:
/**
* Function is looping infinitely and waiting
* for new incoming connections.
* It handles each connection on thread pool's worker thread
*/
result_t thread_pool_stream_server_loop(server_info_t *server_info, connection_handler_t conn_handler) {
sock_fd_t cs_fd, ps_fd;
thread_pool_t *thread_pool;
// get passive server socket
ps_fd = server_info_sock(server_info);
// initialize thread pool and set its size
thread_pool_init(&thread_pool, 5, 10, 3000 /* [ms] */);
while(1) {
if(server_info_should_shut_down(server_info)) {
thread_pool_shutdown(thread_pool);
return CLOSED;
}
if(server_info_force_shut_down(server_info)) {
thread_pool_shutdown(thread_pool);
return FORCE_CLOSED;
}
// check to accept new connections on the main thread...
cs_fd = accept_new_connection(ps_fd);
if(cs_fd == FAILURE) {
fprintf(stderr, "accept_new_connection: failed!\n");
server_info_connection_error_event(server_info, cs_fd, CONN_ERROR_ACCEPT, "accept_new_connection: failed!");
break;
} else if(cs_fd == CONTINUE) {
continue;
}
// publish client connected event
server_info_client_connected_event(server_info, cs_fd);
// revert connection socket to non-blocking
int opts = fcntl(cs_fd, F_GETFL);
opts = opts & (~O_NONBLOCK);
fcntl(cs_fd, F_SETFL, opts);
// handle new connection by thread pool's worker thread
conn_thread_runner_attr_t *connection_thread_runner_attr;
conn_thread_runner_attr_init(&connection_thread_runner_attr);
conn_thread_runner_attr_fill(connection_thread_runner_attr, conn_handler, server_info, cs_fd, NULL, NULL);
thread_pool_run(thread_pool, (runner_t) connection_thread_runner, (runner_attr_t) connection_thread_runner_attr, NULL);
// adjust thread pool actual size to number of added connection handling tasks
thread_pool_adjust_size(thread_pool);
}
thread_pool_force_free(thread_pool);
return FAILURE;
}
The code is busy waiting, for accept() to return something different then EAGAIN/EWOULDBLOCK, so what else then burning CPU cycles do you expect?
You could fix this by calling select(), with 10ms time-out, before calling accept().
Related
I have a main program that generates a few threads (using a while loop with accept() to get clients), and one that all it has to do is "listen to the keyboard" and when the user enters the word exit it will close the entire program.
first, the main program create the listening thread, then it enters a while loop that accept the clients. even if the listening thread get the exit input the loop is still stuck on accept.
i don't have to use a seperate thread to listen to the keyboard but i could'nt find a none blocking way that would work.
the listening thread:
DWORD WINAPI ListenService(LPVOID lpParam)
{
char buffer[5];
if (EOF == scanf("%s", buffer))
{
printf("faile get word from keyboard\n");
}
if (buffer[4] != '\0')
strcat(buffer, "\0");
if (STRINGS_ARE_EQUAL(buffer, "exit"))
{
return 999;
}
return -1;
}
in the main code:
ThreadListen = CreateThread(NULL,0,ListenService,NULL,0,&(ThreadId));
while(1)
{
SOCKET AcceptSocket = accept(MainSocket, NULL, NULL);
if (AcceptSocket == INVALID_SOCKET)
{
printf("Accepting connection with client failed, error %ld\n", WSAGetLastError());
CleanupWorkerThreads();
WSACleanup();
}
printf("Client Connected.\n");
}
There are many different ways you can handle this.
You can abort a blocked accept() by simply closing the listening socket.
Or, you can use select() with a short timeout to detect when a new client is waiting before then calling accept(). You can check your exit condition in between calls to select(). Just be aware that there is a small race condition where a client may disconnect between the time select() and accect() are called, so accept() may still block, if there are no more clients waiting.
Or, you can get rid of your threads and just use non-blocking sockets in a single thread, checking your exit condition periodically in between socket operations.
Or, you can use asynchronous sockets, using WSACreateEvent(), WSAEventSelect(), and WSAWaitForMultipleEvents() to detect socket activity. Then you can create an addition event to wait on for when the exit condition happens.
Or, you can use an I/O Completion Port to handle socket activity, and then you can post a custom exit packet into the IOCP queue using PostQueuedCompletionStatus() to "wake up" any waiting threads.
Background: My code structure: I have a master socket on main thread, then each time a new client is coming, the threadpool will be notified and let one pre allocated thread take the task.
Inside this thread, I will pass a slave socket to it, and let it using accept call to listen to the client.
Scenario: In my thread pool, thread A is listening to a client right now, now I want to stop all the pre-allocated thread and close all the connection to the client, the main thread is trying to close the connection using close the connection to the client, and trying to terminate thread A using pthread_join.
main() {
// create threadpool
// logic to create mastersocket
startServer(masterSock)
IwantToCloseServer() // this function is not directly called in main, but simulated by a terminal signal , like kill -quit pid.
}
int startServer(int msock) {
int ssock; // slaveSocket
struct sockaddr_in client_addr; // the address of the client...
unsigned int client_addr_len = sizeof(client_addr); // ... and its length
while (!stopCondition) {
// Accept connection:
ssock = ::accept((int)msock, (struct sockaddr*)&client_addr, &client_addr_len); // the return value is a socket
// I was trying to replace this line of code to poll(), but it's not does the same thing as before
if (ssock < 0) {
if (errno == EINTR) continue;
perror("accept");
running =0;
return 0;
// exit(0);
} else {
// push task to thread pool to deal with logic
}
// main thread continues with the loop...
}
return 1;
}
IwantToCloseServer(slaveSocket) {
// when i want to close() or shutdown() function to close connections, these 2 function always return -1, because the thread is blocked on accept call
// logic try to terminate all the preallocated threads, the pthread_join function is stuck because the thread is blocked on accept
}
Problem: The thread A is keeping blocking on the ::accept function , the close and shutdown function return -1, they won’t close the connection , and the pthread_join is not keep going because thread A is blocked on accept.
Things I tried:
I have try to change my while loop related accept function, for example, set a flag stopCondition,
while(!stopConditon) {
ssock = ::accept((int)msock, (struct sockaddr*)&client_addr, &client_addr_len);
}
However, when the main thread change stopCondtion, the thread A is blocked inside the accept function.
It won’t go inside the while loop, so this solution won’t affect the accept function, it’s not working
I have also tried to send a signal to this blocked Thread A, using
pthread_cancel or pthread_kill(Thread A, 9)
However, if I do this, the whole process gets killed.
3.try to use poll() to replace the line, where the accept functions at, with a timeout
however, the program doesn't behave like before, the program can't listen to client anymore.
How do I terminate thread A (which is blocked on accept function call right now), so that I can clean this pre allocated thread and restart my server ?
btw i can not use library like boost in my current program. And this is under linux system not winsocket
to check periodically stopConditon in your while(!stopConditon) { first call accept/pool with a timeout to know if there is something new about msock, then depending on the result call accept etc else do nothing
I was trying to replace this line of code to poll()
try to use poll() to replace the line, where the accept functions at, with a timeout
you cannot replace accept by poll, you have to call accept / pool first and of course check the result then may be call accept
Out of that
while(!stopConditon) {
if(!stopCondtion) {
is redundant and can be replaced by
while(!stopConditon) {
I need to perform some operations only after the time a TCP connection is fully closed, that's to say - all the data segments, as well as the finishing routine (FIN-ACK or RST) have been performed and done, and no packets will be sent on the wires.
Since closesocket() is not synchronous and could return before a full close of the connection and socket, I've used the SO_LINGER option to get the moment of closing.
According to the instructions in the MSDN for closesocket, and to the fact that my socket is non-blocking (and thus asynchronous), I wrote this code:
int ret;
/* config 2 secs of linger */
struct linger lng = {1, 2};
setsockopt(s, SOL_SOCKET, SO_LINGER, (const char*)&lng, sizeof(lng));
/* graceful close of TCP (FIN-ACK for both sides) */
shutdown(s, SD_BOTH);
/* linger routine for asynchronous sockets according to msdn */
do {
ret = closesocket(s);
} while (ret == SOCKET_ERROR && WSAGetLastError() == WSAEWOULDBLOCK);
/* my code to be run immediately after all the traffic */
printf("code to run after closing");
However, the closesocket call returns zero (success; instead of getting into the loop) and I see in Wireshark that my final printing is called before all the packets were sent, so - it looks like the linger isn't working.
By the way, the functions I used to open and connect the asynchronous socket were socket(), WSAIoctl() and its lpfnConnectEx() callback.
What's the reason that the lingered closesocket return before a full finish of the TCP connection? Is there a solution?
i have a problem with a multi-thread SMTP/POP3 server. The server starts a pool of threads to handle incoming connections. The main thread create the sockets and the the threads, passing the sockets as parameters in a proper structure. The loop function for the threads is the following:
SOCKET SMTP_ListenSocket = (SOCKET) data->SMTPconn;
SOCKET POP3_ListenSocket = (SOCKET) data->POP3conn;
static struct sockaddr_in ClntAddr;
unsigned int clntLen = sizeof(ClntAddr);
hEvents[0] = CreateEvent(NULL, FALSE, FALSE, NULL);
hEvents[1] = CreateEvent(NULL, FALSE, FALSE, NULL);
hEvents[2] = exitEvent; //HANDLE FOR A MANUAL RESET EVENT
WSAEventSelect(SMTP_ListenSocket, hEvents[0], FD_ACCEPT);
WSAEventSelect(POP3_ListenSocket, hEvents[1], FD_ACCEPT);
while(1){
DWORD res = WaitForMultipleObjects(3, hEvents, FALSE, INFINITE);
switch(res){
case WAIT_OBJECT_0: {
ClientSocket = my_accept(SMTP_ListenSocket,(struct sockaddr *) &ClntAddr,&clntLen);
/* ... */
my_shutdown(ClientSocket,2);
my_closesocket(ClientSocket);
ClientSocket = INVALID_SOCKET;
break;
}
case WAIT_OBJECT_0 + 1: {
ClientSocket = my_accept(POP3_ListenSocket,(struct sockaddr *) &ClntAddr,&clntLen);
/* ... */
my_shutdown(ClientSocket,2);
my_closesocket(ClientSocket);
ClientSocket = INVALID_SOCKET;
break;
}
case WAIT_OBJECT_0 + 2:
{
exitHandler(0);
break;
}
}//end switch
}//end while
When the pool contains only one thread there's no problem. When the pool consist of more threads, only one thread accepts the incoming connections
Do you have the pooled threads all calling this same code? If so, then don't use WaitForMultipleObjects() (or WSAWaitForMultipleEvents()) like this. This kind of model only works reliably if one thread is polling connections. If you have multiple threads polling at the same time, then you have race conditions.
Instead, you should use AcceptEx() with Overlapped I/O or Completion Ports instead. The thread that creates the sockets can call AcceptEx() on each socket to queue a new operation on each one, then the pooled threads can use GetQueuedCompletionStatus() or GetOverlappedResult() to dequeue a pending connection without worrying about trampling on other threads. Once a connection is accepted, the receiving thread can process it as needed and then call AcceptEx() to queue a new operation for that socket.
Each thread here is setting a new WSAEventSelect prior to entering the wait. This overwrites any existing event selects. This means that, once a thread (call it thread A) accepts a connection, there is no event associated with the socket.
To solve this, you should call WSAEventSelect again within your switch, immediately after the accept(). This will restore the event binding immediately before going into any potentially lengthy processing.
Note that it's possible that two threads may be awoken for the same event, if the timing works out just right. You can hack around that by going back to your wait loop if the accept fails, but this is a bit unsatisfying.
So, instead of rolling your own version, use IO completion ports here. I/O completion ports have a number of additional features, and avoid potential race conditions in which two threads might pick up the same event. They also take steps to reduce context switches when your code is not CPU bound.
I have a fairly basic TCP server keeping track of a couple connections and recv'ing data when it's available. However, I'd like to artificially trigger an event from within the program itself, so I can send my TCP server data as if it came from sock1 or sock2, but in reality came from somewhere else. Is this possible, or at all clear?
struct pollfd fds[2];
fds[0].fd = sock1;
fds[1].fd = sock2;
while (true) {
int res = poll(fds, 2, timeout);
if ((fds[0].revents & POLLIN)){
//ready to recv data from sock1
}
if ((fds[1].revents & POLLIN)){
//ready to recv data from sock2
}
}
Create a pair of connected sockets (see socketpair(2)), and wait for events on one of the sockets in your poll loop. When you want to wake up the poll thread, write a single byte to the other socket. When the polling loop wakes up, read the byte, do whatever was required and continue.
This is more like a design question -- your polling loop should probably abstract the poll method to allow trapping on other external signals, like from kill -USR1.
If you really want to trigger port traffic, you'll likely want to use netcat to send data to the socket.
I would consider something like this:
struct pollfd fds[2];
fds[0].fd = sock1;
fds[0].events = POLLIN;
fds[1].fd = sock2;
fds[1].events = POLLIN;
for (;;) {
int result = poll(fds, 2, timeout);
if (result) {
if ((fds[0].revents & POLLIN)){
/* Process data from sock1. */
}
if ((fds[1].revents & POLLIN)){
/* Process data from sock2. */
}
} else {
/* Do anything else you like, including
processing data that wasn't from a
real socket. */
}
}
Notes:
don't forget to initialise your events field
for(;;) is more idiomatic C than while(true) and doesn't require true to be defined