Solutions to blocking operations inside an epoll event loop - c

Question

I have an epoll event loop in my TCP server to handle client connections and read data from clients.
while(1) {
int n, i;
n = epoll_wait(efd, events, 64, -1); // This is blocking. It waits till new events arrive
for(i = 0; i < n; i++) {
if((events[i].events & EPOLLERR) || (events[i].events & EPOLLHUP) || (!(events[i].events & EPOLLIN))) {
/* An error has occured on this fd, or the socket is not
ready for reading */
dzlog_error("epoll error: %s", strerror(errno));
close(events[i].data.fd);
continue;
} else if(sock == events[i].data.fd) { // Event on the server socket. Accept client connection
while(1) {
if((cli = accept(sock, (struct sockaddr *)&their_addr, &addr_size)) == -1) {
if((errno == EAGAIN) || (errno == EWOULDBLOCK)) { // We have processed all incoming connections
break;
} else {
dzlog_error("accept: %s", strerror(errno));
break;
}
}
dzlog_info("Client connected: Identifier - %d", cli);
s = fcntl(cli, F_SETFL, O_NONBLOCK); // Make client socket non-blocking
if(s == -1) {
dzlog_error("Client no block: %s", strerror(errno));
close(cli);
break;
}
event.data.fd = cli;
event.events = EPOLLIN | EPOLLET;
s = epoll_ctl (efd, EPOLL_CTL_ADD, cli, &event); // Add the client socket to the list of file descriptors to poll
if(s == -1) {
dzlog_error("epoll_ctl: %s", strerror(errno));
close(cli);
break;
}
}
continue;
} else {
readClientData(events[i].data.fd);
}
}
}
When there is data to be read from the client socket, the readClientData function is called. Lets assume that inside that function, we have a call to a database that fetches some data from a table. If for some reason, the call to the database hangs or takes longer than expected, other clients waiting to be connected or send data will also be blocked.
For example consider the following scenario:
Client 1 connects to server
Client 2 connects to server
Client 1 sends data to server (this will cause the readClientData function to be called to process the data)
readClientData function calls the database and waits for response. (waits for 10 seconds or might hang indefinitely)
Client 2 sends data. This data can't be processed as the server is still waiting for the readClientData to complete for Client 1
A new Client 3 tries to connect but has to wait for its connection to be accepted because server is still processing data from Client 1
Is there a way to solve this problem?
Thanks

Answer 1

You can dedicate separate process for waited operations like database read, listening on socket as well, so that you can use your event loop to check send/recv completes for your DB process as well
And Keep event loop in main process:
Read from client,write to DB handling process in nowait mode,come back to event loop to check for DB process reply or new request from client

Answer 2

Unnecessary Sequentialisation?
Assuming that your mention of readClientData() making a database query is germane; databases are quite clever, any half decent engine these days can quite happily serve more than one client at a time. If there's no particular reason why a client shouldn't talk directly to the database, having them do so would likely make for a faster system.
More Than One Database Shim Required
If that's not an option, using a process/thread to be a shim between your main event loop and the database (as desribed by Pras) is one way of keeping the event loop response time low.
However, you may as well have more than one of these, so that a client making a short request isn't held up by another client that's just made a long request. With a single shim, it'd be getting held up waiting for the long request to complete before even starting the new, short request.
This starts getting complex; you need multiple processes/threads, you have to work out a way to divide up the incoming requests amongst the shims, etc. That's a lot of code to start writing.
ZeroMQ
Fortunately there is an answer; if you use ZeroMQ for the communications between the main event loop and the database shims, you can start exploiting its patterns (= you're not writing a ton of code yourself). PUSH/PULL comes to mind; that can be used to auto-magically farm out client requests amongst the shims.
If you get the high water marks right, new client requests could "overtake" a long running request, because the long running shim simply wouldn't be given the new client request. Provided of course that the database engine itself can actually serve all the shims simultanously.