C HTTP Server & OpenSSL - Works fine for HTTP - Multiple/rapid/concurrent connections being dropped using HTTPS - c

Question

I'm writing an HTTP server in C using sockets. It can listen on multiple ports and works on a 1-thread-per-port basis to run listening loops and each loop spawns another thread to deliver a response
The code works perfectly when delivering standard HTTP responses. I have it set up to respond with an HTML page with JavaScript code that just refreshes the browser repeatedly in order to stress test the server. I've tested this with my computer running as the server and 4 other devices spamming it with requests at the same time.
No crashes, no dropped connections and no memory leaks. CPU usage never jumps beyond 5% running on a 2.0 GHz Intel Core 2 Duo in HTTP mode with 4 devices spamming requests.
I just added OpenSSL yesterday so it can deliver secure responses over HTTPS. That went fairly smoothly as it seems that all I had to do with replace some standard socket calls with their OSSL counterparts for secure mode (based on the solution to this question: Turn a simple socket into an SSL socket).
There is one SSL context and SSL struct per connection. It does work but not very reliably. Again, each response happens on its own thread but multiple/rapid/concurrent requests in secure mode are getting dropped seemingly at random, though there are still no crashes or memory leaks in my code.
When a connection is dropped the browser will either say its waiting for a response that never happens (Chrome) or just says the connection was reset (Firefox).
For reference, here is the updated connection creation and closing code.
Connection creation code (main part of the listening loop):
// Note: sslCtx and sslConnection exist
// elsewhere in memory allocated specifically
// for each connection.
struct sockaddr_in clientAddr; // memset-ed to 0 before accept
int clientAddrLength = sizeof(clientAddr);
...
int clientSocketHandle = accept(serverSocketHandle, (struct sockaddr *)&clientAddr, &clientAddrLength);
...
if (useSSL)
{
int use_cert, use_privateKey, accept_result;
sslCtx = SSL_CTX_new(SSLv23_server_method());
SSL_CTX_set_options(sslCtx, SSL_OP_SINGLE_DH_USE);
use_cert = SSL_CTX_use_certificate_file(sslCtx, sslCertificatePath , SSL_FILETYPE_PEM);
use_privateKey = SSL_CTX_use_PrivateKey_file(sslCtx, sslCertificatePath , SSL_FILETYPE_PEM);
sslConnection = SSL_new(sslCtx);
SSL_set_fd(sslConnection, clientSocketHandle);
accept_result = SSL_accept(sslConnection);
}
... // Do other things and spawn request handling thread
Connection closing code:
int recvResult = 0;
if (!useSSL)
{
shutdown(clientSocketHandle, SHUT_WR);
while (TRUE)
{
recvResult = recv(clientSocketHandle, NULL, 0, 0);
if (recvResult <= 0) break;
}
}
else
{
SSL_shutdown(sslConnection);
while (TRUE)
{
recvResult = SSL_read(sslConnection, NULL, 0);
if (recvResult <= 0) break;
}
SSL_free(sslConnection);
SSL_CTX_free(sslCtx);
}
closesocket(clientSocketHandle);
Again, this works 100% perfect for HTTP responses. What could be going wrong for HTTPS responses?
Update
I've updated the code with OpenSSL callbacks for mutli-threaded environments and the server is slightly more reliable using code from an answer to this question: OpenSSL and multi-threads.
I wrote a small command line program to spam the server with HTTPS requests and it is not dropping any connections with 5 multiple instances of it running at the same time. Multiple instances of Firefox also appear not to be dropping any connections.
What is interesting however is that connections are still being dropped with modern WebKit-based browsers. Chrome starts to drop connections at under 30 seconds of spamming, Safari on an iPhone 4 (iOS 5.1) rarely makes it past 3 refreshes before saying the connection was lost, but Safari on an iPad 2 (iOS 5.0) seems to cope the longest but ultimately ends up dropping connections as well.

Answer 1

You should call SSL_accept() in your request handling thread. This will allow your listening thread to process the TCP accept/listen queue more quickly, and reduce the chance of new connections getting a RESET from the TCP stack because of a full accept/listen queue.
SSL handshake is compute intensive. I would guess that your spammer is probably not using SSL session cache, so this causes your server to use the maximum amount of CPU. This will cause it to be CPU starved in regards to servicing the other connections, or new incoming connections.