How to set connection timeout and operation timeout in OpenSSL - c

libcurl has timeout options like these:
CURLOPT_CONNECTTIMEOUT - maximum time in seconds that you allow the connection to the server to take.
CURLOPT_TIMEOUT - maximum time in seconds that you allow the libcurl transfer operation to take.
I'd like to implement a similar timeout mechanism in OpenSSL.
What changes would be required in the code below so that a timeout value is applied to BIO_do_connect(), BIO_write() and BIO_read()?
I'm connecting to a server and sending/receiving data to/from the server using BIO_write()/BIO_read() that OpenSSL provides. My code is based on the following sample code available from here.
int main()
{
BIO * bio;
SSL * ssl;
SSL_CTX * ctx;
int p;
char * request = "GET / HTTP/1.1\x0D\x0AHost: www.verisign.com\x0D\x0A\x43onnection: Close\x0D\x0A\x0D\x0A";
char r[1024];
/* Set up the library */
ERR_load_BIO_strings();
SSL_load_error_strings();
OpenSSL_add_all_algorithms();
/* Set up the SSL context */
ctx = SSL_CTX_new(SSLv23_client_method());
/* Load the trust store */
if(! SSL_CTX_load_verify_locations(ctx, "TrustStore.pem", NULL))
{
fprintf(stderr, "Error loading trust store\n");
ERR_print_errors_fp(stderr);
SSL_CTX_free(ctx);
return 0;
}
/* Setup the connection */
bio = BIO_new_ssl_connect(ctx);
/* Set the SSL_MODE_AUTO_RETRY flag */
BIO_get_ssl(bio, & ssl);
SSL_set_mode(ssl, SSL_MODE_AUTO_RETRY);
/* Create and setup the connection */
BIO_set_conn_hostname(bio, "www.verisign.com:https");
if(BIO_do_connect(bio) <= 0)
{
fprintf(stderr, "Error attempting to connect\n");
ERR_print_errors_fp(stderr);
BIO_free_all(bio);
SSL_CTX_free(ctx);
return 0;
}
/* Check the certificate */
if(SSL_get_verify_result(ssl) != X509_V_OK)
{
fprintf(stderr, "Certificate verification error: %i\n", SSL_get_verify_result(ssl));
BIO_free_all(bio);
SSL_CTX_free(ctx);
return 0;
}
/* Send the request */
BIO_write(bio, request, strlen(request));
/* Read in the response */
for(;;)
{
p = BIO_read(bio, r, 1023);
if(p <= 0) break;
r[p] = 0;
printf("%s", r);
}
/* Close the connection and free the context */
BIO_free_all(bio);
SSL_CTX_free(ctx);
return 0;
}
I'm cross-compiling for ARM on Ubuntu (Eclipse with CodeSourcery Lite).

I ended up doing something like the following (pseudocode):
int nRet;
int fdSocket;
fd_set connectionfds;
struct timeval timeout;
BIO_set_nbio(pBio, 1);
nRet = BIO_do_connect(pBio);
if ((nRet <= 0) && !BIO_should_retry(pBio))
// failed to establish connection.
if (BIO_get_fd(pBio, &fdSocket) < 0)
// failed to get fd.
if (nRet <= 0)
{
FD_ZERO(&connectionfds);
FD_SET(fdSocket, &connectionfds);
timeout.tv_usec = 0;
timeout.tv_sec = 10;
nRet = select(fdSocket + 1, NULL, &connectionfds, NULL, &timeout);
if (nRet == 0)
// timeout has occurred.
}
You can use the same approach for BIO_read() too.
You might find this link useful.

For connecting, #jpen gave the best answer there. You have to mark the BIO as non-blocking and use select for determining whether it connected and/or timed out.
Reads are a little different. Because OpenSSL may buffer decrypted data (depending on the TLS cipher suite used), select may timeout when you are trying to read - even if data actually is available. The proper way to handle read timeouts is to first check SSL_pending or BIO_pending. If the pending function returns zero, then use select to set a timeout. If the pending function returns greater than zero, then just call SSL_read or BIO_read or any other read function.

Take a look at SSL_CTX_set_timeout () function, which does similar to libcurl's CURLOPT_TIMEOUT variable:
From http://www.openssl.org/docs/ssl/SSL_CTX_set_timeout.html :
SSL_CTX_set_timeout() sets the timeout for newly created sessions for ctx to t. The timeout value t must be given in seconds.
In your case you could add the following line after you create ctx object:
SSL_CTX_set_timeout (ctx, 60);
Hope it helps !

Related

Non-blocking BIO and hang after BIO_do_connect

I am writing a small little IRC bot in C using openssl to start a secure socket. It isn't the most beautifully written bot, but its mostly just to see how the openssl API works. Currently I have the following code:
#include <stdio.h>
#include <string.h>
#include <openssl/bio.h>
#include <openssl/ssl.h>
#include <openssl/err.h>
int main() {
SSL_load_error_strings();
ERR_load_BIO_strings();
OpenSSL_add_all_algorithms();
BIO *bio;
SSL_CTX * ctx = SSL_CTX_new(SSLv23_client_method());
SSL * ssl;
SSL_CTX_load_verify_locations(ctx, NULL, "/etc/ssl/certs/");
bio = BIO_new_ssl_connect(ctx);
BIO_get_ssl(bio, & ssl);
SSL_set_mode(ssl, SSL_MODE_AUTO_RETRY);
BIO_set_nbio(bio, 1);
BIO_set_conn_hostname(bio, "irc.freenode.net:6697");
BIO_do_connect(bio);
if(SSL_get_verify_result(ssl) != X509_V_OK) {
printf("error\n");
}
char irc1[] = "NICK bartender\r\n";
char irc2[] = "USER bartender * * :serve(&drinks);\r\n";
BIO_write(bio, irc1, strlen(irc1));
BIO_write(bio, irc2, strlen(irc2));
fd_set read_set;
int sock = BIO_get_fd(bio, NULL);
while(1) {
FD_ZERO(&read_set);
FD_SET(sock, &read_set);
struct timeval timeout = { 0, 0 };
select(sock+1, &read_set, NULL, NULL, &timeout);
if(FD_ISSET(sock, &read_set)) {
char buf[21];
size_t x = BIO_read(bio, buf, 20);
if(x == 0) {
continue;
} else if(x == -1){
int code = ERR_get_error();
if(code == 0) {
continue;
}
printf("(%d)%s\n", code, ERR_error_string(code, NULL));
} else {
buf[x] = '\0';
printf("%s", buf);
}
}
}
}
Whenever I compile and run this code, it just hangs and prints nothing. However, if I remove line 20 (which currently puts the socket into nonblocking mode) it works fine. Why does putting the socket in non-blocking mode cause this behavior? Thank you and have a great day!
Whenever I run this code, it just hangs and prints nothing. However, if I remove line 20 (which currently puts the socket into nonblocking mode) it works fine.
BIO_do_connect returns immediately in non-blocking mode. You should loop on BIO_should_retry. Here's what the man page has to say about BIO_do_connect:
BIO_do_connect() attempts to connect the supplied BIO. It returns 1 if
the connection was established successfully. A zero or negative value
is returned if the connection could not be established, the call
BIO_should_retry() should be used for non blocking connect BIOs to
determine if the call should be retried.
Why does putting the socket in non-blocking mode cause this behavior?
The call to BIO_do_connect returns immediately; the socket/bio is probably not ready for data (yet).
An alternative to looping on BIO_do_connect/BIO_should_retry is to wait on the underlying file descriptor. Its the technique used by OpenSSL in the ocsp subcommand (the source can be found in <openssl src>/apps/ocsp.c):
if (req_timeout != -1)
BIO_set_nbio(cbio, 1);
rv = BIO_do_connect(cbio);
if ((rv <= 0) && ((req_timeout == -1) || !BIO_should_retry(cbio))) {
BIO_puts(err, "Error connecting BIO\n");
return NULL;
}
if (BIO_get_fd(cbio, &fd) < 0) {
BIO_puts(bio_err, "Can't get connection fd\n");
goto err;
}
if (req_timeout != -1 && rv <= 0) {
FD_ZERO(&confds);
openssl_fdset(fd, &confds);
tv.tv_usec = 0;
tv.tv_sec = req_timeout;
rv = select(fd + 1, NULL, (void *)&confds, NULL, &tv);
if (rv == 0) {
BIO_puts(err, "Timeout on connect\n");
return NULL;
}
}
Also see Non-blocking BIO and BIO_do_connect problem on the OpenSSL Users mailing list. There's also a few hits on Stack Overflow, but I'm not sure which is the best fit for this question:
nonblocking BIO_do_connect blocked when there is no internet connected
OpenSSL connection fails with non-blocking socket
Changing an OpenSSL BIO from blocking to non-blocking mode
Unable to establish connection using OpenSSL BIO interface

How to make non-blocking OpenSSL connection?

I want make a non-blocking OpenSSL connection
On this connection - if no data available for read, then entire program execution flow make stop on SSL_read(). I want so that if no data available for read it give me the returns values like WANT_READ and i know no more data available.
char *sslRead (connection *c)
{
const int readSize = 1024;
char *rc = NULL;
int r;
int received, count = 0;
int ReallocSize = 0;
char buffer[1024];
if (c)
{
while (1)
{
if (!rc)
{
rc = malloc (readSize + 1);
if (rc == NULL)
printf("the major error have happen. leave program\n");
}
else
{
ReallocSize = (count + 1) * (readSize + 1);
rc = realloc (rc, ReallocSize);
}
// if i have no data available for read after reading data,
// this call will not return anything and wait for more data
// i want change this non blocking connections
received = SSL_read (c->sslHandle, buffer, readSize);
buffer[received] = '\0';
if (received <= 0)
{
printf(" received equal to or less than 0\n");
switch (SSL_get_error(c->sslHandle, r))
{
case SSL_ERROR_NONE:
printf("SSL_ERROR_NONE\n");
break;
case SSL_ERROR_ZERO_RETURN:
printf("SSL_ERROR_ZERO_RETURN\n");
break;
case SSL_ERROR_WANT_READ:
printf("SSL_ERROR_WANT_READ\n");
break;
default:
printf("error happens %i\n", r);
}
break;
}
count++;
}
}
return rc;
}
here is how i make connection
connection *sslConnect (void)
{
connection *c;
c = malloc (sizeof (connection));
c->sslHandle = NULL;
c->sslContext = NULL;
c->socket = tcpConnect ();
if (c->socket)
{
// Register the error strings for libcrypto & libssl
SSL_load_error_strings ();
// Register the available ciphers and digests
SSL_library_init ();
// New context saying we are a client, and using SSL 2 or 3
c->sslContext = SSL_CTX_new (SSLv23_client_method ());
if (c->sslContext == NULL)
ERR_print_errors_fp (stderr);
// Create an SSL struct for the connection
c->sslHandle = SSL_new (c->sslContext);
if (c->sslHandle == NULL)
ERR_print_errors_fp (stderr);
// Connect the SSL struct to our connection
if (!SSL_set_fd (c->sslHandle, c->socket))
ERR_print_errors_fp (stderr);
// Initiate SSL handshake
if (SSL_connect (c->sslHandle) != 1)
ERR_print_errors_fp (stderr);
}
else
{
perror ("Connect failed");
}
return c;
}
thanks you very much.
Creating a non-blocking socket is a pre-requisite to a non-blocking connect...
The following steps summarize: (see complete description in site linked below)
1) Call the fcntl() API to retrieve the socket descriptor's current flag settings into a local variable.
2) In that local variable, set the O_NONBLOCK (non-blocking) flag on. (being careful not to tamper with the other flags)
3) Call the fcntl() API to set the flags for the descriptor to the value in our local variable.
( read more on non-blocking sockets techniques here )
Assuming an existing socket, the following implements the steps outlined above:
BOOL SetSocketBlockingEnabled(SOCKET fd, BOOL blocking)
{
if (fd < 0) return FALSE;
#ifdef WIN32
unsigned long mode = blocking ? 0 : 1;
return (ioctlsocket(fd, FIONBIO, &mode) == 0) ? TRUE : FALSE;
#else
int flags = fcntl(fd, F_GETFL, 0);
if (flags < 0) return false;
flags = blocking ? (flags&~O_NONBLOCK) : (flags|O_NONBLOCK);
return (fcntl(fd, F_SETFL, flags) == 0) ? TRUE : FALSE;
#endif
}
Once you have a non-blocking socket, then see this post explaining how to do a non-blocking connect

Server with ECDHE key and cert not working

I use the below server.c source, i generated
sinful-host-cert.pem
sinful-host.key
as described here: Elliptic Curve CA Guide
When running the program get the following errors:
140722397161136:error:10071065:elliptic curve routines:func(113):reason(101):ec_lib.c:995:
140722397161136:error:0B080075:x509 certificate routines:func(128):reason(117):x509_cmp.c:346:
I compiled using:
gcc server.c -ldl -lcrypto -lssl -o Server
The error occurs at this line I think
if (SSL_CTX_use_PrivateKey_file(ctx, KeyFile, SSL_FILETYPE_PEM) <= 0)
server.c
#include <errno.h>
#include <unistd.h>
#include <malloc.h>
#include <string.h>
#include <arpa/inet.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <netinet/in.h>
#include <resolv.h>
#include "openssl/ssl.h"
#include "openssl/err.h"
#define FAIL -1
int OpenListener(int port)
{ int sd;
struct sockaddr_in addr;
sd = socket(PF_INET, SOCK_STREAM, 0);
bzero(&addr, sizeof(addr));
inet_aton("10.8.0.26", &addr.sin_addr);
addr.sin_family = AF_INET;
addr.sin_port = htons(port);
//addr.sin_addr.s_addr = INADDR_ANY;
if ( bind(sd, (struct sockaddr*)&addr, sizeof(addr)) != 0 )
{
perror("can't bind port");
abort();
}
if ( listen(sd, 10) != 0 )
{
perror("Can't configure listening port");
abort();
}
return sd;
}
SSL_CTX* InitServerCTX(void)
{ const SSL_METHOD *method;
SSL_CTX *ctx;
OpenSSL_add_all_algorithms(); /* load & register all cryptos, etc. */
SSL_load_error_strings(); /* load all error messages */
//method = SSLv23_server_method();
method = TLSv1_2_server_method(); /* create new server-method instance */
ctx = SSL_CTX_new(method); /* create new context from method */
if ( ctx == NULL )
{
ERR_print_errors_fp(stderr);
abort();
}
return ctx;
}
void LoadCertificates(SSL_CTX* ctx, char* CertFile, char* KeyFile)
{
//New lines
if (SSL_CTX_set_cipher_list(ctx, "ECDHE-ECDSA-AES128-GCM-SHA256") != 1)
ERR_print_errors_fp(stderr);
if (SSL_CTX_load_verify_locations(ctx, CertFile, KeyFile) != 1)
ERR_print_errors_fp(stderr);
if (SSL_CTX_set_default_verify_paths(ctx) != 1)
ERR_print_errors_fp(stderr);
//End new lines
/* set the local certificate from CertFile */
if (SSL_CTX_use_certificate_file(ctx, CertFile, SSL_FILETYPE_PEM) <= 0)
{
ERR_print_errors_fp(stderr);
abort();
}
printf("FFFF\n");
/* set the private key from KeyFile (may be the same as CertFile) */
if (SSL_CTX_use_PrivateKey_file(ctx, KeyFile, SSL_FILETYPE_PEM) <= 0)
{
ERR_print_errors_fp(stderr);
abort();
}
printf("GGGG\n");
/* verify private key */
if (!SSL_CTX_check_private_key(ctx))
{
fprintf(stderr, "Private key does not match the public certificate\n");
abort();
}
//New lines - Force the client-side have a certificate
//SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER | SSL_VERIFY_FAIL_IF_NO_PEER_CERT, NULL);
//SSL_CTX_set_verify_depth(ctx, 4);
//End new lines
}
void ShowCerts(SSL* ssl)
{ X509 *cert;
char *line;
cert = SSL_get_peer_certificate(ssl); /* Get certificates (if available) */
if ( cert != NULL )
{
printf("Server certificates:\n");
line = X509_NAME_oneline(X509_get_subject_name(cert), 0, 0);
printf("Subject: %s\n", line);
free(line);
line = X509_NAME_oneline(X509_get_issuer_name(cert), 0, 0);
printf("Issuer: %s\n", line);
free(line);
X509_free(cert);
}
else
printf("No certificates.\n");
}
void Servlet(SSL* ssl) /* Serve the connection -- threadable */
{ char buf[1024];
char reply[1024];
int sd, bytes, err;
const char* HTMLecho="<html><body><pre>%s</pre></body></html>\n\n";
printf("huhupre\n");
err = SSL_accept(ssl);
if ( err <= 0 ) { /* do SSL-protocol accept */
printf("%d\n",err);
ERR_print_errors_fp(stderr);
}
else
{
printf("XXXXXX\n");
//SSL_write(ssl, "huhu\n\r", 8);
ShowCerts(ssl); /* get any certificates */
bytes = SSL_read(ssl, buf, sizeof(buf)); /* get request */
if ( bytes > 0 )
{
buf[bytes] = 0;
printf("Client msg: \"%s\"\n", buf);
sprintf(reply, HTMLecho, buf); /* construct reply */
SSL_write(ssl, reply, strlen(reply)); /* send reply */
}
else
ERR_print_errors_fp(stderr);
}
sd = SSL_get_fd(ssl); /* get socket connection */
SSL_free(ssl); /* release SSL state */
close(sd); /* close connection */
}
int main()
{ SSL_CTX *ctx;
int server;
char portnum[]="5000";
char CertFile[] = "sinful-host-cert.pem";
char KeyFile[] = "sinful-host.key";
SSL_library_init();
ctx = InitServerCTX(); /* initialize SSL */
LoadCertificates(ctx, CertFile, KeyFile); /* load certs */
server = OpenListener(atoi(portnum)); /* create server socket */
printf("%d while\n", server);
while (1)
{ struct sockaddr_in addr;
socklen_t len = sizeof(addr);
SSL *ssl;
int client = accept(server, (struct sockaddr*)&addr, &len); /* accept connection as usual */
printf("Connection: %s:%d\n",inet_ntoa(addr.sin_addr), ntohs(addr.sin_port));
ssl = SSL_new(ctx); /* get new SSL state with context */
if (ssl == NULL) {
ERR_print_errors_fp(stderr);
return 0;
}
SSL_set_fd(ssl, client); /* set connection socket to SSL state */
Servlet(ssl); /* service connection */
}
close(server); /* close server socket */
SSL_CTX_free(ctx); /* release context */
}
as described here: Elliptic Curve CA Guide...
This page has so many errors and omissions I would discard it. The first red flag is the white text and black background. That tells me someone less experienced is providing the page...
From the page:
openssl ecparam -list-curves
This should be -list_curves, not -list-curves.
From the page:
openssl ecparam -out sinful.key -name sect283k1 -genkey
This should be:
openssl ecparam -param_enc named_curve -out sinful.key -name sect283k1 -genkey
If you don't use a named curve, then you will have lots of problems later, like when a client attempts to connect to the server. Here, named curve is the OID for a curve like secp256k1, and not the domain parameters like p, a, b, G, etc.
The "lots of problems later" is documented at the OpenSSL wiki Elliptic Curve Cryptography, Named Curves. Here are some of the problems you will experience:
Client: 139925962778272:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1256:SSL alert number 40
Client: 139925962778272:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:596
Server: 140339533272744:error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher:s3_srvr.c:1353
Also, for maximum interoperability, you should use secp256k1. A close second is secp521r1.
Also, use of/lack of -*form in the openssl ecparam and openssl req commands are discussed below.
SSL_CTX* InitServerCTX(void) { ... }
This code block has quite a few problems. The most notable is lack of the ECDH callback. Where are you setting the SSL_CTX_set_tmp_ecdh callback (OpenSSL 1.0.1 and below), or where is the call to SSL_CTX_set_ecdh_auto (OpenSSL 1.0.2 and above)?
Others include the default protocol, the default cipher list, weak and wounded ciphers, the inclusion of anonymous protocols, and compression. For a partial example of code to provide a server context, see 'No Shared Cipher' Error with EDH-RSA-DES-CBC3-SHA.
The error occurs at this line I think
if (SSL_CTX_use_PrivateKey_file(ctx, KeyFile, SSL_FILETYPE_PEM) <= 0)
I think that traces back to that defective page you referenced. This:
openssl req -x509 -new -key sinful.key -out sinful-ca.pem -outform PEM -days 3650
Should probably be (note the addition of -keyform)
openssl req -x509 -new -key sinful.key -keyform PEM -out sinful-ca.pem -outform PEM -days 3650
Or
if (SSL_CTX_use_PrivateKey_file(ctx, KeyFile, SSL_FILETYPE_ASN1) <= 0)
In general, always use the *form option for a command, whether its -keyform, -certform, -inform, -outform, etc. OpenSSL does not always get it right (even though its supposed to use PEM by default).
The error occurs at this line I think
if (SSL_CTX_use_PrivateKey_file(ctx, KeyFile, SSL_FILETYPE_PEM) <= 0)
If the private key has a password, then you will need to provide a Password Callback or strip the password from the file.
Its OK to strip the password because there's no difference in storing a plaintext private key; or a encrypted private key with the passphrase in a configuration file next to the key. In both cases, the only effective security you have is the filesystem ACLs.
Related, this is known as the Unattended Key Storage problem. Guttman discusses it in his book Engineering Security. Its a problem without a solution.
Here's some more complete error information... It looks like you are using an old version of OpenSSL, and that does not provide the newer error codes.
When running the program get the following errors:
140722397161136:error:10071065:elliptic curve routines:func(113):reason(101):ec_lib.c:995
140722397161136:error:0B080075:x509 certificate routines:func(128):reason(117):x509_cmp.c:346
First, the 0x10071065 error:
$ /usr/local/ssl/macosx-x64/bin/openssl errstr 0x10071065
error:10071065:elliptic curve routines:EC_POINT_cmp:incompatible objects
The 0x10071065 usually means the client and the server are using incompatible EC fields. In this case, you should use either secp256k1 or secp521r1.
Second, the 0x0B080075 error:
$ /usr/local/ssl/macosx-x64/bin/openssl errstr 0x0B080075
error:0B080075:x509 certificate routines:X509_check_private_key:unknown key type
I'm guessing that there's a mismatch in the certificate and private key. But its only a guess. I would (1) clear the named curve issue, (2) clear the sect283k1 issue, and (3) clear the down level library issue (see below). After clearing those issues, then see if this issue remains.
It looks like you are using an old version of OpenSSL, and that does not provide the newer error codes...
Be sure you are running OpenSSL 1.0.0 or above. 0.9.8 had limited EC support, but it was not really cut-in in force until 1.0.0. Better, use OpenSSL 1.0.2.
OpenSSL_add_all_algorithms(); /* load & register all cryptos, etc. */
SSL_load_error_strings(); /* load all error messages */
Also see Library Initialization on the OpenSSL wiki.
if (SSL_CTX_set_cipher_list(ctx, "ECDHE-ECDSA-AES128-GCM-SHA256")
This will get you into trouble on some versions of OS X and iOS due to a bug in the SecureTransport library. Apple only fixed it on some versions of their operating systems.
If you plan on servicing Apple hardwarez, then you will need one additional non-ECDHE-ECDSA cipher. And you need to use the server side context option SSL_OP_SAFARI_ECDHE_ECDSA_BUG.
Related, Apple is pretty bold about not fixing their security bugs. You have the broken ECDHE-ECDSA cipher suites; and gems like CVE-2015-1130 (Hidden Backdoor with Root).
Here's what my ECDH callback looks like in OpenSSL 1.0.1 and below. OpenSSL 1.0.2 should use SSL_CTX_set_ecdh_auto. Its C++ code, but its easy enough to convert back to C code. Also see SL_CTX_set_tmp_ecdh_callback semantics in 1.0.1 on the OpenSSL mailing list.
The code below could be more robust. The callback should fetch the certificate with SSL_get_certificate (not SSL_get_peer_certificate), query the certificate for the EC field, and then provide a temporary key in the appropriate field, like secp256k1 or secp571k1. (It works because my certificates use secp256, and EcdhCallback uses secp256 as its default).
SSL_get_certificate is not documented. But it is used in <openssl src>/apps/s_cb.c. That's the "self documenting" code OpenSSL is famous for.
using SSL_ptr = std::shared_ptr<SSL>;
using SSL_CTX_ptr = std::shared_ptr<SSL_CTX>;
using EC_KEY_ptr = std::unique_ptr<EC_KEY, decltype(&::EC_KEY_free)>;
using EC_GROUP_ptr = std::unique_ptr<EC_GROUP, decltype(&::EC_GROUP_free)>;
using EC_POINT_ptr = std::unique_ptr<EC_POINT, decltype(&::EC_POINT_free)>;
using EVP_PKEY_ptr = std::unique_ptr<EVP_PKEY, decltype(&::EVP_PKEY_free)>;
using BIO_MEM_ptr = std::unique_ptr<BIO, decltype(&::BIO_free)>;
using BIO_FILE_ptr = std::unique_ptr<BIO, decltype(&::BIO_free)>;
...
SSL_CTX* CreateServerContext(const string & domain)
{
const SSL_METHOD* method = SSLv23_server_method();
ASSERT(method != NULL);
SSL_CTX_ptr t(SSL_CTX_new(method), ::SSL_CTX_free);
ASSERT(t.get() != NULL);
long flags = SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3;
flags |= SSL_OP_NO_COMPRESSION;
flags |= SSL_OP_SAFARI_ECDHE_ECDSA_BUG;
flags |= SSL_OP_CIPHER_SERVER_PREFERENCE;
SSL_CTX_set_options(t.get(), flags);
string ciphers = "HIGH:!aNULL:!RC4:!MD5";
rc = SSL_CTX_set_cipher_list(t.get(), ciphers.c_str());
...
LogDebug("GetServerContext: setting ECDH callback");
SSL_CTX_set_tmp_ecdh_callback(t.get(), EcdhCallback);
...
return t.release();
}
EC_KEY* EcdhCallback(SSL *ssl, int is_export, int keylength)
{
UNUSED(ssl);
UNUSED(is_export);
UNUSED(keylength);
/* This callback is OK, but OpenSSL calls it in a broken fashion. */
/* With 1.0.1e and 1.0.1f, the value is 1024-bits. That is more */
/* appropriate for RSA.... We'll try and rewrite it here. */
if (keylength >= 1024)
{
keylength = 256;
LogRelevant("EcdhCallback: field size is wrong, using 256-bit group");
}
#if defined(ALLOW_ECDH_192_PARAMS)
if (keylength <= 192 + 4)
return ECDH192();
#endif
if (keylength <= 224 + 4)
return ECDH224();
else if (keylength <= 256 + 4)
return ECDH256();
else if (keylength <= 384 + 4)
return ECDH384();
else if (keylength <= 521 + 4)
return ECDH521();
return ECDH521();
}
#if defined(ALLOW_ECDH_192_PARAMS)
static EC_KEY* ECDH192()
{
static EC_KEY_ptr key(NULL, NULL);
static once_flag flag;
call_once(flag, []()
{
key = EC_KEY_ptr(InitEcdhkey(192), ::EC_KEY_free);
ASSERT(key.get());
if(!key.get())
LogError("ECDH192: InitEcdhkey failed");
});
return key.get();
}
#endif
static EC_KEY* ECDH224()
{
static EC_KEY_ptr key(NULL, NULL);
static once_flag flag;
call_once(flag, []()
{
key = EC_KEY_ptr(InitEcdhkey(224), ::EC_KEY_free);
ASSERT(key.get());
if(!key.get())
LogError("ECDH224: InitEcdhkey failed");
});
return key.get();
}
static EC_KEY* ECDH256()
{
static EC_KEY_ptr key(NULL, NULL);
static once_flag flag;
call_once(flag, []()
{
key = EC_KEY_ptr(InitEcdhkey(256), ::EC_KEY_free);
ASSERT(key.get());
if(!key.get())
LogError("ECDH256: InitEcdhkey failed");
});
return key.get();
}
static EC_KEY* ECDH384()
{
static EC_KEY_ptr key(NULL, NULL);
static once_flag flag;
call_once(flag, []()
{
key = EC_KEY_ptr(InitEcdhkey(384), ::EC_KEY_free);
ASSERT(key.get());
if(!key.get())
LogError("ECDH384: InitEcdhkey failed");
});
return key.get();
}
static EC_KEY* ECDH521()
{
static EC_KEY_ptr key(NULL, NULL);
static once_flag flag;
call_once(flag, []()
{
key = EC_KEY_ptr(InitEcdhkey(521), ::EC_KEY_free);
ASSERT(key.get());
if(!key.get())
LogError("ECDH521: InitEcdhkey failed");
});
return key.get();
}
static EC_KEY* InitEcdhkey(int bits)
{
if (bits <= 160 + 4)
bits = 160;
else if (bits <= 192 + 4)
bits = 192;
else if (bits <= 224 + 4)
bits = 224;
else if (bits <= 256 + 4)
bits = 256;
else if (bits <= 384 + 4)
bits = 384;
else if (bits <= 521 + 4)
bits = 521;
else
bits = 521;
EC_KEY* key = EC_KEY_new_by_curve_name(CurveToNidByBits(bits));
unsigned long err = ERR_get_error();
ASSERT(key != NULL);
if (key == NULL)
{
ostringstream oss;
oss << "InitEcdhkey: EC_KEY_new_by_curve_name failed for ";
oss << bits << "-bit key, error " << err << ", 0x" << err;
LogError(oss);
}
return key;
}

Socket performance

I just wondered about how Instant Messengers and Online Games can accept and deliver messages so fast. (Network programming with sockets)
I read about that this is done with nonblocking sockets.
I tried blocking sockets with pthreads (each client gets its own thread) and nonblocking sockets with kqueue.Then I profiled both servers with a program which made 99 connections (each connection in one thread) and then writes some garbage to it (with a sleep of 1 second). When all threads are set up, I measured in the main thread how long it took to get a connection from the server (with wall clock time) (while "99 users" are writing to it).
threads (avg): 0.000350 // only small difference to kqueue
kqueue (avg): 0.000300 // and this is not even stable (client side)
The problem is, while testing with kqueue I got multiple times a SIGPIPE error (client-side). (With a little timeout usleep(50) this error was fixed). I think this is really bad because a server should be capable to handle thousands of connections. (Or is it my fault on the client side?) The crazy thing about this is the infamous pthread approach did just fine (with and without timeout).
So my question is: how can you build a stable socket server in C which can handle thousands of clients "asynchronously"? I only see the threads approach as a good thing, but this is considered bad practice.
Greetings
EDIT:
My test code:
double get_wall_time(){
struct timeval time;
if (gettimeofday(&time,NULL)){
// Handle error
return 0;
}
return (double)time.tv_sec + (double)time.tv_usec * .000001;
}
#define NTHREADS 100
volatile unsigned n_threads = 0;
volatile unsigned n_writes = 0;
pthread_mutex_t main_ready;
pthread_mutex_t stop_mtx;
volatile bool running = true;
void stop(void)
{
pthread_mutex_lock(&stop_mtx);
running = false;
pthread_mutex_unlock(&stop_mtx);
}
bool shouldRun(void)
{
bool copy;
pthread_mutex_lock(&stop_mtx);
copy = running;
pthread_mutex_unlock(&stop_mtx);
return copy;
}
#define TARGET_HOST "localhost"
#define TARGET_PORT "1336"
void *thread(void *args)
{
char tmp = 0x01;
if (__sync_add_and_fetch(&n_threads, 1) == NTHREADS) {
pthread_mutex_unlock(&main_ready);
fprintf(stderr, "All %u Threads are ready...\n", (unsigned)n_threads);
}
int fd = socket(res->ai_family, SOCK_STREAM, res->ai_protocol);
if (connect(fd, res->ai_addr, res->ai_addrlen) != 0) {
socket_close(fd);
fd = -1;
}
if (fd <= 0) {
fprintf(stderr, "socket_create failed\n");
}
if (write(fd, &tmp, 1) <= 0) {
fprintf(stderr, "pre-write failed\n");
}
do {
/* Write some garbage */
if (write(fd, &tmp, 1) <= 0) {
fprintf(stderr, "in-write failed\n");
break;
}
__sync_add_and_fetch(&n_writes, 1);
/* Wait some time */
usleep(500);
} while (shouldRun());
socket_close(fd);
return NULL;
}
int main(int argc, const char * argv[])
{
pthread_t threads[NTHREADS];
pthread_mutex_init(&main_ready, NULL);
pthread_mutex_lock(&main_ready);
pthread_mutex_init(&stop_mtx, NULL);
bzero((char *)&hint, sizeof(hint));
hint.ai_socktype = SOCK_STREAM;
hint.ai_family = AF_INET;
if (getaddrinfo(TARGET_HOST, TARGET_PORT, &hint, &res) != 0) {
return -1;
}
for (int i = 0; i < NTHREADS; ++i) {
pthread_create(&threads[i], NULL, thread, NULL);
}
/* wait for all threads to be set up */
pthread_mutex_lock(&main_ready);
fprintf(stderr, "Main thread is ready...\n");
{
double start, end;
int fd;
start = get_wall_time();
fd = socket(res->ai_family, SOCK_STREAM, res->ai_protocol);
if (connect(fd, res->ai_addr, res->ai_addrlen) != 0) {
socket_close(fd);
fd = -1;
}
end = get_wall_time();
if (fd > 0) {
fprintf(stderr, "Took %f ms\n", (end - start) * 1000);
socket_close(fd);
}
}
/* Stop all running threads */
stop();
/* Waiting for termination */
for (int i = 0; i < NTHREADS; ++i) {
pthread_join(threads[i], NULL);
}
fprintf(stderr, "Performed %u successfull writes\n", (unsigned)n_writes);
/* Lol.. */
freeaddrinfo(res);
return 0;
}
SIGPIPE comes when I try to connect to the kqueue server (after 10 connections are made, the server is "stuck"?). And when too many users are writing stuff, the server cannot open a new connection. (kqueue server code from http://eradman.com/posts/kqueue-tcp.html)
SIGPIPE means you're trying to write to a socket (or pipe) where the other end has already been closed (so noone will be able to read it). If you don't care about that, you can ignore SIGPIPE signals (call signal(SIGPIPE, SIG_IGN)) and the signals won't be a problem. Of course the write (or send) calls on the sockets will still be failing (with EPIPE), so you need to make you code robust enough to deal with that.
The reason that SIGPIPE normally kills the process is that its too easy to write programs that ignore errors on write/send calls and run amok using up 100% of CPU time otherwise. As long as you carefully always check for errors and deal with them, you can safely ignore SIGPIPEs
Or is it my fault?
It was your fault. TCP works. Most probably you didn't read all the data that was sent.
And when too many users are writing stuff, the server cannot open a new connection
Servers don't open connections. Clients open connections. Servers accept connections. If your server stops doing that, there something wrong with your accept loop. It should only do two things: accept a connection, and start a thread.

recvfrom() error 10035 using non-blocking sockets

I am using ioctlsocket() function to make my socket non-blocking but when I call recvfrom(), I get the error 10035 (WSAEWOULDBLOCK).
u_long mode = 1;
ioctlsocket(newSocketIdentifier, FIONBIO, &mode);
while(1)
{
if((recv_len = recvfrom(newSocketIdentifier, receiveBuffer, sizeof(receiveBuffer), 0, (struct sockaddr *) &clientSocket, &clientSocketLength)) == SOCKET_ERROR)
{
char err[128];
itoa(WSAGetLastError(),err,10);
MessageBox( NULL,"Could not Receive Data",err,MB_ICONINFORMATION);
BREAK;
}
}
Can anybody explain why this happens? :(
This is normal if no data is available. The code is WSAEWOULDBLOCK (see this table) and means, that on a blocking port the function would have to sit and wait until it could be served.
while(1)
{
if((recv_len = recvfrom(newSocketIdentifier, receiveBuffer, sizeof(receiveBuffer), 0, (struct sockaddr *) &clientSocket, &clientSocketLength)) == SOCKET_ERROR)
{
int ierr= WSAGetLastError();
if (ierr==WSAEWOULDBLOCK) { // currently no data available
Sleep(50); // wait and try again
continue;
}
// Other errors
char err[128];
itoa(ierr,err,10);
MessageBox( NULL,"Could not Receive Data",err,MB_ICONINFORMATION);
break;
}
}
I cannot agree that this is "normal" like posted above.
In your call of recvfrom you will receive an error in recv_len. I recommend to check that value - it will be SOCKET_ERROR and by calling WSAGetLastErrorenter you will see error WSAEWOULDBLOCK.
I am not a (Windows) socket expert, but based on my tests, I cannot use combination of ioctlsocket and recvfrom for receiving data via UDP in non-blocking mode (I did the same thing like you in your example).
I am planning to use combination of select and recvfrom with minimum possible timeout (1us). I do not know any other possibly better solution now.
Note: you should check also the return value of ioctlsocket for possible error.
I will provide my code sample later today.
UPDATE (adding code as promised):
/* define list of sockets for function select(..) */
fd_set readfds;
/* define timeout for function select(..) */
TIMEVAL tv;
/* timeout: 1us */
tv.tv_usec = 1;
/* timeout: 0s */
tv.tv_sec = 0;
/* just 1 socket is used */
readfds.fd_count = 1;
readfds.fd_array[0] = receivingSocket;
/* determine the status of one or more sockets with timeout */
int selectReturnValue = select(0, &readfds, 0, 0, &tv);
/* check return value of the call of function select(..) */
switch (selectReturnValue)
{
/* select(..) function timeout */
case 0:
/* time limit expired */
break;
/* select(..) function error */
case SOCKET_ERROR:
/* check the error status for the last windows sockets operation */
selectError(WSAGetLastError());
break;
/* no timeout and no error */
default:
/* receive data from UDP */
resultOfrecvfrom = recvfrom(receivingSocket, receivingBuffer, sizeof(receivingBuffer), 0, (SOCKADDR *)&serverReceptionInfo, &serverReceptionInfoLength);
/* check result of call of recvfrom(..) */
switch (resultOfrecvfrom)
{
/* connection has been gracefully closed */
case 0:
/* socket was closed */
break;
/* socket error occurred during last call of socket operation */
case SOCKET_ERROR:
/* check the error status for the last Windows Sockets operation */
recvfromError(WSAGetLastError());
break;
/* resultOfrecvfrom amount of data received */
default:
/* ... add your code here */
break;
}
break;
}
I just copied crucial part of my code if more is needed, let me know in comments.

Resources