Openssl custom extension callback functions - c

Question

I am creating a custom extension using the OpenSSL custom extension API.
The functions SSL_CTX_add_client_custom_ext and SSL_CTX_custom_ext return 1 i.e. success but the issue is there are certain callback functions which get called to operate on the data we need to add or parse. I added certain debug statements to find out whether they get called or not and I think they don't.
static int old_add_cb(SSL *s, unsigned int ext_type, const unsigned
char **out, size_t *outlen, int *al, void *add_arg) {
printf("called!!");
return 1;
}
static void old_free_cb(SSL *s, unsigned int ext_type, const unsigned
char *out, void *add_arg) {
printf("called!!");
OPENSSL_free((unsigned char *)out);
}
static int old_parse_cb(SSL *s, unsigned int ext_type, const
unsigned char *in, size_t inlen, int *al, void *parse_arg) {
printf("called!!");
return 1;
}
The SSL_CTX related code is:
int main(int count, char *strings[]) {
SSL_CTX *ctx;
int server;
SSL *ssl;
char buf[1024];
int bytes;
char *hostname, *portnum;
if ( count != 3 ) {
printf("usage: %s <hostname> <portnum>\n", strings[0]);
exit(0);
}
SSL_library_init();
hostname=strings[1];
portnum=strings[2];
ctx = InitCTX();
int result = SSL_CTX_add_custom_ext(ctx, 1000,
SSL_EXT_CLIENT_HELLO, old_add_cb,
old_free_cb, NULL, old_parse_cb,
NULL);
printf("Extension Register %d", result);
server = OpenConnection(hostname, atoi(portnum));
ssl = SSL_new(ctx); /* create new SSL connection state */
SSL_set_fd(ssl, server); /* attach the socket descriptor */
if ( SSL_connect(ssl) == FAIL ) /* perform the connection */
ERR_print_errors_fp(stderr);
else { char *msg = "Hello???";
printf("Connected with %s encryption\n", SSL_get_cipher(ssl));
ShowCerts(ssl); /* get any certs */
SSL_write(ssl, msg, strlen(msg)); /* encrypt & send message */
bytes = SSL_read(ssl, buf, sizeof(buf)); /* get reply & decrypt */
buf[bytes] = 0;
printf("Received: \"%s\"\n", buf);
SSL_free(ssl); /* release connection state */
}
close(server); /* close socket */
SSL_CTX_free(ctx); /* release context */
return 0;
}
The 'SSL_CTX_add_custom_ext' function returns 1 but the print statements in callback functions are not being executed.

Answer 1

From Openssl doc about SSL_extension_supported
we can see the following statements:
For the ServerHello and EncryptedExtension messages every registered add_cb is called once if and only if the requirements of the specified context are met and the corresponding extension was received in the ClientHello. That is, if no corresponding extension was received in the ClientHello then add_cb will not be called.
I mean, the callbacks from both side(here is client and server) will execute only if server verify and accept the ClientHello which includes extensions. So you should add extension(here callback) to server like client to make sure callback to be executed. Here is my example:
static int ext_add_cb(SSL *s, unsigned int ext_type,
const unsigned char **out,
size_t *outlen, int *al, void *add_arg)
{
switch (ext_type) {
case 65280:
printf("ext_add_cb from client called!\n");
break;
default:
break;
}
return 1;
}
static void ext_free_cb(SSL *s, unsigned int ext_type,
const unsigned char *out, void *add_arg)
{
printf("ext_free_cb from client called\n");
}
static int ext_parse_cb(SSL *s, unsigned int ext_type,
const unsigned char *in,
size_t inlen, int *al, void *parse_arg)
{
printf("ext_parse_cb from client called!\n");
return 1;
}
server is similar to client. And then add register in main:
int result = SSL_CTX_add_client_custom_ext(ctx, 65280, ext_add_cb, ext_free_cb, NULL, ext_parse_cb, NULL);
Run server and then run client, I got this message:
# server:
ext_parse_cb from server called!
ext_add_cb from server called!
ext_free_cb from server called!
# client:
ext_add_cb from client called!
ext_free_cb from client called
ext_parse_cb from client called!