OpenSSL: certificate in memory not getting validated properly - c

Question

I'm hoping someone can help with my OpenSSL(1.1.1k) issue I'm having when trying to validate a certificate on the client against a server. The certificate contents are specified in text and not in a .PEM file on the client side. Some sample code is here:
bool setCert(SSL_CTX* ctx, LPCSTR cert)
{
//cert is the text of the certificate e.g. -----BEGIN CERTIFICATE----- blah -----END CERTIFICATE-----
bool bOk = true;
if (!cert.empty()) // no cert on disk
{
BIO *bioCert = BIO_new_mem_buf((void*)cert, -1);
X509* pCert = PEM_read_bio_X509(bioCert, NULL, 0, NULL);
if (SSL_CTX_use_certificate(ctx, pCert) != 1)
{
//log errror, "Invalid certificate"
bOk = false;
}
BIO_free_all(bioCert);
X509_free(pCert);
}
SSL_CTX_set_options(ctx, SSL_OP_NO_SSLv2|SSL_OP_NO_SSLv3); // other options not used atm SSL_OP_NO_TLSv1 | SSL_OP_NO_TLSv1_1 | SSL_OP_NO_TLSv1_2
SSL_CTX_set_session_cache_mode(ctx, SSL_SESS_CACHE_OFF);
return bOk;
}
After this method returns, I create an SSL object based on the context, with SSL_new(ctx).
I then call SSL_Connect(ssl)
The handshake occurs and is accepted even if I pass in an invalid certificate.
My assumption here is that it's doing this because I do not set the verify method like:
SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, NULL);
My understanding is that I cannot use SSL_VERIFY_PEER unless there is a cert / .PEM file on disk on the client side to validate against.
How can I validate the handshake when keeping the certificate contents in memory?
If I change the code to use a PEM file, call SSL_CTX_use_certificate_file instead of SSL_CTX_use_certificate, it works. In that case I did set SSL_CTX_set_verify(ctx, SSL_VERIFY_PEER, NULL);
Any help here is greatly appreciated. Thanks

Answer 1

SSL_CTX_use_certificate is used for setting up the certificate which should be used locally to authenticate against the peer. It needs to be accompanied with a private key, setup with SSL_CTX_use_PrivateKey. But authenticating against a peer, i.e. using a client certificate, is not what you intend to do in your client.
Instead you want to setup a certificate for use as a trusted CA when validating the server certificate. The correct function for this would be X509_STORE_add_cert instead:
X509_STORE_add_cert(SSL_CTX_get_cert_store(ctx), pCert);