Function is not generating proper openssl rsa keys - c

Question

This is a c function I wrote to generate openssl rsa 4096 bit keys.
bool rsa_gen_keys()
{
int ret = 0;
RSA *rsa = NULL;
BIGNUM *bignum = NULL;
BIO *bio_private = NULL;
BIO *bio_public = NULL;
int bits = 4096;
unsigned long k = RSA_F4;
bignum = BN_new();
ret = BN_set_word(bignum,k);
if(ret != 1){
goto cleanup;
}
rsa = RSA_new();
ret = RSA_generate_key_ex(rsa, bits, bignum, NULL);
if(ret != 1){
goto cleanup;
}
// write rsa private key to file
bio_private = BIO_new_file("private_new.pem", "w+");
ret = PEM_write_bio_RSAPrivateKey(bio_private, rsa, NULL, NULL, 0, NULL, NULL);
BIO_flush(bio_private);
// write rsa public key to file
bio_public = BIO_new_file("public_new.pem", "w+");
ret = PEM_write_bio_RSAPublicKey(bio_public, rsa);
if(ret != 1){
goto cleanup;
}
BIO_flush(bio_public);
cleanup:
BIO_free_all(bio_private);
BIO_free_all(bio_public);
RSA_free(rsa);
BN_free(bignum);
return ret;
}
The keys generated by the above function seem to be missing something. When I try to use the public_new.pem file in another program, I get the following error:
140286309791384:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: PUBLIC KEY
However, if I use the openssl command to generate the key files, the files work fine.
$openssl genpkey -algorithm RSA -out private_key.pem -pkeyopt rsa_keygen_bits:4096
I noticed the key sizes generated from the function and from the command line don't match. This is a clue, but what do I need to change in my function to fix this?
-rw-rw-r-- 1 3272 Feb 6 09:19 private_key.pem
-rw-rw-r-- 1 800 Feb 6 09:20 public_key.pem
-rw-rw-r-- 1 3243 Feb 6 10:43 private_new.pem
-rw-rw-r-- 1 775 Feb 6 10:43 public_new.pem
BTW, I tried the above with 2048 bits keys and I get the same result and same size mismatch

Answer 1

openssl genpkey is using PEM_write_bio_PrivateKey (PKCS#8) instead of PEM_write_bio_RSAPrivateKey (PKCS#1): https://github.com/openssl/openssl/blob/master/apps/genpkey.c#L161-L164.
You don't show how you generated public_key.pem, but it was probably written with PEM_write_bio_PUBKEY (X.509 SubjectPublicKeyInfo) vs PEM_write_bio_RSAPublicKey (PKCS#1).
From a PEM armor perspective:
PKCS#1 Public: BEGIN RSA PUBLIC KEY
X.509 SubjectPublicKeyInfo: BEGIN PUBLIC KEY
PKCS#1 Private: BEGIN RSA PRIVATE KEY
PKCS#8: BEGIN PRIVATE KEY

Answer 2

I realized I needed to use the format of the keys for PKCS#8 and X.509. So I switched to EVP functions to generate them. Here is a very simplified version of the code I ended up using (no error checking):
bool rsa_gen_keys() {
int ret = 0;
BIO *bio_private = NULL;
BIO *bio_public = NULL;
int bits = 4096;
EVP_PKEY_CTX *ctx;
EVP_PKEY *pkey = NULL;
// Get the context
ctx = EVP_PKEY_CTX_new_id(EVP_PKEY_RSA, NULL);
if (!ctx)
goto cleanup;
// init keygen
if (EVP_PKEY_keygen_init(ctx) <= 0)
goto cleanup;
// set the bit size
if (EVP_PKEY_CTX_set_rsa_keygen_bits(ctx, bits) <= 0)
goto cleanup;
/* Generate key */
if (EVP_PKEY_keygen(ctx, &pkey) <= 0)
goto cleanup;
// write rsa private key to file
bio_private = BIO_new_file("private_new.pem", "w+");
ret = PEM_write_bio_PrivateKey(bio_private, pkey, NULL, NULL, 0, NULL, NULL);
if (ret != 1) {
goto cleanup;
}
BIO_flush(bio_private);
// write rsa public key to file
bio_public = BIO_new_file("public_new.pem", "w+");
//ret = PEM_write_bio_RSAPublicKey(bio_public, rsa);
ret = PEM_write_bio_PUBKEY(bio_public, pkey);
if (ret != 1) {
goto cleanup;
}
BIO_flush(bio_public);
cleanup:
if(bio_private) BIO_free_all(bio_private);
if(bio_public) BIO_free_all(bio_public);
if(pkey) EVP_PKEY_free(pkey);
return ret;
}