Segmentation fault with generating an RSA and saving in ASN.1/DER? - c

Question

#include <string.h>
#include <openssl/aes.h>
#include <openssl/rand.h>
#include <openssl/bio.h>
#include <openssl/rsa.h>
#include <openssl/evp.h>
#include <openssl/pem.h>
#define RSA_LEN 2048
#define RSA_FACTOR 65537
int genRSA2048(unsigned char **pub,unsigned int *pub_l,unsigned char **priv,unsigned int *priv_l){
RSA *pRSA = NULL;
pRSA = RSA_generate_key(RSA_LEN,RSA_FACTOR,NULL,NULL);
if (pRSA){
pub_l = malloc(sizeof(pub_l));
*pub_l = i2d_RSAPublicKey(pRSA,pub);
priv_l = malloc(sizeof(priv_l));
*priv_l = i2d_RSAPrivateKey(pRSA,priv);
return 0;
} else {
return 1;
}
}
int main(){
unsigned char *pub = NULL;
unsigned int publ;
unsigned char *priv = NULL;
unsigned int privl;
genRSA2048(&pub,&publ,&priv,&privl);
RSA *privrsa = NULL;
d2i_RSAPrivateKey(&privrsa,(const unsigned char **)&priv,privl);
RSA *pubrsa = NULL;
d2i_RSAPublicKey(&pubrsa,(const unsigned char **)&pub,publ);
unsigned char * data ="01234567890123456789012345678912";
unsigned char encrypted[256];
unsigned char decrypted[32];
int len = RSA_private_encrypt(32,data,encrypted,privrsa,RSA_PKCS1_PADDING);
RSA_public_decrypt(len,encrypted,decrypted,pubrsa,RSA_PKCS1_PADDING);
}
I've tried to find the bug by checking with gdb but as being fairly new to C I haven't find any clue to tell me what is happening but I believe it's an allocation problem, however according to the d2i_RSAPrivateKey and similar, they're supposed to allocate the space by itself.
Any help would be greatly appreciated.
compiled as cc foo.c -lcrypto
This is the follow up of this question:
Generate RSA public/private key with OpenSSL?
As I reference I used #WhozCraig example in the comments, which can be found here, even when it's quite different, it was a lot of help.
http://coliru.stacked-crooked.com/a/ae64a70076436165

Answer 1

pub_l = malloc(sizeof(pub_l)); is simply not needed. Nor is priv_l = malloc(sizeof(priv_l));. Remove them both from your function.
You should be populating your out-parameters; instead you're throwing out the caller's provided addresses to populate and (a) populating your own, then (b) leaking the memory you just allocated.
The result is the caller's privl and publ are untouched and thus the decoding back to RSA is dysfunctional, as both values are indeterminate.

Answer 2

pRSA = RSA_generate_key(RSA_LEN,RSA_FACTOR,NULL,NULL);
I think this is wrong. I know you are supposed to use RSA_generate_key_ex, and I think it needs a BIGNUM, not an integer. You should have gotten a warning. See RSA_generate_key(3) for details.
Your code should look something like:
BIGNUM* exp = BN_new();
ASSERT(exp != NULL);
int rc = BN_set_word(exp, RSA_F4);
ASSERT(rc == 1);
RSA* rsa = RSA_new();
ASSERT(rsa != NULL);
rc = RSA_generate_key_ex(rsa, 2048, exp, NULL);
ASSERT(rc == 1);
Be sure to call BN_free on the BIGNUM, and RSA_free on the RSA pointer.
RSA *privrsa = NULL;
d2i_RSAPrivateKey(&privrsa,(const unsigned char **)&priv,privl);
RSA *pubrsa = NULL;
d2i_RSAPublicKey(&pubrsa,(const unsigned char **)&pub,publ);
For this, it looks like you are trying to separate the public key and private key. For that, use RSAPublicKey_dup and RSAPrivateKey_dup. See Separating public and private keys from RSA keypair variable.
Its not clear to me what you are trying to do with the following. You should state what you are trying to do...
pub_l = malloc(sizeof(pub_l));
*pub_l = i2d_RSAPublicKey(pRSA,pub);
priv_l = malloc(sizeof(priv_l));
*priv_l = i2d_RSAPrivateKey(pRSA,priv);
I'm just guessing, but I'm going to say its all wrong. sizeof(priv_l) is the size of a pointer, so its 4 or 8 bytes. You're also overwriting the pointer passed in by the caller...
Also see OpenSSL's rsautl cannot load public key created with PEM_write_RSAPublicKey. It talks about saving the keys with SubjectPublicKeyInfo and PrivateKeyInfo in both ASN.1/DER and PEM formats.
By writing the {Public|Private}KeyInfo, the OID gets written to the key. That's important for interop. You also use the RSA* (and even an EVP_PKEY*), and not byte arrays.