RSA sign in OPENSSL - c

Question

I am trying to sign a message with a RSA private key. I read a private key to pkey and then sign a string as what says on openssl wiki, but failed in the final step. It always returns 0 on the line commented in program, which means sign failed. Could anyone helps me find out what’s wrong?
void main() {
EVP_MD_CTX * mdctx ;
EVP_PKEY * pkey ;
char dmessage[20] = "The messages";
int ret = 0;
FILE * fp;
unsigned char * sig = NULL;
size_t * slen = malloc(sizeof(size_t));
fp = fopen ("privkey.pem", "r");
if (fp == NULL) exit (1);
pkey = PEM_read_PrivateKey(fp, NULL, NULL, NULL);
fclose (fp);
if (pkey == NULL) {
ERR_print_errors_fp (stderr);
exit (1);
}
if(!(mdctx = EVP_MD_CTX_create())) goto err;
if(1 != EVP_DigestSignInit(mdctx, NULL, EVP_sha256(), NULL, pkey)) goto err;
if(1 != EVP_DigestSignUpdate(mdctx, dmessage, 12)) goto err;
if(1 != EVP_DigestSignFinal(mdctx, NULL, slen)) goto err;
if(!(sig = OPENSSL_malloc(sizeof(unsigned char) * (int)(*slen)))) goto err;
if(1 != (ret = EVP_DigestSignFinal(mdctx, sig, slen))) goto err;//*****it return 0 here,which means sign failed
ret = 1;
err:
if(ret != 1)
{
printf("%d somthing wrong\n",ret);
}
/* Clean up */
if(sig && !ret) OPENSSL_free(sig);
if(mdctx) EVP_MD_CTX_destroy(mdctx);
return;
}
Thanks a lot!
I’m using openssl 1.0.1j on linux mint 17, and the private key is generated by
openssl genrsa -out privkey.pem 256

Answer 1

Your key is way too small, that's bits not bytes. Try again with a good secure key size that can hold the hash and PKCS#1 padding. I would recommend at least 2048 bits instead of the 256 bits that you generated using the OpenSSL command line.
See keylength.com for more information about key sizes. Note that RSA requires a key size a lot larger than those required for symmetric algorithms such as AES.