Convert code using openssl to use mbedtls - c

Question

I am trying to convert some C code written to run on a Mac to an embedded device that does not have any encryption libraries. The code for the Mac is using libcrypto. I tried building libcrypto from openssl sources for the embedded device but I get hundreds of errors due to function pointer prototypes not matching. openssl is riddled with huge macros. As an alternative I am now trying to use mbedtls but I have not been able to get a decrypt function to work.
The code I am trying to port is a bit odd. It has what it calls a public key and is actually calling RSA_public_encrypt() with no padding to decrypt data. As a test, I changed the Mac code to call RSA_public_decrypt() and it worked so I assume the key is symmetric. The key it is using looks like this:
"-----BEGIN PUBLIC KEY-----\n"
5 lines of Base64 strings
"-----END PUBLIC KEY-----\n"
For mbedtls I am using mbedtls_pk_parse_public_key() to parse the key. If I inspect the low level RSA key structure after parsing the key there is a 128 byte N component and a 16 byte E component. I get the same key data with both openssl and mbedtls so it appears that the key is parsed properly. When decrypting on the Mac with RSA_public_decrypt(), the input and output are both 128 bytes. For mbedtls, I am calling mbedtls_pk_decrypt() to decrypt but when I trace through the code it calls mbedtls_rsa_rsaes_pkcs1_v15_decrypt() which forces the padding to be 11 bytes.
So my questions are: 1) exactly what kind of "public" key contains only N and E components and uses no padding; 2) Am I calling the correct mbedtls decryption function?
EDIT: Tried another approach and my output buffer just gets filled with zeros.
mbedtls_rsa_context rsa;
mbedtls_rsa_init(&rsa, MBEDTLS_RSA_PKCS_V15, 0);
mbedtls_rsa_import_raw(&rsa, modulus, sizeof(modulus), NULL, 0, NULL, 0, NULL, 0, exp, sizeof(exp));
mbedtls_rsa_complete(&rsa);
mbedtls_rsa_public(&rsa, inBfr, outBfr);
mbedtls_rsa_free(&rsa);
EDIT 2: My ultimate target is an embedded device with an ARM processor but I was testing on Windows to see if mbedtls would work. I started with VS 2010 because that is what was being used for the project I am working on. I switched to VS 2015 and the 2nd approach of importing the raw key data and calling mbedtls_rsa_public() worked perfectly. I guess the VS 2010 compiler just isn't good enough. I then ported the code to the devlepment system for my embedded device and it also worked.

Answer 1

The equivalent of OpenSSL's RSA_public_encrypt(…, RSA_NO_PADDING) would be mbedtls_rsa_public. The function mbedtls_pk_encrypt only lets you access encryption mechanisms based on RSA (RSAES-PKCS1-v1_5 and RSAES-OAEP), not the raw RSA primitive (“textbook RSA” a.k.a. “RSA with no padding”).
You need to call mbedtls_pk_parse_public_key to parse the key, then get a pointer to the RSA key object with mbedtls_pk_rsa, and call mbedtls_rsa_public using this RSA key object.
mbedtls_pk_context pk;
mbedtls_pk_init(&pk);
mbedtls_rsa_context *rsa = mbedtls_pk_rsa(&pk);
mbedtls_rsa_public(&rsa, inBfr, outBfr);
mbedtls_pk_free(&pk);
I think this should work from your description, but obviously I can't test this without sample data.