Related
I'm trying to write AES encryption/decryption program in C using openssl. However, when I tried to decrypt the message, I got error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block length.
void aes_encrypt(unsigned char* in, int inl, unsigned char *out, int* len, unsigned char * key){
unsigned char iv[16] = "encryptionIntVec";
EVP_CIPHER_CTX ctx;
EVP_CIPHER_CTX_init(&ctx);
EVP_EncryptInit_ex(&ctx, EVP_aes_256_cbc(), NULL, key, iv);
*len = 0;
int outl = 0;
EVP_EncryptUpdate(&ctx, out+*len, &outl, in+*len, inl);
*len+=outl;
int test = inl>>4;
if(inl != test<<4){
EVP_EncryptFinal_ex(&ctx,out+*len,&outl);
*len+=outl;
}
EVP_CIPHER_CTX_cleanup(&ctx);
}
void aes_decrypt(unsigned char* in, int inl, unsigned char *out, unsigned char *key){
unsigned char iv[16] = "encryptionIntVec";
EVP_CIPHER_CTX ctx;
EVP_CIPHER_CTX_init(&ctx);
int result;
result = EVP_DecryptInit_ex(&ctx, EVP_aes_256_cbc(), NULL, key, iv);
if(result > 0) {
printf("passed\n");
} else {
printf("failed\n");
}
int len = 0;
int outl = 0;
result = EVP_DecryptUpdate(&ctx, out+len, &outl, in+len, inl);
if(result > 0) {
printf("passed\n");
} else {
printf("failed\n");
}
len += outl;
result = EVP_DecryptFinal_ex(&ctx, out+len, &outl);
if(result > 0) {
printf("passed\n");
} else {
printf("failed\n");
ERR_print_errors_fp(stdout);
}
len+=outl;
out[len]=0;
EVP_CIPHER_CTX_cleanup(&ctx);
}
int main()
{
unsigned char content[400];
unsigned char key[] = "0123456789abcdef0123456789abcdef";
/******************Block 1*****************************/
unsigned char en[400],de[400],base64[400], base64_out[400];
int len;
memset(content, 0,400);
memset(en, 0, 400);
memset(de, 0, 400);
memset(base64, 0,400);
memset(base64_out, 0, 400);
strcpy((char *)content, "loc: 123.2132412, -39.123142");
printf("%d %s\n", strlen((const char*)content), content);
//encrypt content
aes_encrypt(content,strlen((const char*)content), en, &len, (unsigned char*)key);
//base64 encode ciphertext
int encode_str_size = EVP_EncodeBlock(base64, en, len);
printf("%d %s\n", encode_str_size, base64);
//base64 decode
int length = EVP_DecodeBlock(base64_out, base64, strlen((const char*)base64));
while(base64[--encode_str_size] == '=') length--;
//decrypt
aes_decrypt(base64_out, length, de, (unsigned char*)key);
printf("%d %s\n", strlen((const char*)de), de);
/***********************Block 2*******************************/
unsigned char msg_out[400];
unsigned char msg[400] = "6hKe8RGg+4p1N1R6Y9aaTovxLtuH115JoWUO8plrAJE=";
unsigned char result[400];
int l = EVP_DecodeBlock(msg_out, msg, strlen((const char*)msg));
if(strcmp((const char*)msg, (const char*)base64)==0) {
printf("match\n");
}
if(strcmp((const char*)en, (const char*)msg_out)==0) {
printf("match\n");
}
while(msg[--encode_str_size] == '=') l--;
aes_decrypt(msg_out, l, result, (unsigned char*)key);
printf("%d %s\n", strlen((const char*)result), result);
return 0;
}
In block 1 of main function, I encrypted, base64 encoded, base64 decoded, decrypted, then got the exactly the same String as it was, and produced NO error.
However, in the block 2, I directly used the base64 encoded string which was produced from block 1, decoded and decypted, but an error occurred at result = EVP_DecryptFinal_ex(&ctx, out+len, &outl); that was error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block length.
If I printed the decrypted string anyways, it is loc: 123.2132412, -39.123142^D^D^D^D, there are four '^D' appended at the end. I compared strings, both match was printed, which means the ciphertexts that was decoded from base64 with the one(en) in block 1 were the same. (NOTE: same key, same IV)
Any ideas why it failed?
I am using version OpenSSL 3.0.0-dev and therefore had to modify the code for the reasons described in David C. Rankin's comment. The problem occurs even after these changes.
Debugging reveals that the length of the Base64 decoded data in block 2 is determined incorrectly with l = 0x21 = 33. The correct value is l = 0x20 = 32.
This is because in block 2, when determining the length of the Base64 decoded data without the 0x00 padding bytes (added by EVP_DecodeBlock()) with
while (msg[--encode_str_size] == '=') l--;
the value for encode_str_size already changed in block 1 is used. The fix is to apply the current value for the length of the Base64 encoded data, e.g.
encode_str_size = strlen((const char*)msg);
Then decryption for block 2 works on my machine.
Note that EVP_DecodeBlock() always returns the Base64 decoded data with a length equal to an integer multiple of 3, padded with 0x00 values if necessary.
The actual length of the data can therefore also be determined with msg_out and l alone. l specifies the length including the 0x00 padding bytes, so that only the number of the 0x00 padding bytes has to be subtracted from l.
Thus, msg and encode_str_size are actually not needed (although both can be used, since the number of = padding bytes is equal to the number of 0x00 padding bytes).
As already mentioned in the comment by Joël Hecht, the EVP_EncryptFinal_ex() call is necessary, which among other things performs the padding (PKCS7).
In the currently posted code, this only occurs if the length of the plaintext does not equal an integer multiple of 16 bytes (the AES block size) which is satisfied for the test data.
However, for a plaintext whose length is a multiple of 16 bytes, EVP_EncryptFinal_ex() is not called, so padding is not performed, resulting in a decryption failure in EVP_DecryptFinal_ex().
Apart from the Base64 encoding, the OpenSSL documentation here provides a complete example for EVP_aes_256_cbc(), which could be used for comparison.
I'm porting an AES encryption procedure from Javascript to C.
I'm using OpenSSL.
For the encryption I'm using ECB 256 mode.
The encryption goes successfully but the final Base64 does not correspond to the result I get in JS.
By testing with several online tools, the output I was getting from the C version was actually correct but different from the one from JS.
Then I realized the problem thanks to this tool https://gchq.github.io/CyberChef.
The key I'm providing is a MD5 hash like this:
aa8744256a89cba0d77f0686916f8b2e
If I pass the key as an HEX string in the CyberChef tool, the output corresponds to the JS version.
The C version is reading the key as an UTF string.
How can I force it to read the key as an HEX string?
Here's the code:
//generating MD5 of key
char keyHash[33];
unsigned char digest[MD5_DIGEST_LENGTH];
MD5((unsigned char*)keyString, strlen(keyString), (unsigned char*)&digest);
for (int i = 0; i < 16; i++)
sprintf(&keyHash[i * 2], "%02x", (unsigned int)digest[i]);
char out[256];
int outlen = 0;
//initializing context
EVP_CIPHER_CTX* ctx = EVP_CIPHER_CTX_new();
//inizializing encryption with 256 ECB Cipher and MD5 Hash as Key
EVP_EncryptInit(ctx, EVP_aes_256_ecb(), keyHash, 0);
//setting padding
EVP_CIPHER_CTX_set_padding(ctx, EVP_PADDING_PKCS7);
//encrypting
EVP_EncryptUpdate(ctx, out, &outlen, user, strlen(user));
EVP_EncryptFinal(ctx, out, &outlen);
EVP_CIPHER_CTX_free(ctx);
//parsing the result to Base64
BIO *mbio = BIO_new(BIO_s_mem());
BIO *b64bio = BIO_new(BIO_f_base64());
BIO_set_flags(b64bio, BIO_FLAGS_BASE64_NO_NL);
BIO *bio = BIO_push(b64bio, mbio);
BIO_write(bio, out, outlen);
BIO_flush(bio);
char* data = NULL;
size_t datalen = 0;
datalen = BIO_get_mem_data(mbio, &data);
data[datalen] = '\0';
//printing the result
printf("encrypted data: [%s]\n", data);
I have to generate JWT token and so i found some online resource on using OpenSSL dll (libcrypto-1_1.dll) for the same.
This method is used to get the rsa key
void *getRSAKey(const char *file1, int type, const char *password)
{
void * file = NULL;
void *rsa;
file = (void*)BIO_new_file(file1, "r");
if (file == NULL) return 0;
if (type == 0) {
rsa=(void*)d2i_RSAPrivateKey_bio(file, NULL);
} else if (type == 1) {
if(password)
rsa=(void*)PEM_read_bio_RSAPrivateKey(file, NULL, NULL, (void*)password);
else
rsa=(void*)PEM_read_bio_RSAPrivateKey(file, NULL, NULL, NULL);
}
if (rsa == NULL){
lr_message("ras NULL ");
}
BIO_free(file);
return(rsa);
}
First I have generated the header and payload and done base24 encoding on them. I am them concatenating these with a seperator "." and saving in input
input = (unsigned char*)calloc(strlen(base24encodedHeader) + strlen(base24encodedPayload) + 1,sizeof(unsigned char));
strcpy((char*)input,base24encodedHeader);
strcat(input, ".");
strcat((char*)input,base24encodedPayload);
Then this input is being passed to RSA_sign to get the sign
private_key = (void*)getRSAKey(private_key_file, X509_FILETYPE_PEM, "abc");
sign = (unsigned char*)calloc(RSA_size(private_key), sizeof(unsigned char));
RSA_sign(NID_sha256,input, strlen((char*)input), sign, &sign_len, private_key);
printf("private_key = %d",RSA_size(private_key));
printf("signed len = %d\n", sign_len);
printf("signed string = %s\n", sign);
finally i am concatenating input with sign
jwt= (char *)calloc(strlen(input) + strlen(sign) + 2,sizeof(unsigned char));
strcpy(jwt,input);
strcat(jwt,".");
strcat(jwt,sign);
printf("%s",jwt);
however the in the returned jwt string the "sign" is empty. And RSA_Sign is throwing error digest too big
private_key = 512
input size = 761
Signed len = 5
signed string =
signed error = error:04075070:rsa routines:RSA_sign:digest too big for rsa key
The length of input string (base64) was 761 bytes, RSA signing for
SHA256 it should be 256 bits = 32 bytes, so tahts why it was throwing the error "too big"
So i needed basically to first generate a SHA256 hash of the input string before passing it to RSA_sign.
I want to use AES-128-cbc encryption/decryption algorithm to encrypt/decrypt some data,and use nodejs to encrypt the data and use c to decrypt them.But found that using the same key and IV,the two languages have different encryption results.See follows:
nodes js code:
var crypto = require('crypto');
var encrypt = function (key, iv, data) {
var cipher = crypto.createCipheriv('aes-128-cbc', key, iv);
var crypted = cipher.update(data, 'utf8', 'binary');
var cbase64 = new Buffer(crypted, 'binary').toString('base64');
//console.log(crypted);
//console.log(cbase64);
crypted += cipher.final('binary');
//console.log("hahahaaaaaa:"+crypted.toString('hex'));
crypted = new Buffer(crypted, 'binary').toString('base64');
//var c16 = new Buffer(crypted, 'binary').toString(16);
//console.log(crypted);
//console.log(c16);
return crypted;
};
var decrypt = function (key, iv, crypted) {
crypted = new Buffer(crypted, 'base64').toString('binary');
var decipher = crypto.createDecipheriv('aes-128-cbc', key, iv);
var decoded = decipher.update(crypted, 'binary', 'utf8');
//console.log(decoded);
decoded += decipher.final('utf8');
//console.log(decoded);
return decoded;
};
var key='ABCDEFGHIJKLMNOP';
//var iv = new Buffer(crypto.randomBytes(16));
//var iv16 = iv.toString('hex').slice(0,16);
var iv16='0000000000000000';
var fs = require('fs');
fs.readFile('file.txt','utf8',function(err,data){
console.log(data);
var encrypted = encrypt(key,iv16,data);
console.log(encrypted);
var decrypted = decrypt(key,iv16,encrypted);
console.log(decrypted);
fs.writeFile('encrypted.txt',encrypted,function(err){});
});
c codes:
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <openssl/aes.h>
#include <openssl/pem.h>
#include <openssl/bio.h>
#include <openssl/evp.h>
#define AES_BITS 128
#define MSG_LEN 128
int base64_encode(char *in_str, int in_len, char *out_str)
{
BIO *b64, *bio;
BUF_MEM *bptr = NULL;
size_t size = 0;
if (in_str == NULL || out_str == NULL)
return -1;
b64 = BIO_new(BIO_f_base64());
bio = BIO_new(BIO_s_mem());
bio = BIO_push(b64, bio);
BIO_write(bio, in_str, in_len);
BIO_flush(bio);
BIO_get_mem_ptr(bio, &bptr);
memcpy(out_str, bptr->data, bptr->length);
out_str[bptr->length] = '\0';
size = bptr->length;
BIO_free_all(bio);
return size;
}
int aes_encrypt(char* in, char* key, char* out)//, int olen)
{
if(!in || !key || !out) return 0;
unsigned char iv[16];
for(int i=0; i<16; ++i)
iv[i]='0';
//printf("size:%d",AES_BLOCK_SIZE);
//unsigned char *iv = (unsigned char *)"0123456789ABCDEF";
printf("iv: %s\n",iv);
AES_KEY aes;
if(AES_set_encrypt_key((unsigned char*)key, 128, &aes) < 0)
{
return 0;
}
int len=strlen(in);
AES_cbc_encrypt((unsigned char*)in, (unsigned char*)out, len, &aes, iv, AES_ENCRYPT);
return 1;
}
int aes_decrypt(char* in, char* key, char* out)
{
if(!in || !key || !out) return 0;
unsigned char iv[16];
for(int i=0; i<16; ++i)
iv[i]='0';
//unsigned char *iv = (unsigned char *)"0123456789ABCDEF";
AES_KEY aes;
if(AES_set_decrypt_key((unsigned char*)key, 128, &aes) < 0)
{
return 0;
}
int len=strlen(in);
AES_cbc_encrypt((unsigned char*)in, (unsigned char*)out, len, &aes, iv, AES_DECRYPT);
return 1;
}
int main(int argc,char *argv[])
{
char sourceStringTemp[MSG_LEN];
char dstStringTemp[MSG_LEN];
char dstStringTemp_base64[MSG_LEN];
memset((char*)sourceStringTemp, 0 ,MSG_LEN);
memset((char*)dstStringTemp, 0 ,MSG_LEN);
strcpy((char*)sourceStringTemp, "My name is Harlan Chen!");
//strcpy((char*)sourceStringTemp, argv[1]);
char key[AES_BLOCK_SIZE]={0};
int i;
for(i = 0; i < 16; i++)
{
key[i] = 'A' + i;
}
printf("keys:%s\n",key);
if(!aes_encrypt(sourceStringTemp,key,dstStringTemp))
{
printf("encrypt error\n");
return -1;
}
/*To Base64 encrypted data */
base64_encode(dstStringTemp, strlen(dstStringTemp),dstStringTemp_base64);
printf("Base64 Encrypted data: %s\n",dstStringTemp_base64);
printf("encrypted:%s\n",dstStringTemp);
printf("enc %lu:",strlen((char*)dstStringTemp));
for(i= 0;dstStringTemp[i];i+=1){
printf("%x",(unsigned char)dstStringTemp[i]);
}
memset((char*)sourceStringTemp, 0 ,MSG_LEN);
if(!aes_decrypt(dstStringTemp,key,sourceStringTemp))
{
printf("decrypt error\n");
return -1;
}
printf("\n");
printf("dec %lu:",strlen((char*)sourceStringTemp));
printf("%s\n",sourceStringTemp);
//for(i= 0;sourceStringTemp[i];i+=1){
// printf("%x",(unsigned char)sourceStringTemp[i]);
//}
printf("\n");
return 0;
}
nodejs results:
bogon:AES_128_encryption zexu$ node encrypt.js
My name is Harlan Chen!
jERcWr8ZMzSJcKPGB7RYAYRpMftlThxyZcjfbFYlU3g=
My name is Harlan Chen!
and the c language results:
bogon:AES_128_encryption zexu$ ./a.out
keys:ABCDEFGHIJKLMNOP
iv: 0000000000000000
Base64 Encrypted data: jERcWr8ZMzSJcKPGB7RYAf6kEOmjJgUksDtrttx4r3k=
encrypted:?D\Z?34?p???X???&$?;k??x?y
enc 32:8c445c5abf1933348970a3c67b4581fea410e9a326524b03b6bb6dc78af79
dec 23:My name is Harlan Chen!
Compare the two base64 strings :
jERcWr8ZMzSJcKPGB7RYAYRpMftlThxyZcjfbFYlU3g=
jERcWr8ZMzSJcKPGB7RYAf6kEOmjJgUksDtrttx4r3k=
The first 21 characters are the same,the following ones are different.I don't know why.
AES is a block cipher, which means, that you always encrypt full blocks of 128 bit (i.e. 16 byte).
Now if you look at your plaintext "My name is Harlan Chen!", you will notice, that these are 23 bytes. This means, that 9 bytes are still left, which is filled with PKCS#7-Padding (9 bytes of value 09) in the nodejs-case, but with zero-bytes in the C case. Furthermore, I suspect there is a trailing newline in the file.txt with your string, which isn't present in the C example, too.
So, you should check with a hex editor, that your string "My name is Harlan Chen!" without a newline and padded with zero bytes up to 32 bytes is inside your file.txt.
This will still not yield exactly the same result, since one full block of padding will be added by nodejs (see PKCS#7-Padding) because one byte is always added as padding. But you can disable the automatic padding in your nodejs-script wih
cipher.setAutoPadding(0);
and
decipher.setAutoPadding(0);
Then you should get the same results. As mentioned above, make sure your file.txt does not contain a newline and is padded to 32 bytes with zero-bytes.
Alternatively, you can change your C program to implement PKCS#7 padding; then you should also get the same results. The string to encrypt would be
"My name is Harlan Chen!\x09\x09\x09\x09\x09\x09\x09\x09\x09"
then
I'm working on a simple program that uses OpenSSL to do basic RSA encryption and decryption. It is working fine for small messages (<16 bytes), but fails for anything over that. I understand that a limitation of public key cryptography is that you cannot encrypt anything longer than the key size. In my case, I'm using a 1024bit key, so I should have 128bytes to work with (maybe slightly less due to padding), correct? If so, that's not what I'm experiencing.
Here's the output from my program with 15 bytes:
Generating RSA keypair...done.
Message to encrypt: 0123456789ABCDE
16 bytes encrypted
Decrypted message: 0123456789ABCDE
And with 16 bytes:
Generating RSA keypair...done.
Message to encrypt: 0123456789ABCDEF
16 bytes encrypted
140153837057696:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:467:
Decrypted message: (null)
It seems that no matter what, only a total of 16 bytes are encrypted.
My encryption function (updated with fix):
unsigned char* rsa_seal(EVP_PKEY *pub_key, unsigned char *msg, size_t **enc_msg_len, unsigned char **sym_key, int *sym_key_len, unsigned char **iv) {
size_t msg_len = strlen((char*)msg);
unsigned char *encrypt = malloc(EVP_PKEY_size(pub_key));
EVP_CIPHER_CTX *ctx = malloc(sizeof(EVP_CIPHER_CTX));
EVP_CIPHER_CTX_init(ctx);
*sym_key = malloc(EVP_PKEY_size(pub_key));
*iv = malloc(EVP_MAX_IV_LENGTH);
**enc_msg_len = 0;
if(!EVP_SealInit(ctx, EVP_aes_128_cbc(), sym_key, sym_key_len, *iv, &pub_key, 1)) {
ERR_print_errors_fp(stderr);
encrypt = NULL;
goto return_free;
}
if(!EVP_SealUpdate(ctx, encrypt, (int*)*enc_msg_len, msg, (int)msg_len)) {
ERR_print_errors_fp(stderr);
encrypt = NULL;
goto return_free;
}
if(!EVP_SealFinal(ctx, encrypt, (int*)*enc_msg_len)) {
ERR_print_errors_fp(stderr);
encrypt = NULL;
goto return_free;
}
return_free:
EVP_CIPHER_CTX_cleanup(ctx);
free(ctx);
ctx = NULL;
return encrypt;
}
The corresponding decryption function (updated with fix):
char* rsa_open(EVP_PKEY *pri_key, unsigned char *enc_msg, size_t *enc_msg_len, unsigned char *sym_key, int sym_key_len, unsigned char *iv) {
size_t dec_len = 0;
unsigned char *decrypt = malloc((*enc_msg_len) + EVP_MAX_IV_LENGTH);
if(decrypt == NULL) return NULL;
EVP_CIPHER_CTX *ctx = malloc(sizeof(EVP_CIPHER_CTX));
EVP_CIPHER_CTX_init(ctx);
if(!EVP_OpenInit(ctx, EVP_aes_128_cbc(), sym_key, sym_key_len, iv, pri_key)) {
ERR_print_errors_fp(stderr);
decrypt = NULL;
goto return_free;
}
if(!EVP_OpenUpdate(ctx, decrypt, (int*)&dec_len, enc_msg, (int)*enc_msg_len)) {
ERR_print_errors_fp(stderr);
decrypt = NULL;
goto return_free;
}
if(!EVP_OpenFinal(ctx, decrypt, (int*)&dec_len)) {
ERR_print_errors_fp(stderr);
decrypt = NULL;
goto return_free;
}
decrypt[dec_len] = '\0';
return_free:
EVP_CIPHER_CTX_cleanup(ctx);
free(ctx);
ctx = NULL;
return (char*)decrypt;
}
The key generation function:
int rsa_init(EVP_PKEY **rsa_keypair) {
EVP_PKEY_CTX *ctx = EVP_PKEY_CTX_new_id(EVP_PKEY_RSA, NULL);
if(!EVP_PKEY_keygen_init(ctx)) {
ERR_print_errors_fp(stderr);
return -1;
}
if(!EVP_PKEY_CTX_set_rsa_keygen_bits(ctx, KEY_LENGTH)) {
ERR_print_errors_fp(stderr);
return -1;
}
if(!EVP_PKEY_keygen(ctx, rsa_keypair)) {
ERR_print_errors_fp(stderr);
return -1;
}
EVP_PKEY_CTX_free(ctx);
return 0;
}
And finally, my main:
int main() {
EVP_PKEY *rsa_keypair = NULL; // RSA keypair
char msg[BUFFER]; // Message to encrypt
unsigned char *encrypt = NULL; // Encrypted message
char *decrypt = NULL; // Decrypted message
// Generate key pair
printf("Generating RSA keypair...");
if(rsa_init(&rsa_keypair) == -1) {
fprintf(stderr, "\nError generating RSA keypair.\n");
exit(1);
}
printf("done.\n");
// Get the message to encrypt
printf("Message to encrypt: ");
fgets(msg, BUFFER-1, stdin);
msg[strlen(msg)-1] = '\0';
// Load error strings in anticipation of error
ERR_load_crypto_strings();
// Encrypt the message
size_t *encrypt_len = malloc(sizeof(size_t));
unsigned char *sym_key = NULL;
unsigned char *iv = NULL;
int sym_key_len;
encrypt = rsa_seal(rsa_keypair, (unsigned char*)msg, &encrypt_len, &sym_key, &sym_key_len, &iv);
printf("%d bytes encrypted\n", (int)*encrypt_len);
// Decrypt it
decrypt = rsa_open(rsa_keypair, (unsigned char*)encrypt, (size_t*)encrypt_len, sym_key, sym_key_len, iv);
printf("Decrypted message: %s\n", decrypt);
free(encrypt);
free(decrypt);
free(encrypt_len);
free(sym_key);
free(iv);
EVP_PKEY_free(rsa_keypair);
return 0;
}
Any help is greatly appreciated! Thank you.
EDIT: As pointed out by math below, it seems that the answer to my mistake was hiding in the OpenSSL here: https://www.openssl.org/docs/crypto/EVP_EncryptInit.html#
This is because you do not properly handle out and outl parameters in EVP_SealUpdate(), EVP_SealFinal(), EVP_OpenUpdate() and EVP_OpenFinal().
Each EVP_XxxxUpdate() and EVP_XxxxFinal() call will contribute to the output buffer. So, you are required to keep track of the seal/open process by summing each outl returned and providing the expected buffer each time (start of buffer + already handled bytes).
unsigned char* rsa_seal(...)
{
...
**enc_msg_len = 0;
EVP_SealUpdate(ctx, encrypt + **enc_msg_len, &outl, msg, (int)msg_len);
**enc_msg_len += outl;
EVP_SealFinal(ctx, encrypt + **enc_msg_len, &outl);
**enc_msg_len += outl;
...
}
char* rsa_open(...)
{
...
dec_len = 0;
EVP_OpenUpdate(ctx, decrypt + dec_len, &outl, enc_msg, (int)*enc_msg_len);
dec_len += outl;
EVP_OpenFinal(ctx, decrypt + dec_len, &outl);
dec_len += outl;
...
}
The program was working with 15-bytes buffer because in that case, the EVP_XxxxUpdate() call is returning 0 in outl (not enough data to seal/open a block), hiding the problem in your code logic.
Note: The data is not directly encrypted using the RSA key but using a generated symetric key (AES-128 in your case). This is why the block size is 16 bytes.