AES cbc 256 decryption failed in C openssl - c

Question

I'm trying to write AES encryption/decryption program in C using openssl. However, when I tried to decrypt the message, I got error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block length.
void aes_encrypt(unsigned char* in, int inl, unsigned char *out, int* len, unsigned char * key){
unsigned char iv[16] = "encryptionIntVec";
EVP_CIPHER_CTX ctx;
EVP_CIPHER_CTX_init(&ctx);
EVP_EncryptInit_ex(&ctx, EVP_aes_256_cbc(), NULL, key, iv);
*len = 0;
int outl = 0;
EVP_EncryptUpdate(&ctx, out+*len, &outl, in+*len, inl);
*len+=outl;
int test = inl>>4;
if(inl != test<<4){
EVP_EncryptFinal_ex(&ctx,out+*len,&outl);
*len+=outl;
}
EVP_CIPHER_CTX_cleanup(&ctx);
}
void aes_decrypt(unsigned char* in, int inl, unsigned char *out, unsigned char *key){
unsigned char iv[16] = "encryptionIntVec";
EVP_CIPHER_CTX ctx;
EVP_CIPHER_CTX_init(&ctx);
int result;
result = EVP_DecryptInit_ex(&ctx, EVP_aes_256_cbc(), NULL, key, iv);
if(result > 0) {
printf("passed\n");
} else {
printf("failed\n");
}
int len = 0;
int outl = 0;
result = EVP_DecryptUpdate(&ctx, out+len, &outl, in+len, inl);
if(result > 0) {
printf("passed\n");
} else {
printf("failed\n");
}
len += outl;
result = EVP_DecryptFinal_ex(&ctx, out+len, &outl);
if(result > 0) {
printf("passed\n");
} else {
printf("failed\n");
ERR_print_errors_fp(stdout);
}
len+=outl;
out[len]=0;
EVP_CIPHER_CTX_cleanup(&ctx);
}
int main()
{
unsigned char content[400];
unsigned char key[] = "0123456789abcdef0123456789abcdef";
/******************Block 1*****************************/
unsigned char en[400],de[400],base64[400], base64_out[400];
int len;
memset(content, 0,400);
memset(en, 0, 400);
memset(de, 0, 400);
memset(base64, 0,400);
memset(base64_out, 0, 400);
strcpy((char *)content, "loc: 123.2132412, -39.123142");
printf("%d %s\n", strlen((const char*)content), content);
//encrypt content
aes_encrypt(content,strlen((const char*)content), en, &len, (unsigned char*)key);
//base64 encode ciphertext
int encode_str_size = EVP_EncodeBlock(base64, en, len);
printf("%d %s\n", encode_str_size, base64);
//base64 decode
int length = EVP_DecodeBlock(base64_out, base64, strlen((const char*)base64));
while(base64[--encode_str_size] == '=') length--;
//decrypt
aes_decrypt(base64_out, length, de, (unsigned char*)key);
printf("%d %s\n", strlen((const char*)de), de);
/***********************Block 2*******************************/
unsigned char msg_out[400];
unsigned char msg[400] = "6hKe8RGg+4p1N1R6Y9aaTovxLtuH115JoWUO8plrAJE=";
unsigned char result[400];
int l = EVP_DecodeBlock(msg_out, msg, strlen((const char*)msg));
if(strcmp((const char*)msg, (const char*)base64)==0) {
printf("match\n");
}
if(strcmp((const char*)en, (const char*)msg_out)==0) {
printf("match\n");
}
while(msg[--encode_str_size] == '=') l--;
aes_decrypt(msg_out, l, result, (unsigned char*)key);
printf("%d %s\n", strlen((const char*)result), result);
return 0;
}
In block 1 of main function, I encrypted, base64 encoded, base64 decoded, decrypted, then got the exactly the same String as it was, and produced NO error.
However, in the block 2, I directly used the base64 encoded string which was produced from block 1, decoded and decypted, but an error occurred at result = EVP_DecryptFinal_ex(&ctx, out+len, &outl); that was error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block length.
If I printed the decrypted string anyways, it is loc: 123.2132412, -39.123142^D^D^D^D, there are four '^D' appended at the end. I compared strings, both match was printed, which means the ciphertexts that was decoded from base64 with the one(en) in block 1 were the same. (NOTE: same key, same IV)
Any ideas why it failed?

Answer 1

I am using version OpenSSL 3.0.0-dev and therefore had to modify the code for the reasons described in David C. Rankin's comment. The problem occurs even after these changes.
Debugging reveals that the length of the Base64 decoded data in block 2 is determined incorrectly with l = 0x21 = 33. The correct value is l = 0x20 = 32.
This is because in block 2, when determining the length of the Base64 decoded data without the 0x00 padding bytes (added by EVP_DecodeBlock()) with
while (msg[--encode_str_size] == '=') l--;
the value for encode_str_size already changed in block 1 is used. The fix is to apply the current value for the length of the Base64 encoded data, e.g.
encode_str_size = strlen((const char*)msg);
Then decryption for block 2 works on my machine.
Note that EVP_DecodeBlock() always returns the Base64 decoded data with a length equal to an integer multiple of 3, padded with 0x00 values if necessary.
The actual length of the data can therefore also be determined with msg_out and l alone. l specifies the length including the 0x00 padding bytes, so that only the number of the 0x00 padding bytes has to be subtracted from l.
Thus, msg and encode_str_size are actually not needed (although both can be used, since the number of = padding bytes is equal to the number of 0x00 padding bytes).
As already mentioned in the comment by Joël Hecht, the EVP_EncryptFinal_ex() call is necessary, which among other things performs the padding (PKCS7).
In the currently posted code, this only occurs if the length of the plaintext does not equal an integer multiple of 16 bytes (the AES block size) which is satisfied for the test data.
However, for a plaintext whose length is a multiple of 16 bytes, EVP_EncryptFinal_ex() is not called, so padding is not performed, resulting in a decryption failure in EVP_DecryptFinal_ex().
Apart from the Base64 encoding, the OpenSSL documentation here provides a complete example for EVP_aes_256_cbc(), which could be used for comparison.