Segmentation Fault While Encrypting with Openssl EVP: EVP_EncryptUpdate() - c

Question

I want to use EVP and OpenSSL API to encrypt the binary data from a .exe file that I read into an unsigned char *. I'm not super familiar with this API and I'm afraid I'm doing something wrong that is causing the segmentation fault that I get when I compile. Here is my code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <openssl/conf.h>
#include <openssl/evp.h>
#include <openssl/err.h>
#include <openssl/aes.h>
void handleErrors(void){
ERR_print_errors_fp(stderr);
abort();
}
int main(){
//read in the exe file and convert its bytes to a string
FILE *inFile;
inFile = fopen("Quasar.exe","rb");
if (inFile == NULL){
perror("Failed to read in file\n");
}
printf("File read in complete!\n");
if(fseek(inFile , 0 , SEEK_END) == -1){
perror("Offset error\n");
};
unsigned long lSize = ftell(inFile);
if (lSize == -1){
perror("Size error\n");
}
rewind(inFile);
unsigned char *unencryptedText = (unsigned char*) malloc (sizeof(unsigned char)*lSize);
fread(unencryptedText,1,lSize,inFile);
fclose(inFile);
unsigned char *encryptedText = (unsigned char*) malloc (3 *(sizeof(unsigned char)*lSize));
//encrypt these bytes with open ssl
printf("Encrypting...\n");
int outlen, tmplen;
unsigned char key[] = {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15};
unsigned char iv[] = {1,2,3,4,5,6,7,8};
EVP_CIPHER_CTX *ctx = NULL;
EVP_CIPHER_CTX_init(ctx);
EVP_EncryptInit_ex(ctx, EVP_idea_cbc(), NULL, key, iv);
if(!EVP_EncryptUpdate(ctx, encryptedText, &outlen, unencryptedText, lSize)){
return 0;
}
if(!EVP_EncryptFinal_ex(ctx, encryptedText + outlen, &tmplen)){
return 0;
}
outlen += tmplen;
EVP_CIPHER_CTX_cleanup(ctx);
return 0;
}
Here is the console output:
File read in complete!
Encrypting...
zsh: segmentation fault ./binary
Here is the Make script:
all: crypter.c
gcc -Wall -g -I/usr/local/include/openssl -L/usr/lib -lssl -lcrypto crypter.c -o binary
clean:
rm -f crypter

Answer 1

EVP_CIPHER_CTX *ctx = NULL;
EVP_CIPHER_CTX_init(ctx);
EVP_EncryptInit_ex(ctx, EVP_idea_cbc(), NULL, key, iv);
At no time do you assign ctx any value other than NULL. You pass a NULL to both of the functions above.
What you probably wanted was:
ECP_CIPHER_CTX ctx;
EVP_CIPHER_CTX_init(&ctx);
Notice how this doesn't pass NULL to EVP_CIPHER_CTX_init?

Answer 2

You're passing a NULL object to EVP_CIPHER_CTX_init This function expects the address of an existing EVP_CIPHER_CTX object.
You can fix this by creating an instance of EVP_CIPHER_CTX and passing its address to each function that needs it, or if you want to minimize changes to your code you can assign the address of this object to the existing ctx pointer.
EVP_CIPHER_CTX ctx_obj;
EVP_CIPHER_CTX *ctx = &ctx_obj;
EDIT:
Based on the comments, it seems you're using OpenSSL 1.1 or later. In that case, you want to instead use EVP_CIPHER_CTX_new to instantiate a new context:
EVP_CIPHER_CTX *ctx = EVP_CIPHER_CTX_new();