Problems with C XOR executable file encryption / decryption - c

Question

I'm trying to create a simple XOR crypter / decrypter in C for .exe files. I'm still pretty new in C and don't understand everything yet, especially memory stuff. So I've been following an online tutorial on how to make a simple XOR string crypter which worked fine. Now I wanted to modify it so I can en/decrypt executable files and decided to utilize the fwrite() and fread() functions. This is what I've come up with:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h> // execve function
#define XOR_KEY 0xAA // key
#define JOB_CRYPT 1 // alter flow depending on the job
#define JOB_DECRYPT 2
//////////////////////////////////////////////////////////////////
//////////////////////////////////////////////////////////////////
void xorFile (char *infile, char *outfile) {
FILE *nFile, *eFile;
long nFileSize; // store file size of the file we want to read
char *buffer; // buffer for reading
char *eBuffer; // buffer for storing encrypted data
size_t rResult;
size_t wResult;
///// READ FILE /////
nFile = fopen(infile, "rb");
if(nFile == NULL) {
fputs("Error opening file...", stderr);
exit(EXIT_FAILURE);
}
fseek(nFile, 0, SEEK_END);
nFileSize = ftell(nFile);
rewind(nFile);
buffer = (char *) malloc(sizeof(char) * nFileSize);
if(buffer == NULL) {
fputs("Error allocating memory...", stderr);
exit(EXIT_FAILURE);
}
rResult = fread(buffer, 1, nFileSize, nFile);
if(rResult != nFileSize) {
fputs("Error reading file...", stderr);
exit(EXIT_FAILURE);
}
fclose(nFile);
printf("File size is: %ld\n", nFileSize);
printf("Buffer size is (pointer): %u\n", sizeof(buffer));
printf("Reading result: %lu\n", rResult);
////// WRITE TO FILE //////
eFile = fopen(outfile, "wb");
if(eFile == NULL) {
fputs("Error creating file...", stderr);
exit(EXIT_FAILURE);
}
eBuffer = (char *) malloc(sizeof(char) * nFileSize);
if(eBuffer == NULL) {
fputs("Error allocating memory (2)...", stderr);
exit(EXIT_FAILURE);
}
// encrypt byte by byte and save to buffer
printf("Proceeding with encryption!\n");
for(int i = 0; buffer[i] != EOF; i++) {
eBuffer[i] = buffer[i] ^ XOR_KEY;
}
printf("Proceeding with fwrite()!\n");
wResult = fwrite(eBuffer, 1, nFileSize, eFile);
fclose(eFile);
printf("eBuffer size is (pointer)%u\n", sizeof(eBuffer));
printf("Writing result: %lu\n", wResult);
free(buffer); // free buffers in heap
free(eBuffer);
}
//////////////////////////////////////////////////////////////////
//////////////////////////////////////////////////////////////////
int main(int argc, char *argv[]) {
// checking if all parameters were given
if(argc < 4) {
fprintf(stderr, "Usage: %s [CRYPT | DECRYPT] [IN-FILE] [OUT-FILE]\n", argv[0]);
exit(EXIT_FAILURE);
}
int job;
// DOLOCIMO JOB
if(strcmp(argv[1], "CRYPT") == 0) {
job = JOB_CRYPT;
} else if (strcmp(argv[1], "DECRYPT") == 0) {
job = JOB_DECRYPT;
} else {
fprintf(stderr, "Please select [CRYPT | DECRYPT]!");
exit(EXIT_FAILURE);
}
// CRYPT/DECRYPT OUR FILE
xorFile(argv[2], argv[3]);
if(job == JOB_DECRYPT) {
char *args[] = {argv[3], NULL};
int errExec = execve(args[0], args, NULL);
if(errExec == -1) {
perror("Error executing file...");
exit(EXIT_FAILURE);
}
}
return 0;
}
I'm sorry for the ugly looking code but I first wanted to make it work and I'll refine it later.
Anyways, when I run it in command prompt, the encryption works fine, it generates an encrypted file, but when I run the decrpytion job, the program
crashes during the decryption process. Here's a picture of what happens so you can imagine it better.
Since I have less than 10 reputation, I'm not allowed to embedd pictures.
Here is a link to Imgur.
What's going wrong here? Am I creating a buffer overflow when I'm decrypting it?
Thank you!

Answer 1

Here's the problem:
for(int i = 0; buffer[i] != EOF; i++) {
eBuffer[i] = buffer[i] ^ XOR_KEY;
}
Binary files can contain bytes with any value. So the EOF value is valid and does not designate the end of the file. This means that if the file contains a byte with this value, the loop will quit early and you won't XOR all the bytes. If the file does not contain this byte, the loop will run past the end of the allocated memory which invokes undefined behavior which in this case manifests in a crash.
You know how many bytes you need to processes, so use that as your loop control:
for(int i = 0; i < nFileSize; i++) {
A few other minor corrections:
buffer = (char *) malloc(sizeof(char) * nFileSize);
if(buffer == NULL) {
fputs("Error allocating memory...", stderr);
exit(EXIT_FAILURE);
}
Don't cast the return value of malloc. Also, sizeof(char) is 1 by definition, so you can leave that out.
Also, if a system or library function fails, you should use perror to print the error message. This will print additional information regarding why the function failed.
buffer = malloc(nFileSize);
if(buffer == NULL) {
perror("Error allocating memory...");
exit(EXIT_FAILURE);
}