XOR two hexadecimal strings(unmasking) in C - c

Question

I wanted to unmask a hex string using the following algorithm in C
j = i MOD 4 transformed-octet-i = original-octet-i XOR masking-key-octet-j
#include<stdlib.h>
#include<string.h>
int main()
{
char masked[]="951bfdcdc113ebca921fe9dc";
char masking_key[]="e17e8eb9";
char *unmasked;
int length=strlen(masked);
unmasked=malloc(sizeof(char)*(length+1));
int i=0;
for(i=0;i<length;i++)
{
unmasked[i]=masked[i]^masking_key[i%4];
}
printf("%s\n",unmasked);
return 0;
}
The output I am getting is \UT instead of 74657374206d657373616765. It would be really helpful if someone could help me fix the error here.

Answer 1

You need to convert from string format to raw integers before doing the XOR, then convert back before printing it as a string. Otherwise you'll XOR the symbol values, not the raw values.
You can convert the whole string in one go with strtol(data, 0, 16). But one of the common details that makes programmers different from the code monkeys is the ability to code trivial string-integer conversions without the help of library functions. So here is a a simplified code doing just that - be aware of the complete lack of error handling, as this is just quick & dirty code:
#include <stdio.h>
#include <stdlib.h>
char hexlify (char nibble)
{
return "0123456789abcdef"[nibble];
}
char unhexlify (char ch)
{
if(ch>='0' && ch<='9')
return ch - '0';
if(ch>='a' && ch<='f')
return ch - 'a' + 0xA;
return 0;
}
int main (void)
{
char masked[]="951bfdcdc113ebca921fe9dc";
char masking_key[]="e17e8eb9";
char *unmasked;
size_t length = sizeof masked - 1;
unmasked = malloc(length + 1);
for(size_t i=0;i<length;i++)
{
char op1 = unhexlify(masked[i]);
char op2 = unhexlify(masking_key[i%4]);
unmasked[i]= hexlify(op1 ^ op2);
}
unmasked[length]='\0';
printf("%s\n",unmasked);
free(unmasked);
return 0;
}
Output:
74651cb3206d0ab4736108a2

Answer 2

What you're doing here is not xoring the hexadecimal bytes, but xoring the characters representing them.
While the ideal process would be (0x95 ^ 0xe1) + (0xbf ^ 7e) + ..., what you're doing is xoring the ascii values of each character, meaning ('9' ^ 'b') + ('5' ^ '1') + ....
What you need to do is first transform the hexadecimal string to the bytes themselves (bytes.fromhex("951bfdcdc113ebca921fe9dc") in python), and only then XOR them. The way I would do that is with sscanf("%2x", ...).
For example:
#include <stdio.h>
int main()
{
unsigned char a = 0;
scanf("%2hhx", &a);
}