Making conversion similar to BN_hex2bn + BN_bn2bin without openSSL in C - c

Question

I currently use openSSL to convert values from encrypted string to what I thought was a binary array. I then decrypt this "array" (pass to EVP_DecryptUpdate). I make the conversion like this:
BIGNUM *bnEncr = BN_new();
if (0 == BN_hex2bn(&bnEncr, encrypted)) { // from hex to big number
printf("ERROR\n");
}
unsigned int numOfBytesEncr = BN_num_bytes(bnEncr);
unsigned char encrBin[numOfBytesEncr];
if (0 == BN_bn2bin(bnEncr, encrBin)) { // from big number to binary
printf("ERROR\n");
}
Then I pass encrBin to EVP_DecryptUpdate and decryption works.
I do this in many places in my code and now want to write my own C function of converting hex to binary array, which I can then pass to EVP_DecryptUpdate. I had a go at this and converted my encrypted hex string to an array of 0s and 1s, but turns out that EVP_DecryptUpdate won't work with that. From what I could find online, BN_bn2bin "creates a representation that is truly binary (i.e. a sequence of bits). More specifically, it creates a big-endian representation of the number." So this is not just an array of 0s and 1s, right?
Can someone explain how I can make the hex->(truly) binary conversion myself in C, so I would get the format that EVP_DecryptUpdate expects? Is this complicated?

Answer 1

BN_bn2bin "creates a representation that is truly binary (i.e. a
sequence of bits). More specifically, it creates a big-endian
representation of the number." So this is not just an array of 0s and
1s, right?
The sequence of bits mentioned here is represented as an array of bytes. With each of those bytes containing 8 bits, this can be interpreted as an "array of 0s and 1s". It is not an "array of integers that have the value 0 or 1", if that is what you are asking.
Since you are unclear of the workings of BN_bn2bin(), it helps to just analyze the end result of your code snippet. You could do that like this (omitting any error checking):
#include <stdio.h>
#include <openssl/bn.h>
int main(
int argc,
char **argv)
{
const char *hexString = argv[1];
BIGNUM *bnEncr = BN_new();
BN_hex2bn(&bnEncr, hexString);
unsigned int numOfBytesEncr = BN_num_bytes(bnEncr);
unsigned char encrBin[numOfBytesEncr];
BN_bn2bin(bnEncr, encrBin);
fwrite(encrBin, 1, numOfBytesEncr, stdout);
}
This outputs the contents of encrBin to the standard output, which is never a nice thing to happen, but you can then pipe it through a tool like hexdump, or redirect it to a file for analyzing with a hex editor. It looks like this:
$ ./bntest 74162ac74759e85654e0e7762c2cdd26 | hexdump -C
00000000 74 16 2a c7 47 59 e8 56 54 e0 e7 76 2c 2c dd 26 |t.*.GY.VT..v,,.&|
00000010
Or, if you do want to see those 0s and 1s:
$ ./bntest 74162ac74759e85654e0e7762c2cdd26 | xxd -b -c 4
00000000: 01110100 00010110 00101010 11000111 t.*.
00000004: 01000111 01011001 11101000 01010110 GY.V
00000008: 01010100 11100000 11100111 01110110 T..v
0000000c: 00101100 00101100 11011101 00100110 ,,.&
This shows that your question
Can someone explain how I can make the hex->(truly) binary conversion
myself in C, so I would get the format that EVP_DecryptUpdate expects?
Is this complicated?
is essentially the same as the SO question How to turn a hex string into an unsigned char array?, like I commented.

Answer 2

It is unclear why you want this, and it's definitely not advisable to roll your own implementation of the conversion functions (they may stop working with any number of internal changes to OpenSSL) but if you're interested in what it looks like:
static int bn2binpad(const BIGNUM *a, unsigned char *to, int tolen)
{
int n;
size_t i, lasti, j, atop, mask;
BN_ULONG l;
/*
* In case |a| is fixed-top, BN_num_bytes can return bogus length,
* but it's assumed that fixed-top inputs ought to be "nominated"
* even for padded output, so it works out...
*/
n = BN_num_bytes(a);
if (tolen == -1) {
tolen = n;
} else if (tolen < n) { /* uncommon/unlike case */
BIGNUM temp = *a;
bn_correct_top(&temp);
n = BN_num_bytes(&temp);
if (tolen < n)
return -1;
}
/* Swipe through whole available data and don't give away padded zero. */
atop = a->dmax * BN_BYTES;
if (atop == 0) {
OPENSSL_cleanse(to, tolen);
return tolen;
}
lasti = atop - 1;
atop = a->top * BN_BYTES;
for (i = 0, j = 0, to += tolen; j < (size_t)tolen; j++) {
l = a->d[i / BN_BYTES];
mask = 0 - ((j - atop) >> (8 * sizeof(i) - 1));
*--to = (unsigned char)(l >> (8 * (i % BN_BYTES)) & mask);
i += (i - lasti) >> (8 * sizeof(i) - 1); /* stay on last limb */
}
return tolen;
}