Encrypt/decrypt single octet with no padding using RSA_public_encrypt - c

Question

I am trying to encrypt the decimal number 1 with no padding using RSA_public_encrypt. Of course the result of this encryption should be 1 again. But in fact it is
Cipher:
5228673350895653896383201058815462877426144378065091716669226352446563314310753759122681282019606231774744798481087922688401463790373072832646006963123849664906627085625735903412727878929921552918090336305565457632762612646244666075746315455854479912774071351362988587769873374454538153517081634942043196173525640004822126101409481060928041484882764412543090155480476597339865942466034635200613687987398189458867055031285752787781897557950334515480742629110423562374837915117562936777536259795189526199285672603820591631423849227226304977053415509662563290672486408474162339095049681422645956742727171481170704
What am I doing wrong?
I am aware that there are high level routines in OpenSSL and also that using RSA without padding is insecure. This is just an experiment.
I am also aware that OpenSSL uses the Big Endian format. So I also tried to change the order of the input bytes to the encryption - then 1 was encrypted to 1. But implementing the decryption, taking into account the reverse byte order in both encryption and decryption, some numbers, like 2, were encrypted-decrypted correctly and others were not...very strange!
Here is my code for the encryption:
#include <stdio.h>
#include "rsa.h"
#include "pem.h"
#define LENGTH 1000
int main(void)
{
//Variables for message and cipher
//BIGNUM format
BIGNUM message_bignum, cipher_bignum;
BIGNUM *ptr_message_bignum=&message_bignum;
BIGNUM *ptr_cipher_bignum=&cipher_bignum;
//Formatted as decimal string
unsigned char message_decimal[]="1";
//binary format for input to RSA encryption
unsigned char message[LENGTH]={0},cipher[LENGTH]={0};
//Initialise RSA structure
RSA *rsa=RSA_new();
//Get public key
BIO *publickey_handle=BIO_new_file("rsa_publickey.txt","rb");
if(publickey_handle==NULL)
{
fprintf(stderr,"Could not open key file!\n");
return -1;
}
PEM_read_bio_RSA_PUBKEY(publickey_handle,&rsa,NULL,NULL);
if(BIO_free(publickey_handle)==0)
{
fprintf(stderr,"Error closing key file!\n");
return -1;
}
//Convert message to BIGNUM format
BN_init(ptr_message_bignum);
BN_dec2bn(&ptr_message_bignum,(const char *)message_decimal);
//Convert message as BIGNUM to binary format
BN_bn2bin(ptr_message_bignum,message);
//Encrypt message in binary format
if(RSA_public_encrypt(RSA_size(rsa),(const unsigned char*)message,cipher,rsa,RSA_NO_PADDING)==-1)
{
fprintf(stderr,"Error during encryption!\n");
return -1;
}
//Convert cipher to BIGNUM format
BN_init(ptr_cipher_bignum);
BN_bin2bn((const unsigned char*)cipher,RSA_size(rsa),ptr_cipher_bignum);
//Convert cipher from BIGNUM format to decimal format and print it to stdout
printf("Cipher:\n%s",BN_bn2dec(ptr_cipher_bignum));
printf("\nDone!\n");
return 0;
}

Answer 1

As documented:
BN_bn2bin() converts the absolute value of a into big-endian form and stores it at to. to must point to BN_num_bytes(a) bytes of memory.
But it doesn't. It first contains a single 00 because of message[LENGTH]={0}. Then it contains, well, whatever is in LENGTH amount of bytes.
In RSA_public_encrypt you use RSA_size(rsa) as first parameter, flen. flen however should be the result of BN_num_bytes(&ptr_message_bignum), i.e. 1 byte. The output location cipher on the other hand must be able to store RSA_size(rsa) bytes.
Possibly the confusion comes from the fact that BN_bn2bin will store the value as the shortest Big Endian encoding (in bytes). For the value 1 that will of course be one byte, not RSA_size(rsa) bytes.
Edit: I found the RSA_padding_add_none method that could be able to resize.