How to encrypt/decrypt long input messages with RSA? [Openssl, C] - c

Question

I wrote a simple test program that encrypts/decrypts a message.
I have a keylength:
int keylength = 1024; // it can also be 2048, 4096
and max input length:
int maxlen = (keylength/8)-11;
and I know that my input size should be < than maxlen, something like this:
if(insize >= maxlen)
printf("cannot encrypt/decrypt!\n");
My question is simple - is it possible (if so, how can I do this) to encrypt/decrypt with RSA messages LONGER than maxlen?
Main code is also, very simple but works only when insize < maxlen:
if((encBytes=RSA_public_encrypt(strlen(buff1)+1, buff1, buff2, keypair, RSA_PKCS1_PADDING)) == -1)
{
printf("error\n");
}
if((decBytes=RSA_private_decrypt(encBytes, buff2, buff3, keypair, RSA_PKCS1_PADDING)) == -1)
{
printf("error\n");
}

Answer 1

Encrypting long messages requires combined scheme - RSA algorithm encrypts session key (i.e. AES key), and data itself is encrypted with that key.
I would recommend to not invent another bicycle and use well established scheme, i.e. PKCS#7/CMS or OpenPGP, depending on your needs.

Answer 2

You would be able to encrypt long messages with RSA the same way as it is done with block ciphers. That is, encrypt the messages in blocks and bind the blocks with an appropriate chaining mode. However, this is not the usual way to do it and you won't find support for it (RSA chaining) in the libraries you use.
Since RSA is quite slow, the usual way to encrypt large messages is using hybrid encryption. In hybrid encryption you use a fast symmetric encryption algorithm (like AES) for encrypting the data with a random key. The random key is then encrypted with RSA and send along with the symmetric key encrypted data.
EDIT:
As fore your implementation, you have insize = 1300 and keylength = 1024 which gives maxlen = 117. To encrypt the full message you those needs 12 encrypts, that each produce 128 bytes, giving an encrypted size of 1536 bytes. In your code you only allocates buffers of 1416 bytes. Also, you don't seem to allow for 128 bytes output as you only increment with 117 in:
RSA_public_encrypt(maxlen, buff1+i, buff2+i, keypair, RSA_PKCS1_PADDING)
and
i += maxlen;

Answer 3

You can use RSA as block cipher in that case. That is break the message to several blocks smaller than maxlen.
Otherwise impossible.

Answer 4

If you want to run RSA in a "block cipher" kind of mode, you would need to run it in a loop.
Like most of the other commenters, I'd like to point out that this is a bad use of RSA - You should just encrypt a AES key with RSA then use AES for the longer message.
However, I'm not one to let practicality get in the way of learning, so here's how you'd do it. This code isn't tested, since I don't know what libraries you are using. It's also a little overly-verbose, for readability.
int inLength = strlen(buff1)+1;
int numBlocks = (inLength / maxlen) + 1;
for (int i = 0; i < numBlocks; i++) {
int bytesDone = i * maxlen;
int remainingLen = inLength - bytesDone;
int thisLen; // The length of this block
if (remainingLen > maxlen) {
thisLen = maxlen;
} else {
thisLen = remainingLen;
}
if((encBytes=RSA_public_encrypt(thisLen, buff1 + bytesDone, buff2 + bytesDone, keypair, RSA_PKCS1_PADDING)) == -1)
{
printf("error\n");
}
// Okay, IDK what the first parameter to this should be. It depends on the library. You can figure it out, hopefully.
if((decBytes=RSA_private_decrypt(idk, buff2 + bytesDone, buff3 + bytesDone, keypair, RSA_PKCS1_PADDING)) == -1)
{
printf("error\n");
}
}

Answer 5

maxlen actually depends on a key length and padding mode. Think newer padding scheme ´OAEP´ e.g. in Java Encryption Engine takes additional 42 bytes instead of 11. Known libraries are not designed for using RSA in a block cipher mode.
For that purpose, beyond fragmentation as answered above, security aspects require further modification of the padding scheme, e,g, https://crypto.stackexchange.com/a/97974/98888