OPENSSL 1.1.1d signature verification faills, RSASSA-PSS signature scheme - c

Question

I am trying to generate an RSASSA-PSS signature with openssl 1.1.1d.
The code below does generate some signature, different each time I run it, as expected for PSS scheme.
However, whenever I try to verify the signature online, it fails (I am using this site to do the verification: https://8gwifi.org/RSAFunctionality?rsasignverifyfunctions=rsasignverifyfunctions&keysize=2048).
Any expert here who could see if anything is missing in this source code?
int main()
{
EVP_MD_CTX *mdctx = NULL;
int ret = 0;
unsigned char *sig;
EVP_PKEY *key = NULL;
EVP_PKEY_CTX *ctx = NULL;
char *msg = "This is a nice message to be signed.";
int len;
size_t siglen;
FILE *fp;
char sig64[512];
unsigned char md[SHA256_DIGEST_LENGTH];
SHA256_CTX sha256;
SHA256_Init(&sha256);
SHA256_Update(&sha256, msg, strlen(msg));
SHA256_Final(md, &sha256);
sig = NULL;
len = strlen(msg);
fp = fopen("MYPRIVATEKEY.KEY", "r");
key = EVP_PKEY_new();
PEM_read_PrivateKey(fp, &key, NULL, NULL);
ctx = EVP_PKEY_CTX_new(key, NULL);
if (EVP_PKEY_sign_init(ctx) <= 0)
goto err;
if (EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_PSS_PADDING) <= 0)
goto err;
if (EVP_PKEY_CTX_set_signature_md(ctx, EVP_sha256()) <= 0)
goto err;
/* Determine buffer length */
if (EVP_PKEY_sign(ctx, NULL, &siglen, md, SHA256_DIGEST_LENGTH) <= 0)
goto err;
sig = OPENSSL_malloc(siglen);
if (!sig)
goto err;
if (EVP_PKEY_sign(ctx, sig, &siglen, md, SHA256_DIGEST_LENGTH) <= 0)
goto err;
/* Signature is siglen bytes written to buffer sig */
/* Success */
ret = 1;
Encode_B64(sig, siglen, sig64);
printf("Signature: %s\n", sig64);
err:
if(ret != 1)
{
/* Do some error handling */
}
/* Clean up */
if(mdctx) EVP_MD_CTX_free(mdctx);
EVP_PKEY_free(key);
EVP_PKEY_CTX_free(ctx);
if(sig && !ret) OPENSSL_free(sig);
return 0;
}

Answer 1

The PSS parameters must match on both sides for successful verification:
the PSS digest: this is set to SHA256 in the C code with EVP_PKEY_CTX_set_signature_md(ctx, EVP_sha256()).
the MGF1 digest: This corresponds to the PSS digest by default, so it is also SHA256 here. If a different MGF1 digest is to be set, this is possible with EVP_PKEY_CTX_set_rsa_mgf1_md().
the salt length: this is set by OpenSSL by default equal to the maximum salt length (modulus - output length of the digest - 2, in bytes), which thus also applies here. The salt length can be set explicitly with EVP_PKEY_CTX_set_rsa_pss_saltlen() with the values RSA_PSS_SALTLEN_MAX (maximum salt length, default) and RSA_PSS_SALTLEN_DIGEST (output length of the PSS digest).
For completeness: Actually, more generally, the mask generation function must be specified, but by default MGF1 is applied, so that only the digest used by this function must be set (see point 2). In addition there is the trailer field number, which is not critical here, since both sides take the usual value of 1 by default.
The website, in contrast, uses different values for the salt length and the two digests:
RSASSA-PSS and SHA1WithRSA/PSS: SHA1 for the PSS and the MGF1 digest, salt length = output length of the digest (20 bytes)
SHA224WithRSA/PSS: SHA224 for the PSS and MGF1 digest, salt length = output length of the digest (28 bytes)
SHA384WithRSA/PSS: SHA384 for the PSS and the MGF1 digest, salt length = output length of the digest (48 bytes)
None of the PSS settings use SHA256 or the maximum salt length.
Due to the different PSS parameters, the verification fails.
However, verification is possible e.g. with the OpenSSL command line tool (e.g. with openssl pkeyutl):
openssl pkeyutl -verify -in <message hash file> -inkey <public key file> -sigfile <signature file> -pubin -pkeyopt rsa_padding_mode:pss -pkeyopt digest:sha256
The switches -pkeyopt rsa_mgf1_md:sha256 -pkeyopt rsa_pss_saltlen:max are set by default in the above statement.
The key file contains the PEM encoded X.509/SPKI key, the other two files contain the raw (i.e. not Base64 encoded) data of message hash and signature.