RSA: Add a less loaded PSS-parameter structure

RSA_PSS_PARAMS carries with it a lot of baggage in form of X509_ALGOR
and ASN1_INTEGER, which we would rather avoid in our providers.
Therefore, we create a parallell structure - RSA_PSS_PARAMS_30 - that
contains the same information, but uses numeric identities (*) and C
integers (**).  This makes it simpler to handle.

Note that neither this structure nor its contents are passed between
libcrypto and the providers.  Instead, the numeric identities are
translated to and from names, which are then passed over that
boundary.

For future considerations, we might consider dropping RSA_PSS_PARAMS
entirely.  For now, it's still reserved for EVP_PKEY_ASN1_METHOD code,
which RSA_PSS_PARAMS_30 is (almost entirely) reserved for use in our
providers.

(*) We use NIDs in this case, because we already have them and because
only algorithms that libcrypto knows about are permitted in PSS
restrictions.  We could use any number series we want, as long as we
know for sure what they represent.

(**) That's for saltlen and for trailerfield, which are never expect
to surpass the set of numbers that fit in a regular 'int'.

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11710)

1 files changed, 10 insertions, 1 deletions

diff --git a/crypto/rsa/rsa_lib.c b/crypto/rsa/rsa_lib.c
index 81daec4b34..d1fee590a8 100644
--- a/crypto/rsa/rsa_lib.c
+++ b/crypto/rsa/rsa_lib.c
@@ -162,7 +162,6 @@ void RSA_free(RSA *r)
     BN_clear_free(r->dmp1);
     BN_clear_free(r->dmq1);
     BN_clear_free(r->iqmp);
-    /* TODO(3.0): Support PSS in FIPS_MODULE */
 #ifndef FIPS_MODULE
     RSA_PSS_PARAMS_free(r->pss);
     sk_RSA_PRIME_INFO_pop_free(r->prime_infos, rsa_multip_info_free);
@@ -637,7 +636,17 @@ const BIGNUM *RSA_get0_iqmp(const RSA *r)
 
 const RSA_PSS_PARAMS *RSA_get0_pss_params(const RSA *r)
 {
+#ifdef FIPS_MODULE
+    return NULL;
+#else
     return r->pss;
+#endif
+}
+
+/* Internal */
+RSA_PSS_PARAMS_30 *rsa_get0_pss_params_30(RSA *r)
+{
+    return &r->pss_params;
 }
 
 void RSA_clear_flags(RSA *r, int flags)