RSA: Add a less loaded PSS-parameter structure

RSA_PSS_PARAMS carries with it a lot of baggage in form of X509_ALGOR
and ASN1_INTEGER, which we would rather avoid in our providers.
Therefore, we create a parallell structure - RSA_PSS_PARAMS_30 - that
contains the same information, but uses numeric identities (*) and C
integers (**).  This makes it simpler to handle.

Note that neither this structure nor its contents are passed between
libcrypto and the providers.  Instead, the numeric identities are
translated to and from names, which are then passed over that
boundary.

For future considerations, we might consider dropping RSA_PSS_PARAMS
entirely.  For now, it's still reserved for EVP_PKEY_ASN1_METHOD code,
which RSA_PSS_PARAMS_30 is (almost entirely) reserved for use in our
providers.

(*) We use NIDs in this case, because we already have them and because
only algorithms that libcrypto knows about are permitted in PSS
restrictions.  We could use any number series we want, as long as we
know for sure what they represent.

(**) That's for saltlen and for trailerfield, which are never expect
to surpass the set of numbers that fit in a regular 'int'.

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11710)

1 files changed, 12 insertions, 1 deletions

diff --git a/crypto/rsa/rsa_local.h b/crypto/rsa/rsa_local.h
index 6c4ae8611b..f94fc79cdd 100644
--- a/crypto/rsa/rsa_local.h
+++ b/crypto/rsa/rsa_local.h
@@ -12,6 +12,7 @@
 
 #include <openssl/rsa.h>
 #include "internal/refcount.h"
+#include "crypto/rsa.h"
 
 #define RSA_MAX_PRIME_NUM       5
 #define RSA_MIN_MODULUS_BITS    512
@@ -50,8 +51,18 @@ struct rsa_st {
     BIGNUM *dmp1;
     BIGNUM *dmq1;
     BIGNUM *iqmp;
-    /* If a PSS only key this contains the parameter restrictions */
+
+    /*
+     * If a PSS only key this contains the parameter restrictions.
+     * There are two structures for the same thing, used in different cases.
+     */
+    /* This is used uniquely by OpenSSL provider implementations. */
+    RSA_PSS_PARAMS_30 pss_params;
+#ifndef FIPS_MODULE
+    /* This is used uniquely by rsa_ameth.c and rsa_pmeth.c. */
     RSA_PSS_PARAMS *pss;
+#endif
+
 #ifndef FIPS_MODULE
     /* for multi-prime RSA, defined in RFC 8017 */
     STACK_OF(RSA_PRIME_INFO) *prime_infos;