signature: Clamp PSS salt len to MD len

FIPS 186-4 section 5 "The RSA Digital Signature Algorithm", subsection
5.5 "PKCS #1" says: "For RSASSA-PSS […] the length (in bytes) of the
salt (sLen) shall satisfy 0 <= sLen <= hLen, where hLen is the length of
the hash function output block (in bytes)."

Introduce a new option RSA_PSS_SALTLEN_AUTO_DIGEST_MAX and make it the
default. The new value will behave like RSA_PSS_SALTLEN_AUTO, but will
not use more than the digest length when signing, so that FIPS 186-4 is
not violated. This value has two advantages when compared with
RSA_PSS_SALTLEN_DIGEST: (1) It will continue to do auto-detection when
verifying signatures for maximum compatibility, where
RSA_PSS_SALTLEN_DIGEST would fail for other digest sizes. (2) It will
work for combinations where the maximum salt length is smaller than the
digest size, which typically happens with large digest sizes (e.g.,
SHA-512) and small RSA keys.

J.-S. Coron shows in "Optimal Security Proofs for PSS and Other
Signature Schemes. Advances in Cryptology – Eurocrypt 2002, volume 2332
of Lecture Notes in Computer Science, pp. 272 – 287. Springer Verlag,
2002." that longer salts than the output size of modern hash functions
do not increase security: "For example,for an application in which at
most one billion signatures will be generated, k0 = 30 bits of random
salt are actually sufficient to guarantee the same level of security as
RSA, and taking a larger salt does not increase the security level."

Signed-off-by: Clemens Lang <cllang@redhat.com>

Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/19724)

2 files changed, 14 insertions, 2 deletions

diff --git a/doc/man3/EVP_PKEY_CTX_ctrl.pod b/doc/man3/EVP_PKEY_CTX_ctrl.pod
index 3075eaafd6..9b96f42dbc 100644
--- a/doc/man3/EVP_PKEY_CTX_ctrl.pod
+++ b/doc/man3/EVP_PKEY_CTX_ctrl.pod
@@ -270,8 +270,8 @@ EVP_PKEY_CTX_get_rsa_padding() gets the RSA padding mode for I<ctx>.
 
 EVP_PKEY_CTX_set_rsa_pss_saltlen() sets the RSA PSS salt length to I<saltlen>.
 As its name implies it is only supported for PSS padding. If this function is
-not called then the maximum salt length is used when signing and auto detection
-when verifying. Three special values are supported:
+not called then the salt length is maximized up to the digest length when
+signing and auto detection when verifying. Four special values are supported:
 
 =over 4
 
@@ -289,6 +289,13 @@ causes the salt length to be automatically determined based on the
 B<PSS> block structure when verifying.  When signing, it has the same
 meaning as B<RSA_PSS_SALTLEN_MAX>.
 
+=item B<RSA_PSS_SALTLEN_AUTO_DIGEST_MAX>
+
+causes the salt length to be automatically determined based on the B<PSS> block
+structure when verifying, like B<RSA_PSS_SALTLEN_AUTO>.  When signing, the salt
+length is maximized up to a maximum of the digest length to comply with FIPS
+186-4 section 5.5.
+
 =back
 
 EVP_PKEY_CTX_get_rsa_pss_saltlen() gets the RSA PSS salt length for I<ctx>.
diff --git a/doc/man7/EVP_SIGNATURE-RSA.pod b/doc/man7/EVP_SIGNATURE-RSA.pod
index 440e1c634f..de6869786c 100644
--- a/doc/man7/EVP_SIGNATURE-RSA.pod
+++ b/doc/man7/EVP_SIGNATURE-RSA.pod
@@ -68,6 +68,11 @@ Use the maximum salt length.
 
 Auto detect the salt length.
 
+=item "auto-digestmax" (B<OSSL_PKEY_RSA_PSS_SALT_LEN_AUTO_DIGEST_MAX>)
+
+Auto detect the salt length when verifying.  Maximize the salt length up to the
+digest size when signing to comply with FIPS 186-4 section 5.5.
+
 =back
 
 =back