rsa-pss: add tests checking for SHAKE usage in RSA-PSS

FIPS 186-5, RFC 8692, RFC 8702 all agree and specify that Shake shall
be used directly as MGF (not as a hash in MGF1). Add tests that try to
specify shake hash as MGF1 to ensure that fails.

Separately the above standards specify how to use SHAKE as a message
digest with either fixed or minimum output lengths. However, currently
shake is not part of allowed hashes.

Note that rsa_setup_md()/rsa_setup_mgf1_md() call
ossl_digest_rsa_sign_get_md_nid() ->
ossl_digest_get_approved_nid_with_sha1() ->
ossl_digest_get_approved_nid() which only contain sha1/sha2/sha3
digests without XOF.

The digest test case will need to be replace if/when shake with
minimum output lengths is added to ossl_digest_get_approved_nid().

Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/24387)

0 files changed, 0 insertions, 0 deletions