Add multiple fixes for ffc key generation using invalid p,q,g parameters.

Fixes #11864

- The dsa keygen assumed valid p, q, g values were being passed. If this is not correct then it is
  possible that dsa keygen can either hang or segfault.
  The fix was to do a partial validation of p, q, and g inside the keygen.
- Fixed a potential double free in the dsa keypair test in the case when in failed (It should never fail!).
  It freed internal object members without setting them to NULL.
- Changed the FFC key validation to accept 1024 bit keys in non fips mode.
- Added tests that use both the default provider & fips provider to test these cases.

Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/12176)

1 files changed, 4 insertions, 0 deletions

diff --git a/crypto/dh/dh_key.c b/crypto/dh/dh_key.c
index 5d2acca25c..3b4da19cd2 100644
--- a/crypto/dh/dh_key.c
+++ b/crypto/dh/dh_key.c
@@ -287,6 +287,10 @@ static int generate_key(DH *dh)
             } else
 #endif
             {
+                /* Do a partial check for invalid p, q, g */
+                if (!ffc_params_simple_validate(dh->libctx, &dh->params,
+                                                FFC_PARAM_TYPE_DH))
+                    goto err;
                 /*
                  * For FFC FIPS 186-4 keygen
                  * security strength s = 112,