Add multiple fixes for ffc key generation using invalid p,q,g parameters.

Fixes #11864

- The dsa keygen assumed valid p, q, g values were being passed. If this is not correct then it is
  possible that dsa keygen can either hang or segfault.
  The fix was to do a partial validation of p, q, and g inside the keygen.
- Fixed a potential double free in the dsa keypair test in the case when in failed (It should never fail!).
  It freed internal object members without setting them to NULL.
- Changed the FFC key validation to accept 1024 bit keys in non fips mode.
- Added tests that use both the default provider & fips provider to test these cases.

Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/12176)

1 files changed, 7 insertions, 0 deletions

diff --git a/crypto/dsa/dsa_key.c b/crypto/dsa/dsa_key.c
index 7bd9c5ff2e..b537ec0b3c 100644
--- a/crypto/dsa/dsa_key.c
+++ b/crypto/dsa/dsa_key.c
@@ -74,6 +74,11 @@ static int dsa_keygen(DSA *dsa, int pairwise_test)
         priv_key = dsa->priv_key;
     }
 
+    /* Do a partial check for invalid p, q, g */
+    if (!ffc_params_simple_validate(dsa->libctx, &dsa->params,
+                                    FFC_PARAM_TYPE_DSA))
+        goto err;
+
     /*
      * For FFC FIPS 186-4 keygen
      * security strength s = 112,
@@ -110,6 +115,8 @@ static int dsa_keygen(DSA *dsa, int pairwise_test)
         if (!ok) {
             BN_free(dsa->pub_key);
             BN_clear_free(dsa->priv_key);
+            dsa->pub_key = NULL;
+            dsa->priv_key = NULL;
             BN_CTX_free(ctx);
             return ok;
         }