Pick a q size consistent with the digest for DSA param generation

There are two undocumented DSA parameter generation options available in
the genpkey command line app:
dsa_paramgen_md and dsa_paramgen_q_bits.

These can also be accessed via the EVP API but only by using
EVP_PKEY_CTX_ctrl() or EVP_PKEY_CTX_ctrl_str() directly. There are no
helper macros for these options.

dsa_paramgen_q_bits sets the length of q in bits (default 160 bits).
dsa_paramgen_md sets the digest that is used during the parameter
generation (default SHA1). In particular the output length of the digest
used must be equal to or greater than the number of bits in q because of
this code:

            if (!EVP_Digest(seed, qsize, md, NULL, evpmd, NULL))
                goto err;
            if (!EVP_Digest(buf, qsize, buf2, NULL, evpmd, NULL))
                goto err;
            for (i = 0; i < qsize; i++)
                md[i] ^= buf2[i];

            /* step 3 */
            md[0] |= 0x80;
            md[qsize - 1] |= 0x01;
            if (!BN_bin2bn(md, qsize, q))
                goto err;

qsize here is the number of bits in q and evpmd is the digest set via
dsa_paramgen_md. md and buf2 are buffers of length SHA256_DIGEST_LENGTH.
buf2 has been filled with qsize bits of random seed data, and md is
uninitialised.

If the output size of evpmd is less than qsize then the line "md[i] ^=
buf2[i]" will be xoring an uninitialised value and the random seed data
together to form the least significant bits of q (and not using the
output of the digest at all for those bits) - which is probably not what
was intended. The same seed is then used as an input to generating p. If
the uninitialised data is actually all zeros (as seems quite likely)
then the least significant bits of q will exactly match the least
significant bits of the seed.

This problem only occurs if you use these undocumented and difficult to
find options and you set the size of q to be greater than the message
digest output size. This is for parameter generation only not key
generation. This scenario is considered highly unlikely and
therefore the security risk of this is considered negligible.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/5884)

3 files changed, 13 insertions, 4 deletions

diff --git a/crypto/dsa/dsa.h b/crypto/dsa/dsa.h
index 545358fd02..7f8346d72d 100644
--- a/crypto/dsa/dsa.h
+++ b/crypto/dsa/dsa.h
@@ -307,6 +307,7 @@ void ERR_load_DSA_strings(void);
 # define DSA_F_I2D_DSA_SIG                                111
 # define DSA_F_OLD_DSA_PRIV_DECODE                        122
 # define DSA_F_PKEY_DSA_CTRL                              120
+# define DSA_F_PKEY_DSA_CTRL_STR                          127
 # define DSA_F_PKEY_DSA_KEYGEN                            121
 # define DSA_F_SIG_CB                                     114
 
diff --git a/crypto/dsa/dsa_err.c b/crypto/dsa/dsa_err.c
index f5ddc66b8a..7e507faa36 100644
--- a/crypto/dsa/dsa_err.c
+++ b/crypto/dsa/dsa_err.c
@@ -1,6 +1,6 @@
 /* crypto/dsa/dsa_err.c */
 /* ====================================================================
- * Copyright (c) 1999-2013 The OpenSSL Project.  All rights reserved.
+ * Copyright (c) 1999-2018 The OpenSSL Project.  All rights reserved.
  *
  * Redistribution and use in source and binary forms, with or without
  * modification, are permitted provided that the following conditions
@@ -95,6 +95,7 @@ static ERR_STRING_DATA DSA_str_functs[] = {
     {ERR_FUNC(DSA_F_I2D_DSA_SIG), "i2d_DSA_SIG"},
     {ERR_FUNC(DSA_F_OLD_DSA_PRIV_DECODE), "OLD_DSA_PRIV_DECODE"},
     {ERR_FUNC(DSA_F_PKEY_DSA_CTRL), "PKEY_DSA_CTRL"},
+    {ERR_FUNC(DSA_F_PKEY_DSA_CTRL_STR), "PKEY_DSA_CTRL_STR"},
     {ERR_FUNC(DSA_F_PKEY_DSA_KEYGEN), "PKEY_DSA_KEYGEN"},
     {ERR_FUNC(DSA_F_SIG_CB), "SIG_CB"},
     {0, NULL}
diff --git a/crypto/dsa/dsa_gen.c b/crypto/dsa/dsa_gen.c
index 21af2e159f..db52a38a18 100644
--- a/crypto/dsa/dsa_gen.c
+++ b/crypto/dsa/dsa_gen.c
@@ -146,9 +146,16 @@ int dsa_builtin_paramgen(DSA *ret, size_t bits, size_t qbits,
         /* invalid q size */
         return 0;
 
-    if (evpmd == NULL)
-        /* use SHA1 as default */
-        evpmd = EVP_sha1();
+    if (evpmd == NULL) {
+        if (qsize == SHA_DIGEST_LENGTH)
+            evpmd = EVP_sha1();
+        else if (qsize == SHA224_DIGEST_LENGTH)
+            evpmd = EVP_sha224();
+        else
+            evpmd = EVP_sha256();
+    } else {
+        qsize = EVP_MD_size(evpmd);
+    }
 
     if (bits < 512)
         bits = 512;