CTR, HASH and HMAC DRBGs in provider

Move the three different DRBGs to the provider.

As part of the move, the DRBG specific data was pulled out of a common
structure and into their own structures.  Only these smaller structures are
securely allocated.  This saves quite a bit of secure memory:

    +-------------------------------+
    | DRBG         | Bytes | Secure |
    +--------------+-------+--------+
    | HASH         |  376  |   512  |
    | HMAC         |  168  |   256  |
    | CTR          |  176  |   256  |
    | Common (new) |  320  |     0  |
    | Common (old) |  592  |  1024  |
    +--------------+-------+--------+

Bytes is the structure size on the X86/64.
Secure is the number of bytes of secure memory used (power of two allocator).

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
(Merged from https://github.com/openssl/openssl/pull/11682)

2 files changed, 30 insertions, 16 deletions

diff --git a/doc/man3/RAND_DRBG_new.pod b/doc/man3/RAND_DRBG_new.pod
index 3ff98ae052..cd770fd673 100644
--- a/doc/man3/RAND_DRBG_new.pod
+++ b/doc/man3/RAND_DRBG_new.pod
@@ -35,9 +35,6 @@ RAND_DRBG_free
                                  unsigned int flags,
                                  RAND_DRBG *parent);
 
- int RAND_DRBG_set(RAND_DRBG *drbg,
-                   int type, unsigned int flags);
-
  int RAND_DRBG_set_defaults(int type, unsigned int flags);
 
  int RAND_DRBG_instantiate(RAND_DRBG *drbg,
@@ -47,18 +44,27 @@ RAND_DRBG_free
 
  void RAND_DRBG_free(RAND_DRBG *drbg);
 
+Deprecated since OpenSSL 3.0, can be hidden entirely by defining
+B<OPENSSL_API_COMPAT> with a suitable version value, see
+L<openssl_user_macros(7)>:
+
+ int RAND_DRBG_set(RAND_DRBG *drbg,
+                   int type, unsigned int flags);
 
 =head1 DESCRIPTION
 
-RAND_DRBG_new_ex() and RAND_DRBG_secure_new_ex()
-create a new DRBG instance of the given B<type>, allocated from the heap resp.
-the secure heap, for the given OPENSSL_CTX <ctx>
-(using OPENSSL_zalloc() resp. OPENSSL_secure_zalloc()). The <ctx> parameter can
-be NULL in which case the default OPENSSL_CTX is used. RAND_DRBG_new() and
-RAND_DRBG_secure_new() are the same as RAND_DRBG_new_ex() and
-RAND_DRBG_secure_new_ex() except that the default OPENSSL_CTX is always used.
+RAND_DRBG_new_ex() and RAND_DRBG_secure_new_ex() create a new DRBG instance
+of the given B<type> for the given OPENSSL_CTX <ctx>.
+The <ctx> parameter can be NULL in which case the default OPENSSL_CTX is used.
+RAND_DRBG_new() and RAND_DRBG_secure_new() are the same as RAND_DRBG_new_ex()
+and RAND_DRBG_secure_new_ex() except that the default OPENSSL_CTX is always
+used.
+As of OpenSSL 3.0, there is no different between the new and secure_new
+functions.
 
 RAND_DRBG_set() initializes the B<drbg> with the given B<type> and B<flags>.
+This function is deprecated.  Applications should instead use
+RAND_DRBG_new_ex() to create a new DRBG.
 
 RAND_DRBG_set_defaults() sets the default B<type> and B<flags> for new DRBG
 instances.
@@ -124,7 +130,7 @@ uninstantiated state.
 
 RAND_DRBG_new_ex(), RAND_DRBG_new(), RAND_DRBG_secure_new_ex() and
 RAND_DRBG_secure_new() return a pointer to a DRBG instance allocated on the
-heap, resp. secure heap.
+heap.
 
 RAND_DRBG_set(),
 RAND_DRBG_instantiate(), and
@@ -149,6 +155,11 @@ To ensure that they are applied to the global and thread-local DRBG instances
 RAND_DRBG_set_defaults() before creating any thread and before calling any
 cryptographic routines that obtain random data directly or indirectly.
 
+As of OpenSSL 3.0, RAND_DRBG_new() and RAND_DRBG_secure_new() are
+functionally identical.  The DRBG is allocated on the normal heap and its
+sensitive state is allocated on the secure heap.  Likewise for,
+RAND_DRBG_new_ex() and RAND_DRBG_secure_new_ex().
+
 =head1 SEE ALSO
 
 L<OPENSSL_zalloc(3)>,
@@ -158,6 +169,8 @@ L<RAND_DRBG(7)>
 
 =head1 HISTORY
 
+The RAND_DRBG_set() function was deprecated in OpenSSL 3.0.
+
 The RAND_DRBG functions were added in OpenSSL 1.1.1.
 
 =head1 COPYRIGHT
diff --git a/doc/man3/RAND_DRBG_set_callbacks.pod b/doc/man3/RAND_DRBG_set_callbacks.pod
index d00397da62..0ae3028a5a 100644
--- a/doc/man3/RAND_DRBG_set_callbacks.pod
+++ b/doc/man3/RAND_DRBG_set_callbacks.pod
@@ -127,11 +127,12 @@ entropy from a live entropy source (section 5.5.2 of [NIST SP 800-90C]).
 It is up to the user to ensure that a live entropy source is configured
 and is being used.
 
-The derivation function is disabled during initialization by calling the
-RAND_DRBG_set() function with the RAND_DRBG_FLAG_CTR_NO_DF flag.
-For more information on the derivation function and when it can be omitted,
-see [NIST SP 800-90A Rev. 1]. Roughly speaking it can be omitted if the random
-source has "full entropy", i.e., contains 8 bits of entropy per byte.
+The derivation function is disabled by calling the RAND_DRBG_new_ex()
+function with the RAND_DRBG_FLAG_CTR_NO_DF flag.  For more information on
+the derivation function and when it can be omitted, see [NIST SP 800-90A
+Rev. 1]. Roughly speaking it can be omitted if the random source has "full
+entropy", i.e., contains 8 bits of entropy per byte. In a FIPS context,
+the derivation function can never be omitted.
 
 Even if a nonce is required, the B<get_nonce>() and B<cleanup_nonce>()
 callbacks can be omitted by setting them to NULL.