Deprecate most of debug-memory

Fixes #8322

The leak-checking (and backtrace option, on some platforms) provided
by crypto-mdebug and crypto-mdebug-backtrace have been mostly neutered;
only the "make malloc fail" capability remains.  OpenSSL recommends using
the compiler's leak-detection instead.

The OPENSSL_DEBUG_MEMORY environment variable is no longer used.
CRYPTO_mem_ctrl(), CRYPTO_set_mem_debug(), CRYPTO_mem_leaks(),
CRYPTO_mem_leaks_fp() and CRYPTO_mem_leaks_cb() return a failure code.
CRYPTO_mem_debug_{malloc,realloc,free}() have been removed.  All of the
above are now deprecated.

Merge (now really small) mem_dbg.c into mem.c

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/10572)

2 files changed, 20 insertions, 53 deletions

diff --git a/doc/man3/OPENSSL_malloc.pod b/doc/man3/OPENSSL_malloc.pod
index 198251fcd7..8286b4aba8 100644
--- a/doc/man3/OPENSSL_malloc.pod
+++ b/doc/man3/OPENSSL_malloc.pod
@@ -60,20 +60,18 @@ OPENSSL_MALLOC_FD
 
  void CRYPTO_get_alloc_counts(int *m, int *r, int *f)
 
- int CRYPTO_set_mem_debug(int onoff)
-
  env OPENSSL_MALLOC_FAILURES=... <application>
  env OPENSSL_MALLOC_FD=... <application>
 
- int CRYPTO_mem_ctrl(int mode);
+Deprecated:
 
  int CRYPTO_mem_leaks(BIO *b);
  int CRYPTO_mem_leaks_fp(FILE *fp);
  int CRYPTO_mem_leaks_cb(int (*cb)(const char *str, size_t len, void *u),
                          void *u);
 
-Deprecated:
-
+ int CRYPTO_set_mem_debug(int onoff)
+ int CRYPTO_mem_ctrl(int mode);
  int OPENSSL_mem_debug_push(const char *info)
  int OPENSSL_mem_debug_pop(void);
  int CRYPTO_mem_debug_push(const char *info, const char *file, int line);
@@ -115,37 +113,15 @@ OPENSSL_strlcat() and OPENSSL_strnlen() are equivalents of the common C
 library functions and are provided for portability.
 
 If no allocations have been done, it is possible to "swap out" the default
-implementations for OPENSSL_malloc(), OPENSSL_realloc and OPENSSL_free()
-and replace them with alternate versions (hooks).
+implementations for OPENSSL_malloc(), OPENSSL_realloc() and OPENSSL_free()
+and replace them with alternate versions.
 CRYPTO_get_mem_functions() function fills in the given arguments with the
 function pointers for the current implementations.
 With CRYPTO_set_mem_functions(), you can specify a different set of functions.
 If any of B<m>, B<r>, or B<f> are NULL, then the function is not changed.
-
-The default implementation can include some debugging capability (if enabled
-at build-time).
-This adds some overhead by keeping a list of all memory allocations, and
-removes items from the list when they are free'd.
-This is most useful for identifying memory leaks.
-CRYPTO_set_mem_debug() turns this tracking on and off.  In order to have
-any effect, is must be called before any of the allocation functions
-(e.g., CRYPTO_malloc()) are called, and is therefore normally one of the
-first lines of main() in an application.
-CRYPTO_mem_ctrl() provides fine-grained control of memory leak tracking.
-To enable tracking call CRYPTO_mem_ctrl() with a B<mode> argument of
-the B<CRYPTO_MEM_CHECK_ON>.
-To disable tracking call CRYPTO_mem_ctrl() with a B<mode> argument of
-the B<CRYPTO_MEM_CHECK_OFF>.
-
-At the end of the program, calling CRYPTO_mem_leaks() or
-CRYPTO_mem_leaks_fp() will report all "leaked" memory, writing it
-to the specified BIO B<b> or FILE B<fp>. These functions return 1 if
-there are no leaks, 0 if there are leaks and -1 if an error occurred.
-
-CRYPTO_mem_leaks_cb() does the same as CRYPTO_mem_leaks(), but instead
-of writing to a given BIO, the callback function is called for each
-output string with the string, length, and userdata B<u> as the callback
-parameters.
+While it's permitted to swap out only a few and not all the functions
+with CRYPTO_set_mem_functions(), it's recommended to swap them all out
+at once.
 
 If the library is built with the C<crypto-mdebug> option, then one
 function, CRYPTO_get_alloc_counts(), and two additional environment
@@ -180,19 +156,12 @@ to use this (will not work on all platforms):
   export OPENSSL_MALLOC_FD
   ...app invocation... 3>/tmp/log$$
 
-OPENSSL_mem_debug_push(), OPENSSL_mem_debug_pop(),
-CRYPTO_mem_debug_push(), and CRYPTO_mem_debug_pop()
-have been deprecated and replaced with functions that only return zero.
-
 =head1 RETURN VALUES
 
 OPENSSL_malloc_init(), OPENSSL_free(), OPENSSL_clear_free()
 CRYPTO_free(), CRYPTO_clear_free() and CRYPTO_get_mem_functions()
 return no value.
 
-CRYPTO_mem_leaks(), CRYPTO_mem_leaks_fp() and CRYPTO_mem_leaks_cb() return 1 if
-there are no leaks, 0 if there are leaks and -1 if an error occurred.
-
 OPENSSL_malloc(), OPENSSL_zalloc(), OPENSSL_realloc(),
 OPENSSL_clear_realloc(),
 CRYPTO_malloc(), CRYPTO_zalloc(), CRYPTO_realloc(),
@@ -200,25 +169,24 @@ CRYPTO_clear_realloc(),
 OPENSSL_strdup(), and OPENSSL_strndup()
 return a pointer to allocated memory or NULL on error.
 
-CRYPTO_set_mem_functions() and CRYPTO_set_mem_debug()
-return 1 on success or 0 on failure (almost
+CRYPTO_set_mem_functions() returns 1 on success or 0 on failure (almost
 always because allocations have already happened).
 
-CRYPTO_mem_ctrl() returns -1 if an error occurred, otherwise the
-previous value of the mode.
-
-=head1 NOTES
-
-While it's permitted to swap out only a few and not all the functions
-with CRYPTO_set_mem_functions(), it's recommended to swap them all out
-at once, especially if OpenSSL was built with the
-configuration option> C<crypto-mdebug>.
+CRYPTO_mem_leaks(), CRYPTO_mem_leaks_fp(), CRYPTO_mem_leaks_cb(),
+CRYPTO_set_mem_debug(), and CRYPTO_mem_ctrl() are deprecated and return -1.
+OPENSSL_mem_debug_push(), OPENSSL_mem_debug_pop(),
+CRYPTO_mem_debug_push(), and CRYPTO_mem_debug_pop()
+are deprecated and return 0.
 
 =head1 HISTORY
 
 OPENSSL_mem_debug_push(), OPENSSL_mem_debug_pop(),
-CRYPTO_mem_debug_push(), and CRYPTO_mem_debug_pop()
+CRYPTO_mem_debug_push(), CRYPTO_mem_debug_pop(),
+CRYPTO_mem_leaks(), CRYPTO_mem_leaks_fp(),
+CRYPTO_mem_leaks_cb(), CRYPTO_set_mem_debug(), CRYPTO_mem_ctrl()
 were deprecated in OpenSSL 3.0.
+The memory-leak checking has been deprecated in OpenSSL 3.0 in favor of
+clang's memory and leak sanitizer.
 
 
 =head1 COPYRIGHT
diff --git a/doc/man7/provider-base.pod b/doc/man7/provider-base.pod
index 2dc29b37bc..ccc5bf309a 100644
--- a/doc/man7/provider-base.pod
+++ b/doc/man7/provider-base.pod
@@ -108,7 +108,6 @@ provider):
  CRYPTO_secure_free             OSSL_FUNC_CRYPTO_SECURE_FREE
  CRYPTO_secure_clear_free       OSSL_FUNC_CRYPTO_SECURE_CLEAR_FREE
  CRYPTO_secure_allocated        OSSL_FUNC_CRYPTO_SECURE_ALLOCATED
- CRYPTO_mem_ctrl                OSSL_FUNC_CRYPTO_MEM_CTRL
  BIO_new_file                   OSSL_FUNC_BIO_NEW_FILE
  BIO_new_mem_buf                OSSL_FUNC_BIO_NEW_MEMBUF
  BIO_read_ex                    OSSL_FUNC_BIO_READ_EX
@@ -183,7 +182,7 @@ CRYPTO_malloc(), CRYPTO_zalloc(), CRYPTO_memdup(), CRYPTO_strdup(),
 CRYPTO_strndup(), CRYPTO_free(), CRYPTO_clear_free(),
 CRYPTO_realloc(), CRYPTO_clear_realloc(), CRYPTO_secure_malloc(),
 CRYPTO_secure_zalloc(), CRYPTO_secure_free(),
-CRYPTO_secure_clear_free(), CRYPTO_secure_allocated(), CRYPTO_mem_ctrl(),
+CRYPTO_secure_clear_free(), CRYPTO_secure_allocated(),
 BIO_new_file(), BIO_new_mem_buf(), BIO_read_ex(), BIO_free(),
 BIO_vprintf(), OPENSSL_cleanse(), and OPENSSL_hexstr2buf()
 correspond exactly to the public functions with the same name.