Fix internal memory leaks from OPENSSL_MALLOC_FAILURES

There is a rarely used feature that can be enabled
with `./config enable-crypto-mdebug` when additionally
the environment variable OPENSSL_MALLOC_FAILURES is used.
It turns out to be possible that CRYPTO_zalloc may
create a leak when the memory is allocated and then
the shouldfail happens, then the memory is lost.
Likewise when OPENSSL_realloc is used with size=0,
then the memory is to be free'd but here the shouldfail
check is too early, and the failure may prevent the
memory to be freed thus creating a bogus memory leak.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/21944)

1 files changed, 1 insertions, 2 deletions

diff --git a/crypto/mem.c b/crypto/mem.c
index 74bf3b892c..b9fca98a83 100644
--- a/crypto/mem.c
+++ b/crypto/mem.c
@@ -214,7 +214,6 @@ void *CRYPTO_zalloc(size_t num, const char *file, int line)
     void *ret;
 
     ret = CRYPTO_malloc(num, file, line);
-    FAILTEST();
     if (ret != NULL)
         memset(ret, 0, num);
 
@@ -227,7 +226,6 @@ void *CRYPTO_realloc(void *str, size_t num, const char *file, int line)
     if (realloc_impl != CRYPTO_realloc)
         return realloc_impl(str, num, file, line);
 
-    FAILTEST();
     if (str == NULL)
         return CRYPTO_malloc(num, file, line);
 
@@ -236,6 +234,7 @@ void *CRYPTO_realloc(void *str, size_t num, const char *file, int line)
         return NULL;
     }
 
+    FAILTEST();
     return realloc(str, num);
 }