Stop raising ERR_R_MALLOC_FAILURE in most places

Since OPENSSL_malloc() and friends report ERR_R_MALLOC_FAILURE, and
at least handle the file name and line number they are called from,
there's no need to report ERR_R_MALLOC_FAILURE where they are called
directly, or when SSLfatal() and RLAYERfatal() is used, the reason
`ERR_R_MALLOC_FAILURE` is changed to `ERR_R_CRYPTO_LIB`.

There were a number of places where `ERR_R_MALLOC_FAILURE` was reported
even though it was a function from a different sub-system that was
called.  Those places are changed to report ERR_R_{lib}_LIB, where
{lib} is the name of that sub-system.
Some of them are tricky to get right, as we have a lot of functions
that belong in the ASN1 sub-system, and all the `sk_` calls or from
the CRYPTO sub-system.

Some extra adaptation was necessary where there were custom OPENSSL_malloc()
wrappers, and some bugs are fixed alongside these changes.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/19301)

1 files changed, 3 insertions, 10 deletions

diff --git a/ssl/priority_queue.c b/ssl/priority_queue.c
index 0f864a0ffa..67e85df73f 100644
--- a/ssl/priority_queue.c
+++ b/ssl/priority_queue.c
@@ -305,17 +305,13 @@ int ossl_pqueue_reserve(OSSL_PQUEUE *pq, size_t n)
     }
 
     h = OPENSSL_realloc(pq->heap, new_max * sizeof(*pq->heap));
-    if (h == NULL) {
-        ERR_raise(ERR_LIB_SSL, ERR_R_MALLOC_FAILURE);
+    if (h == NULL)
         return 0;
-    }
     pq->heap = h;
 
     e = OPENSSL_realloc(pq->elements, new_max * sizeof(*pq->elements));
-    if (e == NULL) {
-        ERR_raise(ERR_LIB_SSL, ERR_R_MALLOC_FAILURE);
+    if (e == NULL)
         return 0;
-    }
     pq->elements = e;
 
     pq->hmax = new_max;
@@ -331,10 +327,8 @@ OSSL_PQUEUE *ossl_pqueue_new(int (*compare)(const void *, const void *))
         return NULL;
 
     pq = OPENSSL_malloc(sizeof(*pq));
-    if (pq == NULL) {
-        ERR_raise(ERR_LIB_SSL, ERR_R_MALLOC_FAILURE);
+    if (pq == NULL)
         return NULL;
-    }
     pq->compare = compare;
     pq->hmax = min_nodes;
     pq->htop = 0;
@@ -343,7 +337,6 @@ OSSL_PQUEUE *ossl_pqueue_new(int (*compare)(const void *, const void *))
     pq->elements = OPENSSL_malloc(sizeof(*pq->elements) * min_nodes);
     if (pq->heap == NULL || pq->elements == NULL) {
         ossl_pqueue_free(pq);
-        ERR_raise(ERR_LIB_SSL, ERR_R_MALLOC_FAILURE);
         return NULL;
     }
     pqueue_add_freelist(pq, 0);