Support calling EVP_DigestUpdate instead of EVP_Digest[Sign|Verify]Update

Prior to OpenSSL 3.0 EVP_Digest[Sign|Verify|Update were just macros for
EVP_DigestUpdate. They are now separate functions. Unfortunately some
code assumes that EVP_Digest[Sign|Verify]Update is interchangeable with
EVP_DigestUpdate. For example the dgst app uses an MD bio which always
calls EVP_DigestUpdate(). However the dgst app supports signing instead
of digesting and may initialise with EVP_DigestSignInit_ex() instead of
just EVP_DigestInit().

We now detect these differences and redirect to the correct function
where appropriate.

Fixes #10114

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/10116)

1 files changed, 18 insertions, 0 deletions

diff --git a/crypto/evp/digest.c b/crypto/evp/digest.c
index fa1dfa7303..5ff43fdd64 100644
--- a/crypto/evp/digest.c
+++ b/crypto/evp/digest.c
@@ -285,6 +285,24 @@ int EVP_DigestUpdate(EVP_MD_CTX *ctx, const void *data, size_t count)
     if (count == 0)
         return 1;
 
+    if (ctx->pctx != NULL
+            && EVP_PKEY_CTX_IS_SIGNATURE_OP(ctx->pctx)
+            && ctx->pctx->op.sig.sigprovctx != NULL) {
+        /*
+         * Prior to OpenSSL 3.0 EVP_DigestSignUpdate() and
+         * EVP_DigestVerifyUpdate() were just macros for EVP_DigestUpdate().
+         * Some code calls EVP_DigestUpdate() directly even when initialised
+         * with EVP_DigestSignInit_ex() or EVP_DigestVerifyInit_ex(), so we
+         * detect that and redirect to the correct EVP_Digest*Update() function
+         */
+        if (ctx->pctx->operation == EVP_PKEY_OP_SIGNCTX)
+            return EVP_DigestSignUpdate(ctx, data, count);
+        if (ctx->pctx->operation == EVP_PKEY_OP_VERIFYCTX)
+            return EVP_DigestVerifyUpdate(ctx, data, count);
+        EVPerr(EVP_F_EVP_DIGESTUPDATE, EVP_R_UPDATE_ERROR);
+        return 0;
+    }
+
     if (ctx->digest == NULL || ctx->digest->prov == NULL)
         goto legacy;