Fix handling of NULL sig parameter in ECDSA_sign and similar

The problem is, that it almost works to pass sig=NULL to the
ECDSA_sign, ECDSA_sign_ex and DSA_sign, to compute the necessary
space for the resulting signature.
But since the ECDSA signature is non-deterministic
(except when ECDSA_sign_setup/ECDSA_sign_ex are used)
the resulting length may be different when the API is called again.
This can easily cause random memory corruption.
Several internal APIs had the same issue, but since they are
never called with sig=NULL, it is better to make them return an
error in that case, instead of making the code more complex.

Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23529)

(cherry picked from commit 1fa2bf9b1885d2e87524421fea5041d40149cffa)

1 files changed, 5 insertions, 0 deletions

diff --git a/crypto/ec/ecdsa_ossl.c b/crypto/ec/ecdsa_ossl.c
index 0bf4635e2f..0bdf45e6e7 100644
--- a/crypto/ec/ecdsa_ossl.c
+++ b/crypto/ec/ecdsa_ossl.c
@@ -70,6 +70,11 @@ int ossl_ecdsa_sign(int type, const unsigned char *dgst, int dlen,
 {
     ECDSA_SIG *s;
 
+    if (sig == NULL && (kinv == NULL || r == NULL)) {
+        *siglen = ECDSA_size(eckey);
+        return 1;
+    }
+
     s = ECDSA_do_sign_ex(dgst, dlen, kinv, r, eckey);
     if (s == NULL) {
         *siglen = 0;