Fix OSSL_PARAM_set_BN() to fill the given buffer correctly.

OSSL_PARAM_set_BN() filled the buffer from the left with as many bytes
as that the BIGNUM takes, regardless of buffer size or native
endianness.  This was due to BN_bn2nativepad() being given the size of
the BIGNUM rather than the size of the buffer (which meant it never
had to pad anything).

The fix is to given BN_bn2nativepad() the size of the buffer instead.
This aligns well with the corresponding _set_ functions for native
integer types work.

Reviewed-by: Patrick Steuer <patrick.steuer@de.ibm.com>
(Merged from https://github.com/openssl/openssl/pull/10326)

1 files changed, 5 insertions, 2 deletions

diff --git a/crypto/params.c b/crypto/params.c
index b2ceb13278..0cd13e3b81 100644
--- a/crypto/params.c
+++ b/crypto/params.c
@@ -640,8 +640,11 @@ int OSSL_PARAM_set_BN(OSSL_PARAM *p, const BIGNUM *val)
     p->return_size = bytes;
     if (p->data == NULL)
         return 1;
-    return p->data_size >= bytes
-        && BN_bn2nativepad(val, p->data, bytes) >= 0;
+    if (p->data_size >= bytes) {
+        p->return_size = p->data_size;
+        return BN_bn2nativepad(val, p->data, p->data_size) >= 0;
+    }
+    return 0;
 }
 
 OSSL_PARAM OSSL_PARAM_construct_BN(const char *key, unsigned char *buf,