diff options
author | Bernd Edlinger <bernd.edlinger@hotmail.de> | 2020-05-25 20:13:47 +0200 |
---|---|---|
committer | Bernd Edlinger <bernd.edlinger@hotmail.de> | 2020-05-29 15:23:33 +0200 |
commit | 082c041b4233b17b80129d4ac6b33a28014442b0 (patch) | |
tree | 50e3c3bbe6b4d3a5d93ecea0c144d457b3323a55 /crypto | |
parent | f438f53a4e57462216be271c1c965550b6ff9941 (diff) |
bio printf: Avoid using rounding errors in range check
There is a problem casting ULONG_MAX to double which clang-10 is warning about.
ULONG_MAX typically cannot be exactly represented as a double. ULONG_MAX + 1
can be and this fix uses the latter, however since ULONG_MAX cannot be
represented exactly as a double number we subtract 65535 from this number,
and the result has at most 48 leading one bits, and can therefore be
represented as a double integer without rounding error. By adding
65536.0 to this number we achive the correct result, which should avoid the
warning.
The addresses a symptom of the underlying problem: we print doubles via an
unsigned long integer. Doubles have a far greater range and should be printed
better.
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11955)
Diffstat (limited to 'crypto')
-rw-r--r-- | crypto/bio/b_print.c | 8 |
1 files changed, 7 insertions, 1 deletions
diff --git a/crypto/bio/b_print.c b/crypto/bio/b_print.c index 0d6fafcc2d..6b995f8233 100644 --- a/crypto/bio/b_print.c +++ b/crypto/bio/b_print.c @@ -635,7 +635,13 @@ fmtfp(char **sbuffer, fvalue = tmpvalue; } ufvalue = abs_val(fvalue); - if (ufvalue > ULONG_MAX) { + /* + * By subtracting 65535 (2^16-1) we cancel the low order 15 bits + * of ULONG_MAX to avoid using imprecise floating point values. + * The second condition is necessary to catch NaN values. + */ + if (ufvalue >= (double)(ULONG_MAX - 65535) + 65536.0 + || !(ufvalue == ufvalue) /* NaN */) { /* Number too big */ return 0; } |