diff options
author | Bernd Edlinger <bernd.edlinger@hotmail.de> | 2020-05-25 20:13:47 +0200 |
---|---|---|
committer | Bernd Edlinger <bernd.edlinger@hotmail.de> | 2020-05-29 15:24:36 +0200 |
commit | 7d76c1fa0d6cd085419cb4cfadad8cfdfd24ce1f (patch) | |
tree | d2847aafbdf6815e6eb0c0ce6ef81fec23a177b8 /crypto/bio | |
parent | 3c09a5b0ba78a15311252ab8b7fb3ce16e7109ca (diff) |
bio printf: Avoid using rounding errors in range check
There is a problem casting ULONG_MAX to double which clang-10 is warning about.
ULONG_MAX typically cannot be exactly represented as a double. ULONG_MAX + 1
can be and this fix uses the latter, however since ULONG_MAX cannot be
represented exactly as a double number we subtract 65535 from this number,
and the result has at most 48 leading one bits, and can therefore be
represented as a double integer without rounding error. By adding
65536.0 to this number we achive the correct result, which should avoid the
warning.
The addresses a symptom of the underlying problem: we print doubles via an
unsigned long integer. Doubles have a far greater range and should be printed
better.
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/11955)
(cherry picked from commit 082c041b4233b17b80129d4ac6b33a28014442b0)
Diffstat (limited to 'crypto/bio')
-rw-r--r-- | crypto/bio/b_print.c | 8 |
1 files changed, 7 insertions, 1 deletions
diff --git a/crypto/bio/b_print.c b/crypto/bio/b_print.c index 8ef90ac1d4..48556f72bc 100644 --- a/crypto/bio/b_print.c +++ b/crypto/bio/b_print.c @@ -635,7 +635,13 @@ fmtfp(char **sbuffer, fvalue = tmpvalue; } ufvalue = abs_val(fvalue); - if (ufvalue > ULONG_MAX) { + /* + * By subtracting 65535 (2^16-1) we cancel the low order 15 bits + * of ULONG_MAX to avoid using imprecise floating point values. + * The second condition is necessary to catch NaN values. + */ + if (ufvalue >= (double)(ULONG_MAX - 65535) + 65536.0 + || !(ufvalue == ufvalue) /* NaN */) { /* Number too big */ return 0; } |