Add BN_check_prime()

Add a new API to test for primes that can't be misused, deprecated the
old APIs.

Suggested by Jake Massimo and Kenneth Paterson

Reviewed-by: Paul Dale <paul.dale@oracle.com>
GH: #9272

1 files changed, 57 insertions, 39 deletions

diff --git a/doc/man3/BN_generate_prime.pod b/doc/man3/BN_generate_prime.pod
index 6a3376b1fd..cdd7ed0e8e 100644
--- a/doc/man3/BN_generate_prime.pod
+++ b/doc/man3/BN_generate_prime.pod
@@ -2,7 +2,7 @@
 
 =head1 NAME
 
-BN_generate_prime_ex2, BN_generate_prime_ex, BN_is_prime_ex,
+BN_generate_prime_ex2, BN_generate_prime_ex, BN_is_prime_ex, BN_check_prime,
 BN_is_prime_fasttest_ex, BN_GENCB_call, BN_GENCB_new, BN_GENCB_free,
 BN_GENCB_set_old, BN_GENCB_set, BN_GENCB_get_arg, BN_generate_prime,
 BN_is_prime, BN_is_prime_fasttest - generate primes and test for primality
@@ -18,10 +18,7 @@ BN_is_prime, BN_is_prime_fasttest - generate primes and test for primality
  int BN_generate_prime_ex(BIGNUM *ret, int bits, int safe, const BIGNUM *add,
                           const BIGNUM *rem, BN_GENCB *cb);
 
- int BN_is_prime_ex(const BIGNUM *p, int nchecks, BN_CTX *ctx, BN_GENCB *cb);
-
- int BN_is_prime_fasttest_ex(const BIGNUM *p, int nchecks, BN_CTX *ctx,
-                             int do_trial_division, BN_GENCB *cb);
+ int BN_check_prime(const BIGNUM *p, BN_CTX *ctx, BN_GENCB *cb);
 
  int BN_GENCB_call(BN_GENCB *cb, int a, int b);
 
@@ -45,19 +42,32 @@ L<openssl_user_macros(7)>:
                            BIGNUM *rem, void (*callback)(int, int, void *),
                            void *cb_arg);
 
- int BN_is_prime(const BIGNUM *a, int checks,
+ int BN_is_prime(const BIGNUM *p, int nchecks,
                  void (*callback)(int, int, void *), BN_CTX *ctx, void *cb_arg);
 
- int BN_is_prime_fasttest(const BIGNUM *a, int checks,
+ int BN_is_prime_fasttest(const BIGNUM *p, int nchecks,
                           void (*callback)(int, int, void *), BN_CTX *ctx,
                           void *cb_arg, int do_trial_division);
 
+Deprecated since OpenSSL 3.0:
+
+ int BN_is_prime_ex(const BIGNUM *p, int nchecks, BN_CTX *ctx, BN_GENCB *cb);
+
+ int BN_is_prime_fasttest_ex(const BIGNUM *p, int nchecks, BN_CTX *ctx,
+                             int do_trial_division, BN_GENCB *cb);
+
 =head1 DESCRIPTION
 
 BN_generate_prime_ex2() generates a pseudo-random prime number of
 at least bit length B<bits> using the BN_CTX provided in B<ctx>. The value of
 B<ctx> must not be NULL.
+
 The returned number is probably prime with a negligible error.
+The maximum error rate is 2^-128.
+It's 2^-287 for a 512 bit prime, 2^-435 for a 1024 bit prime,
+2^-648 for a 2048 bit prime, and lower than 2^-882 for primes larger
+than 2048 bit.
+
 If B<add> is B<NULL> the returned prime number will have exact bit
 length B<bits> with the top most two bits set.
 
@@ -111,37 +121,43 @@ B<ctx> parameter is passed.
 In this case the random number generator associated with the default OPENSSL_CTX
 will be used.
 
-BN_is_prime_ex() and BN_is_prime_fasttest_ex() test if the number B<p> is
-prime.  The following tests are performed until one of them shows that
-B<p> is composite; if B<p> passes all these tests, it is considered
-prime.
-
-BN_is_prime_fasttest_ex(), when called with B<do_trial_division == 1>,
-first attempts trial division by a number of small primes;
-if no divisors are found by this test and B<cb> is not B<NULL>,
-B<BN_GENCB_call(cb, 1, -1)> is called.
-If B<do_trial_division == 0>, this test is skipped.
-
-Both BN_is_prime_ex() and BN_is_prime_fasttest_ex() perform a Miller-Rabin
-probabilistic primality test with B<nchecks> iterations. If
-B<nchecks == BN_prime_checks>, a number of iterations is used that
-yields a false positive rate of at most 2^-64 for random input.
-The error rate depends on the size of the prime and goes down for bigger primes.
-The rate is 2^-80 starting at 308 bits, 2^-112 at 852 bits, 2^-128 at 1080 bits,
-2^-192 at 3747 bits and 2^-256 at 6394 bits.
-
-When the source of the prime is not random or not trusted, the number
-of checks needs to be much higher to reach the same level of assurance:
-It should equal half of the targeted security level in bits (rounded up to the
-next integer if necessary).
-For instance, to reach the 128 bit security level, B<nchecks> should be set to
-64.
-
-If B<cb> is not B<NULL>, B<BN_GENCB_call(cb, 1, j)> is called
-after the j-th iteration (j = 0, 1, ...). B<ctx> is a
-pre-allocated B<BN_CTX> (to save the overhead of allocating and
+BN_check_prime(), BN_is_prime_ex(), BN_is_prime_fasttest_ex(), BN_is_prime()
+and BN_is_prime_fasttest() test if the number B<p> is prime.
+The functions tests until one of the tests shows that B<p> is composite,
+or all the tests passed.
+If B<p> passes all these tests, it is considered a probable prime.
+
+The test performed on B<p> are trial division by a number of small primes
+and rounds of the of the Miller-Rabin probabilistic primality test.
+
+The functions do at least 64 rounds of the Miller-Rabin test giving a maximum
+false positive rate of 2^-128.
+If the size of B<p> is more than 2048 bits, they do at least 128 rounds
+giving a maximum false positive rate of 2^-256.
+
+If B<nchecks> is larger than the minimum above (64 or 128), B<nchecks>
+rounds of the Miller-Rabin test will be done.
+
+If B<do_trial_division> set to B<0>, the trial division will be skipped.
+BN_is_prime_ex() and BN_is_prime() always skip the trial division.
+
+BN_is_prime_ex(), BN_is_prime_fasttest_ex(), BN_is_prime()
+and BN_is_prime_fasttest() are deprecated.
+
+BN_is_prime_fasttest() and BN_is_prime() behave just like
+BN_is_prime_fasttest_ex() and BN_is_prime_ex() respectively, but with the old
+style call back.
+
+B<ctx> is a pre-allocated B<BN_CTX> (to save the overhead of allocating and
 freeing the structure in a loop), or B<NULL>.
 
+If the trial division is done, and no divisors are found and B<cb>
+is not B<NULL>, B<BN_GENCB_call(cb, 1, -1)> is called.
+
+After each round of the Miller-Rabin probabilistic primality test,
+if B<cb> is not B<NULL>, B<BN_GENCB_call(cb, 1, j)> is called
+with B<j> the iteration (j = 0, 1, ...).
+
 BN_GENCB_call() calls the callback function held in the B<BN_GENCB> structure
 and passes the ints B<a> and B<b> as arguments. There are two types of
 B<BN_GENCB> structure that are supported: "new" style and "old" style. New
@@ -176,9 +192,9 @@ BN_is_prime_fasttest_ex(), respectively.
 
 BN_generate_prime_ex() return 1 on success or 0 on error.
 
-BN_is_prime_ex(), BN_is_prime_fasttest_ex(), BN_is_prime() and
-BN_is_prime_fasttest() return 0 if the number is composite, 1 if it is
-prime with an error probability of less than 0.25^B<nchecks>, and
+BN_is_prime_ex(), BN_is_prime_fasttest_ex(), BN_is_prime(),
+BN_is_prime_fasttest() and BN_check_prime return 0 if the number is composite,
+1 if it is prime with an error probability of less than 0.25^B<nchecks>, and
 -1 on error.
 
 BN_generate_prime() returns the prime number on success, B<NULL> otherwise.
@@ -220,6 +236,8 @@ L<RAND(7)>
 The BN_GENCB_new(), BN_GENCB_free(),
 and BN_GENCB_get_arg() functions were added in OpenSSL 1.1.0.
 
+BN_check_prime() was added in OpenSSL 3.0.
+
 =head1 COPYRIGHT
 
 Copyright 2000-2019 The OpenSSL Project Authors. All Rights Reserved.