TLS client: allow cert verify callback return -1 for SSL_ERROR_WANT_RETRY_VERIFY

The client-side cert verification callback function may not only return
as usual for success or 0 for failure, but also -1,
typically on failure verifying the server certificate.
This makes the handshake suspend and return control to the calling application
with SSL_ERROR_WANT_RETRY_VERIFY.
The app can for instance fetch further certificates or cert status information
needed for the verification.
Calling SSL_connect() again resumes the connection attempt
by retrying the server certificate verification step.
This process may even be repeated if need be.

The core implementation of the feature is in ssl/statem/statem_clnt.c,
splitting tls_process_server_certificate() into a preparation step
that just copies the certificates received from the server to s->session->peer_chain
(rather than having them in a local variable at first) and returns to the state machine,
and a post-processing step in tls_post_process_server_certificate() that can be repeated:
Try verifying the current contents of s->session->peer_chain basically as before,
but give the verification callback function the chance to pause connecting and
make the TLS state machine later call tls_post_process_server_certificate() again.
Otherwise processing continues as usual.

The documentation of the new feature is added to SSL_CTX_set_cert_verify_callback.pod
and SSL_want.pod.

This adds two tests:
* A generic test in test/helpers/handshake.c
  on the usability of the new server cert verification retry feature.
  It is triggered via test/ssl-tests/03-custom_verify.cnf.in (while the bulky auto-
  generated changes to test/ssl-tests/03-custom_verify.cnf can be basically ignored).
* A test in test/sslapitest.c that demonstrates the effectiveness of the approach
  for augmenting the cert chain provided by the server in between SSL_connect() calls.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/13906)

2 files changed, 33 insertions, 17 deletions

diff --git a/doc/man3/SSL_CTX_set_cert_verify_callback.pod b/doc/man3/SSL_CTX_set_cert_verify_callback.pod
index fa47e49516..4dff998399 100644
--- a/doc/man3/SSL_CTX_set_cert_verify_callback.pod
+++ b/doc/man3/SSL_CTX_set_cert_verify_callback.pod
@@ -33,7 +33,19 @@ argument I<arg> is specified by the application when setting I<callback>.
 
 I<callback> should return 1 to indicate verification success and 0 to
 indicate verification failure. If SSL_VERIFY_PEER is set and I<callback>
-returns 0, the handshake will fail. As the verification procedure may
+returns 0, the handshake will fail.
+
+In client mode I<callback> may also return -1,
+typically on failure verifying the server certificate.
+This makes the handshake suspend and return control to the calling application
+with B<SSL_ERROR_WANT_RETRY_VERIFY>.
+The app can for instance fetch further certificates or cert status information
+needed for the verification.
+Calling L<SSL_connect(3)> again resumes the connection attempt
+by retrying the server certificate verification step.
+This process may even be repeated if need be.
+
+As the verification procedure may
 allow the connection to continue in the case of failure (by always
 returning 1) the verification result must be set in any case using the
 B<error> member of I<x509_store_ctx> so that the calling application
diff --git a/doc/man3/SSL_want.pod b/doc/man3/SSL_want.pod
index 5b7955546c..fcd128783b 100644
--- a/doc/man3/SSL_want.pod
+++ b/doc/man3/SSL_want.pod
@@ -2,9 +2,9 @@
 
 =head1 NAME
 
-SSL_want, SSL_want_nothing, SSL_want_read, SSL_want_write, SSL_want_x509_lookup,
-SSL_want_async, SSL_want_async_job, SSL_want_client_hello_cb - obtain state
-information TLS/SSL I/O operation
+SSL_want, SSL_want_nothing, SSL_want_read, SSL_want_write,
+SSL_want_x509_lookup, SSL_want_retry_verify, SSL_want_async, SSL_want_async_job,
+SSL_want_client_hello_cb - obtain state information TLS/SSL I/O operation
 
 =head1 SYNOPSIS
 
@@ -15,6 +15,7 @@ information TLS/SSL I/O operation
  int SSL_want_read(const SSL *ssl);
  int SSL_want_write(const SSL *ssl);
  int SSL_want_x509_lookup(const SSL *ssl);
+ int SSL_want_retry_verify(const SSL *ssl);
  int SSL_want_async(const SSL *ssl);
  int SSL_want_async_job(const SSL *ssl);
  int SSL_want_client_hello_cb(const SSL *ssl);
@@ -53,47 +54,50 @@ There is no data to be written or to be read.
 
 There are data in the SSL buffer that must be written to the underlying
 B<BIO> layer in order to complete the actual SSL_*() operation.
-A call to L<SSL_get_error(3)> should return
-SSL_ERROR_WANT_WRITE.
+A call to L<SSL_get_error(3)> should return B<SSL_ERROR_WANT_WRITE>.
 
 =item SSL_READING
 
 More data must be read from the underlying B<BIO> layer in order to
 complete the actual SSL_*() operation.
-A call to L<SSL_get_error(3)> should return
-SSL_ERROR_WANT_READ.
+A call to L<SSL_get_error(3)> should return B<SSL_ERROR_WANT_READ>.
 
 =item SSL_X509_LOOKUP
 
 The operation did not complete because an application callback set by
 SSL_CTX_set_client_cert_cb() has asked to be called again.
-A call to L<SSL_get_error(3)> should return
-SSL_ERROR_WANT_X509_LOOKUP.
+A call to L<SSL_get_error(3)> should return B<SSL_ERROR_WANT_X509_LOOKUP>.
+
+=item SSL_RETRY_VERIFY
+
+The operation did not complete because an application callback set by
+SSL_CTX_set_cert_verify_callback() has asked to be called again.
+A call to L<SSL_get_error(3)> should return B<SSL_ERROR_WANT_RETRY_VERIFY>.
 
 =item SSL_ASYNC_PAUSED
 
 An asynchronous operation partially completed and was then paused. See
 L<SSL_get_all_async_fds(3)>. A call to L<SSL_get_error(3)> should return
-SSL_ERROR_WANT_ASYNC.
+B<SSL_ERROR_WANT_ASYNC>.
 
 =item SSL_ASYNC_NO_JOBS
 
 The asynchronous job could not be started because there were no async jobs
 available in the pool (see ASYNC_init_thread(3)). A call to L<SSL_get_error(3)>
-should return SSL_ERROR_WANT_ASYNC_JOB.
+should return B<SSL_ERROR_WANT_ASYNC_JOB>.
 
 =item SSL_CLIENT_HELLO_CB
 
 The operation did not complete because an application callback set by
 SSL_CTX_set_client_hello_cb() has asked to be called again.
-A call to L<SSL_get_error(3)> should return
-SSL_ERROR_WANT_CLIENT_HELLO_CB.
+A call to L<SSL_get_error(3)> should return B<SSL_ERROR_WANT_CLIENT_HELLO_CB>.
 
 =back
 
-SSL_want_nothing(), SSL_want_read(), SSL_want_write(), SSL_want_x509_lookup(),
-SSL_want_async(), SSL_want_async_job(), and SSL_want_client_hello_cb() return
-1, when the corresponding condition is true or 0 otherwise.
+SSL_want_nothing(), SSL_want_read(), SSL_want_write(),
+SSL_want_x509_lookup(), SSL_want_retry_verify(),
+SSL_want_async(), SSL_want_async_job(), and SSL_want_client_hello_cb()
+return 1 when the corresponding condition is true or 0 otherwise.
 
 =head1 SEE ALSO