TLS client: allow cert verify callback return -1 for SSL_ERROR_WANT_RETRY_VERIFY

The client-side cert verification callback function may not only return
as usual for success or 0 for failure, but also -1,
typically on failure verifying the server certificate.
This makes the handshake suspend and return control to the calling application
with SSL_ERROR_WANT_RETRY_VERIFY.
The app can for instance fetch further certificates or cert status information
needed for the verification.
Calling SSL_connect() again resumes the connection attempt
by retrying the server certificate verification step.
This process may even be repeated if need be.

The core implementation of the feature is in ssl/statem/statem_clnt.c,
splitting tls_process_server_certificate() into a preparation step
that just copies the certificates received from the server to s->session->peer_chain
(rather than having them in a local variable at first) and returns to the state machine,
and a post-processing step in tls_post_process_server_certificate() that can be repeated:
Try verifying the current contents of s->session->peer_chain basically as before,
but give the verification callback function the chance to pause connecting and
make the TLS state machine later call tls_post_process_server_certificate() again.
Otherwise processing continues as usual.

The documentation of the new feature is added to SSL_CTX_set_cert_verify_callback.pod
and SSL_want.pod.

This adds two tests:
* A generic test in test/helpers/handshake.c
  on the usability of the new server cert verification retry feature.
  It is triggered via test/ssl-tests/03-custom_verify.cnf.in (while the bulky auto-
  generated changes to test/ssl-tests/03-custom_verify.cnf can be basically ignored).
* A test in test/sslapitest.c that demonstrates the effectiveness of the approach
  for augmenting the cert chain provided by the server in between SSL_connect() calls.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/13906)

1 files changed, 2 insertions, 0 deletions

diff --git a/test/helpers/ssl_test_ctx.c b/test/helpers/ssl_test_ctx.c
index db4752c54f..0921416be2 100644
--- a/test/helpers/ssl_test_ctx.c
+++ b/test/helpers/ssl_test_ctx.c
@@ -177,6 +177,7 @@ const char *ssl_protocol_name(int protocol)
 static const test_enum ssl_verify_callbacks[] = {
     {"None", SSL_TEST_VERIFY_NONE},
     {"AcceptAll", SSL_TEST_VERIFY_ACCEPT_ALL},
+    {"RetryOnce", SSL_TEST_VERIFY_RETRY_ONCE},
     {"RejectAll", SSL_TEST_VERIFY_REJECT_ALL},
 };
 
@@ -184,6 +185,7 @@ __owur static int parse_client_verify_callback(SSL_TEST_CLIENT_CONF *client_conf
                                                const char *value)
 {
     int ret_value;
+
     if (!parse_enum(ssl_verify_callbacks, OSSL_NELEM(ssl_verify_callbacks),
                     &ret_value, value)) {
         return 0;