TLS client: allow cert verify callback return -1 for SSL_ERROR_WANT_RETRY_VERIFY

The client-side cert verification callback function may not only return
as usual for success or 0 for failure, but also -1,
typically on failure verifying the server certificate.
This makes the handshake suspend and return control to the calling application
with SSL_ERROR_WANT_RETRY_VERIFY.
The app can for instance fetch further certificates or cert status information
needed for the verification.
Calling SSL_connect() again resumes the connection attempt
by retrying the server certificate verification step.
This process may even be repeated if need be.

The core implementation of the feature is in ssl/statem/statem_clnt.c,
splitting tls_process_server_certificate() into a preparation step
that just copies the certificates received from the server to s->session->peer_chain
(rather than having them in a local variable at first) and returns to the state machine,
and a post-processing step in tls_post_process_server_certificate() that can be repeated:
Try verifying the current contents of s->session->peer_chain basically as before,
but give the verification callback function the chance to pause connecting and
make the TLS state machine later call tls_post_process_server_certificate() again.
Otherwise processing continues as usual.

The documentation of the new feature is added to SSL_CTX_set_cert_verify_callback.pod
and SSL_want.pod.

This adds two tests:
* A generic test in test/helpers/handshake.c
  on the usability of the new server cert verification retry feature.
  It is triggered via test/ssl-tests/03-custom_verify.cnf.in (while the bulky auto-
  generated changes to test/ssl-tests/03-custom_verify.cnf can be basically ignored).
* A test in test/sslapitest.c that demonstrates the effectiveness of the approach
  for augmenting the cert chain provided by the server in between SSL_connect() calls.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/13906)

1 files changed, 19 insertions, 1 deletions

diff --git a/test/helpers/handshake.c b/test/helpers/handshake.c
index e286df6cf0..03ee9a004a 100644
--- a/test/helpers/handshake.c
+++ b/test/helpers/handshake.c
@@ -314,6 +314,14 @@ static int verify_reject_cb(X509_STORE_CTX *ctx, void *arg) {
     return 0;
 }
 
+static int n_retries = 0;
+static int verify_retry_cb(X509_STORE_CTX *ctx, void *arg) {
+    if (--n_retries < 0)
+        return 1;
+    X509_STORE_CTX_set_error(ctx, X509_V_ERR_APPLICATION_VERIFICATION);
+    return -1;
+}
+
 static int verify_accept_cb(X509_STORE_CTX *ctx, void *arg) {
     return 1;
 }
@@ -527,6 +535,10 @@ static int configure_handshake_ctx(SSL_CTX *server_ctx, SSL_CTX *server2_ctx,
     case SSL_TEST_VERIFY_ACCEPT_ALL:
         SSL_CTX_set_cert_verify_callback(client_ctx, &verify_accept_cb, NULL);
         break;
+    case SSL_TEST_VERIFY_RETRY_ONCE:
+        n_retries = 1;
+        SSL_CTX_set_cert_verify_callback(client_ctx, &verify_retry_cb, NULL);
+        break;
     case SSL_TEST_VERIFY_REJECT_ALL:
         SSL_CTX_set_cert_verify_callback(client_ctx, &verify_reject_cb, NULL);
         break;
@@ -807,8 +819,10 @@ static void do_handshake_step(PEER *peer)
             peer->status = PEER_ERROR;
         } else {
             int error = SSL_get_error(peer->ssl, ret);
+
             /* Memory bios should never block with SSL_ERROR_WANT_WRITE. */
-            if (error != SSL_ERROR_WANT_READ)
+            if (error != SSL_ERROR_WANT_READ
+                    && error != SSL_ERROR_WANT_RETRY_VERIFY)
                 peer->status = PEER_ERROR;
         }
     }
@@ -1674,6 +1688,10 @@ static HANDSHAKE_RESULT *do_handshake_internal(
         ret->session_id = SSL_TEST_SESSION_ID_YES;
     ret->session_ticket_do_not_call = server_ex_data.session_ticket_do_not_call;
 
+    if (extra->client.verify_callback == SSL_TEST_VERIFY_RETRY_ONCE
+            && n_retries != -1)
+        ret->result = SSL_TEST_SERVER_FAIL;
+
 #ifndef OPENSSL_NO_NEXTPROTONEG
     SSL_get0_next_proto_negotiated(client.ssl, &proto, &proto_len);
     ret->client_npn_negotiated = dup_str(proto, proto_len);