Set the ossl_shim to auto retry if not running asynchronously

In certain circumstances in the DTLS code we have to drop a record (e.g. if
it is a stale retransmit). We then have to move on to try and read the next
record. Some applications using blocking sockets (e.g. s_server/s_client
will hang if there isn't actually any data to be read from the socket yet).
Others can tolerate this. Therefore SSL_read()/SSL_write() can sometimes
return SSL_ERROR_WANT_READ/SSL_ERROR_WANT_WRITE even when using blocking
sockets. Applications can use the mode SSL_MODE_AUTO_RETRY, to switch this
behaviour off so that we never return unless we have read the data we
wanted to.

Commit ad96225285 fixed a DTLS problem where we always retried even if
SSL_MODE_AUTO_RETRY was not set. However that fix caused the Boring
ossl_shim to fail in some tests because it was relying on the previous
(buggy) behaviour. The ossl_shim should be set into SSL_MODE_AUTO_RETRY if
it is not operating asynchronously to avoid this problem.

[extended tests]

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6216)

1 files changed, 4 insertions, 0 deletions

diff --git a/test/ossl_shim/ossl_shim.cc b/test/ossl_shim/ossl_shim.cc
index 739d1bd167..b1067e8420 100644
--- a/test/ossl_shim/ossl_shim.cc
+++ b/test/ossl_shim/ossl_shim.cc
@@ -985,6 +985,10 @@ static bool DoExchange(bssl::UniquePtr<SSL_SESSION> *out_session,
     SSL_set_max_cert_list(ssl.get(), config->max_cert_list);
   }
 
+  if (!config->async) {
+    SSL_set_mode(ssl.get(), SSL_MODE_AUTO_RETRY);
+  }
+
   int sock = Connect(config->port);
   if (sock == -1) {
     return false;