Handle SSL_shutdown while in init more appropriately #2

Previous commit 7bb196a71 attempted to "fix" a problem with the way
SSL_shutdown() behaved whilst in mid-handshake. The original behaviour had
SSL_shutdown() return immediately having taken no action if called mid-
handshake with a return value of 1 (meaning everything was shutdown
successfully). In fact the shutdown has not been successful.

Commit 7bb196a71 changed that to send a close_notify anyway and then
return. This seems to be causing some problems for some applications so
perhaps a better (much simpler) approach is revert to the previous
behaviour (no attempt at a shutdown), but return -1 (meaning the shutdown
was not successful).

This also fixes a bug where SSL_shutdown always returns 0 when shutdown
*very* early in the handshake (i.e. we are still using SSLv23_method).

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>

1 files changed, 12 insertions, 9 deletions

diff --git a/ssl/ssl_lib.c b/ssl/ssl_lib.c
index 197a37cd0d..23391327cb 100644
--- a/ssl/ssl_lib.c
+++ b/ssl/ssl_lib.c
@@ -1578,19 +1578,22 @@ int SSL_shutdown(SSL *s)
         return -1;
     }
 
-    if((s->mode & SSL_MODE_ASYNC) && ASYNC_get_current_job() == NULL) {
-        struct ssl_async_args args;
+    if (!SSL_in_init(s)) {
+        if((s->mode & SSL_MODE_ASYNC) && ASYNC_get_current_job() == NULL) {
+            struct ssl_async_args args;
 
-        args.s = s;
-        args.type = OTHERFUNC;
-        args.f.func_other = s->method->ssl_shutdown;
+            args.s = s;
+            args.type = OTHERFUNC;
+            args.f.func_other = s->method->ssl_shutdown;
 
-        return ssl_start_async_job(s, &args, ssl_io_intern);
+            return ssl_start_async_job(s, &args, ssl_io_intern);
+        } else {
+            return s->method->ssl_shutdown(s);
+        }
     } else {
-        return s->method->ssl_shutdown(s);
+        SSLerr(SSL_F_SSL_SHUTDOWN, SSL_R_SHUTDOWN_WHILE_IN_INIT);
+        return -1;
     }
-
-    return s->method->ssl_shutdown(s);
 }
 
 int SSL_renegotiate(SSL *s)