Disallow SSL_key_update() if there are writes pending

If an application is halfway through writing application data it should
not be allowed to attempt an SSL_key_update() operation. Instead the
SSL_write() operation should be completed.

Fixes #12485

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16098)

1 files changed, 5 insertions, 0 deletions

diff --git a/ssl/ssl_lib.c b/ssl/ssl_lib.c
index e1c95ddc60..ffd0a0bc6d 100644
--- a/ssl/ssl_lib.c
+++ b/ssl/ssl_lib.c
@@ -2119,6 +2119,11 @@ int SSL_key_update(SSL *s, int updatetype)
         return 0;
     }
 
+    if (RECORD_LAYER_write_pending(&s->rlayer)) {
+        SSLerr(SSL_F_SSL_KEY_UPDATE, SSL_R_BAD_WRITE_RETRY);
+        return 0;
+    }
+
     ossl_statem_set_in_init(s, 1);
     s->key_update = updatetype;
     return 1;