Disallow SSL_key_update() if there are writes pending

If an application is halfway through writing application data it should
not be allowed to attempt an SSL_key_update() operation. Instead the
SSL_write() operation should be completed.

Fixes #12485

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16077)

1 files changed, 5 insertions, 0 deletions

diff --git a/ssl/ssl_lib.c b/ssl/ssl_lib.c
index c1e8e41f02..892a417d93 100644
--- a/ssl/ssl_lib.c
+++ b/ssl/ssl_lib.c
@@ -2262,6 +2262,11 @@ int SSL_key_update(SSL *s, int updatetype)
         return 0;
     }
 
+    if (RECORD_LAYER_write_pending(&s->rlayer)) {
+        ERR_raise(ERR_LIB_SSL, SSL_R_BAD_WRITE_RETRY);
+        return 0;
+    }
+
     ossl_statem_set_in_init(s, 1);
     s->key_update = updatetype;
     return 1;