Avoid errors with a priori inapplicable protocol bounds

The 'MinProtocol' and 'MaxProtocol' configuration commands now silently
ignore TLS protocol version bounds when configurign DTLS-based contexts,
and conversely, silently ignore DTLS protocol version bounds when
configuring TLS-based contexts.  The commands can be repeated to set
bounds of both types.  The same applies with the corresponding
"min_protocol" and "max_protocol" command-line switches, in case some
application uses both TLS and DTLS.

SSL_CTX instances that are created for a fixed protocol version (e.g.
TLSv1_server_method()) also silently ignore version bounds.  Previously
attempts to apply bounds to these protocol versions would result in an
error.  Now only the "version-flexible" SSL_CTX instances are subject to
limits in configuration files in command-line options.

Expected to resolve #12394

Reviewed-by: Paul Dale <paul.dale@oracle.com>
GH: #12472

1 files changed, 19 insertions, 15 deletions

diff --git a/ssl/statem/statem_lib.c b/ssl/statem/statem_lib.c
index de8212747f..d8aab20e92 100644
--- a/ssl/statem/statem_lib.c
+++ b/ssl/statem/statem_lib.c
@@ -1679,11 +1679,22 @@ int ssl_check_version_downgrade(SSL *s)
  */
 int ssl_set_version_bound(int method_version, int version, int *bound)
 {
+    int valid_tls;
+    int valid_dtls;
+
     if (version == 0) {
         *bound = version;
         return 1;
     }
 
+    valid_tls = version >= SSL3_VERSION && version <= TLS_MAX_VERSION_INTERNAL;
+    valid_dtls =
+        DTLS_VERSION_LE(version, DTLS_MAX_VERSION_INTERNAL) &&
+        DTLS_VERSION_GE(version, DTLS1_BAD_VER);
+
+    if (!valid_tls && !valid_dtls)
+        return 0;
+
     /*-
      * Restrict TLS methods to TLS protocol versions.
      * Restrict DTLS methods to DTLS protocol versions.
@@ -1694,31 +1705,24 @@ int ssl_set_version_bound(int method_version, int version, int *bound)
      * configurations.  If the MIN (supported) version ever rises, the user's
      * "floor" remains valid even if no longer available.  We don't expect the
      * MAX ceiling to ever get lower, so making that variable makes sense.
+     *
+     * We ignore attempts to set bounds on version-inflexible methods,
+     * returning success.
      */
     switch (method_version) {
     default:
-        /*
-         * XXX For fixed version methods, should we always fail and not set any
-         * bounds, always succeed and not set any bounds, or set the bounds and
-         * arrange to fail later if they are not met?  At present fixed-version
-         * methods are not subject to controls that disable individual protocol
-         * versions.
-         */
-        return 0;
+        break;
 
     case TLS_ANY_VERSION:
-        if (version < SSL3_VERSION || version > TLS_MAX_VERSION_INTERNAL)
-            return 0;
+        if (valid_tls)
+            *bound = version;
         break;
 
     case DTLS_ANY_VERSION:
-        if (DTLS_VERSION_GT(version, DTLS_MAX_VERSION_INTERNAL) ||
-            DTLS_VERSION_LT(version, DTLS1_BAD_VER))
-            return 0;
+        if (valid_dtls)
+            *bound = version;
         break;
     }
-
-    *bound = version;
     return 1;
 }