Skip CN DNS name constraint checks when not needed

Only check the CN against DNS name contraints if the
`X509_CHECK_FLAG_NEVER_CHECK_SUBJECT` flag is not set, and either the
certificate has no DNS subject alternative names or the
`X509_CHECK_FLAG_ALWAYS_CHECK_SUBJECT` flag is set.

Add pertinent documentation, and touch up some stale text about
name checks and DANE.

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>

3 files changed, 24 insertions, 6 deletions

diff --git a/doc/crypto/X509_VERIFY_PARAM_set_flags.pod b/doc/crypto/X509_VERIFY_PARAM_set_flags.pod
index 55077e5253..320b258a85 100644
--- a/doc/crypto/X509_VERIFY_PARAM_set_flags.pod
+++ b/doc/crypto/X509_VERIFY_PARAM_set_flags.pod
@@ -133,14 +133,29 @@ B<name> clearing any previously specified host name or names.  If
 B<name> is NULL, or empty the list of hostnames is cleared, and
 name checks are not performed on the peer certificate.  If B<name>
 is NUL-terminated, B<namelen> may be zero, otherwise B<namelen>
-must be set to the length of B<name>.  When a hostname is specified,
+must be set to the length of B<name>.
+
+When a hostname is specified,
 certificate verification automatically invokes L<X509_check_host(3)>
 with flags equal to the B<flags> argument given to
 X509_VERIFY_PARAM_set_hostflags() (default zero).  Applications
 are strongly advised to use this interface in preference to explicitly
-calling L<X509_check_host(3)>, hostname checks are out of scope
+calling L<X509_check_host(3)>, hostname checks may be out of scope
 with the DANE-EE(3) certificate usage, and the internal check will
-be suppressed as appropriate when DANE support is added to OpenSSL.
+be suppressed as appropriate when DANE verification is enabled.
+
+When the subject CommonName will not be ignored, whether as a result of the
+B<X509_CHECK_FLAG_ALWAYS_CHECK_SUBJECT> host flag, or because no DNS subject
+alternative names are present in the certificate, any DNS name constraints in
+issuer certificates apply to the subject CommonName as well as the subject
+alternative name extension.
+
+When the subject CommonName will be ignored, whether as a result of the
+B<X509_CHECK_FLAG_NEVER_CHECK_SUBJECT> host flag, or because some DNS subject
+alternative names are present in the certificate, DNS name constraints in
+issuer certificates will not be applied to the subject DN.
+As described in X509_check_host(3) the B<X509_CHECK_FLAG_NEVER_CHECK_SUBJECT>
+flag takes precendence over the B<X509_CHECK_FLAG_ALWAYS_CHECK_SUBJECT> flag.
 
 X509_VERIFY_PARAM_get_hostflags() returns any host flags previously set via a
 call to X509_VERIFY_PARAM_set_hostflags().
diff --git a/doc/crypto/X509_check_host.pod b/doc/crypto/X509_check_host.pod
index 93848152b5..06089e1402 100644
--- a/doc/crypto/X509_check_host.pod
+++ b/doc/crypto/X509_check_host.pod
@@ -93,6 +93,9 @@ consider the subject DN even if the certificate contains no subject alternative
 names of the right type (DNS name or email address as appropriate); the default
 is to use the subject DN when no corresponding subject alternative names are
 present.
+If both B<X509_CHECK_FLAG_ALWAYS_CHECK_SUBJECT> and
+B<X509_CHECK_FLAG_NEVER_CHECK_SUBJECT> are specified, the latter takes
+precedence and the subject DN is not checked for matching names.
 
 If set, B<X509_CHECK_FLAG_NO_WILDCARDS> disables wildcard
 expansion; this only applies to B<X509_check_host>.
@@ -128,9 +131,9 @@ NULs.
 
 Applications are encouraged to use X509_VERIFY_PARAM_set1_host()
 rather than explicitly calling L<X509_check_host(3)>. Host name
-checks are out of scope with the DANE-EE(3) certificate usage,
+checks may be out of scope with the DANE-EE(3) certificate usage,
 and the internal checks will be suppressed as appropriate when
-DANE support is added to OpenSSL.
+DANE support is enabled.
 
 =head1 SEE ALSO
 
diff --git a/doc/ssl/SSL_set1_host.pod b/doc/ssl/SSL_set1_host.pod
index 3339a0e803..1fc1054013 100644
--- a/doc/ssl/SSL_set1_host.pod
+++ b/doc/ssl/SSL_set1_host.pod
@@ -56,7 +56,7 @@ is cleared or freed, or a renegotiation takes place.  Applications
 must not free the return value.
 
 SSL clients are advised to use these functions in preference to
-explicitly calling L<X509_check_host(3)>.  Hostname checks are out
+explicitly calling L<X509_check_host(3)>.  Hostname checks may be out
 of scope with the RFC7671 DANE-EE(3) certificate usage, and the
 internal check will be suppressed as appropriate when DANE is
 enabled.