X509_cmp(): Fix comparison in case x509v3_cache_extensions() failed to due to invalid cert

This is the upstream fix for #13698 reported for v1.1.1

Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/13755)

3 files changed, 11 insertions, 4 deletions

diff --git a/doc/internal/man3/x509v3_cache_extensions.pod b/doc/internal/man3/x509v3_cache_extensions.pod
index 418a19738c..cd00942333 100644
--- a/doc/internal/man3/x509v3_cache_extensions.pod
+++ b/doc/internal/man3/x509v3_cache_extensions.pod
@@ -17,7 +17,8 @@ This function processes any X509v3 extensions present in an X509 object I<x>
 and caches the result of that processing as well as further derived info,
 for instance whether the certificate is self-issued or has version X.509v1.
 It computes the SHA1 digest of the certificate using the default library context
-and property query string and stores the result in x->sha1_hash.
+and property query string and stores the result in x->sha1_hash,
+or on failure sets B<EXFLAG_NO_FINGERPRINT> in x->flags.
 It sets B<X509_SIG_INFO_VALID> in x->flags if x->siginf was filled successfully,
 which may not be possible if a referenced algorithm is unknown or not available.
 Many OpenSSL functions that use an X509 object call this function implicitly.
diff --git a/doc/man3/X509_cmp.pod b/doc/man3/X509_cmp.pod
index 1e6a166e65..777d055ad8 100644
--- a/doc/man3/X509_cmp.pod
+++ b/doc/man3/X509_cmp.pod
@@ -55,7 +55,8 @@ The B<X509> comparison functions return B<-1>, B<0>, or B<1> if object I<a> is
 found to be less than, to match, or be greater than object I<b>, respectively.
 
 X509_NAME_cmp(), X509_issuer_and_serial_cmp(), X509_issuer_name_cmp(),
-X509_subject_name_cmp() and X509_CRL_cmp() may return B<-2> to indicate an error.
+X509_subject_name_cmp(), X509_CRL_cmp(), and X509_CRL_match()
+may return B<-2> to indicate an error.
 
 =head1 NOTES
 
diff --git a/doc/man3/X509_get_extension_flags.pod b/doc/man3/X509_get_extension_flags.pod
index 3f09939e52..cac43d716e 100644
--- a/doc/man3/X509_get_extension_flags.pod
+++ b/doc/man3/X509_get_extension_flags.pod
@@ -78,12 +78,17 @@ The certificate contains an unhandled critical extension.
 
 =item B<EXFLAG_INVALID>
 
-Some certificate extension values are invalid or inconsistent. The
-certificate should be rejected.
+Some certificate extension values are invalid or inconsistent.
+The certificate should be rejected.
 This bit may also be raised after an out-of-memory error while
 processing the X509 object, so it may not be related to the processed
 ASN1 object itself.
 
+=item B<EXFLAG_NO_FINGERPRINT>
+
+Failed to compute the internal SHA1 hash value of the certificate or CRL.
+This may be due to malloc failure or because no SHA1 implementation was found.
+
 =item B<EXFLAG_INVALID_POLICY>
 
 The NID_certificate_policies certificate extension is invalid or