Add X509_NAME_hash_ex() to be able to check if it failed due to unsupported SHA1

Deprecate X509_NAME_hash()
Document X509_NAME_hash_ex(), X509_NAME_hash(), X509_{subject,issuer}_name_hash()

Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/13762)

2 files changed, 49 insertions, 13 deletions

diff --git a/doc/man3/X509_LOOKUP_hash_dir.pod b/doc/man3/X509_LOOKUP_hash_dir.pod
index 5a660f100d..282b25807b 100644
--- a/doc/man3/X509_LOOKUP_hash_dir.pod
+++ b/doc/man3/X509_LOOKUP_hash_dir.pod
@@ -87,8 +87,8 @@ the directory.
 The directory should contain one certificate or CRL per file in PEM format,
 with a filename of the form I<hash>.I<N> for a certificate, or
 I<hash>.B<r>I<N> for a CRL.
-The I<hash> is the value returned by the L<X509_NAME_hash(3)> function applied
-to the subject name for certificates or issuer name for CRLs.
+The I<hash> is the value returned by the L<X509_NAME_hash_ex(3)> function
+applied to the subject name for certificates or issuer name for CRLs.
 The hash can also be obtained via the B<-hash> option of the
 L<openssl-x509(1)> or L<openssl-crl(1)> commands.
 
diff --git a/doc/man3/X509_get_subject_name.pod b/doc/man3/X509_get_subject_name.pod
index a9c8fb1d87..5a4ff47554 100644
--- a/doc/man3/X509_get_subject_name.pod
+++ b/doc/man3/X509_get_subject_name.pod
@@ -2,20 +2,29 @@
 
 =head1 NAME
 
-X509_get_subject_name, X509_set_subject_name, X509_get_issuer_name,
-X509_set_issuer_name, X509_REQ_get_subject_name, X509_REQ_set_subject_name,
-X509_CRL_get_issuer, X509_CRL_set_issuer_name - get and set issuer or
-subject names
+X509_NAME_hash_ex, X509_NAME_hash,
+X509_get_subject_name, X509_set_subject_name, X509_subject_name_hash,
+X509_get_issuer_name, X509_set_issuer_name, X509_issuer_name_hash,
+X509_REQ_get_subject_name, X509_REQ_set_subject_name,
+X509_CRL_get_issuer, X509_CRL_set_issuer_name -
+get X509_NAME hashes or get and set issuer or subject names
 
 =head1 SYNOPSIS
 
  #include <openssl/x509.h>
 
+ unsigned long X509_NAME_hash_ex(const X509_NAME *x, OSSL_LIB_CTX *libctx,
+                                 const char *propq, int *ok);
+Deprecated since OpenSSL 3.0:
+ #define X509_NAME_hash(x) X509_NAME_hash_ex(x, NULL, NULL, NULL)
+
  X509_NAME *X509_get_subject_name(const X509 *x);
  int X509_set_subject_name(X509 *x, const X509_NAME *name);
+ unsigned long X509_subject_name_hash(X509 *x);
 
  X509_NAME *X509_get_issuer_name(const X509 *x);
  int X509_set_issuer_name(X509 *x, const X509_NAME *name);
+ unsigned long X509_issuer_name_hash(X509 *x);
 
  X509_NAME *X509_REQ_get_subject_name(const X509_REQ *req);
  int X509_REQ_set_subject_name(X509_REQ *req, const X509_NAME *name);
@@ -25,16 +34,29 @@ subject names
 
 =head1 DESCRIPTION
 
-X509_get_subject_name() returns the subject name of certificate B<x>. The
+X509_NAME_hash_ex() returns a hash value of name I<x> or 0 on failure,
+using any given library context I<libctx> and property query I<propq>.
+The I<ok> result argument may be NULL
+or else is used to return 1 for success and 0 for failure.
+Failure may happen on malloc error or if no SHA1 implementation is available.
+
+X509_NAME_hash() returns a hash value of name I<x> or 0 on failure,
+using the default library context and default property query.
+
+X509_get_subject_name() returns the subject name of certificate I<x>. The
 returned value is an internal pointer which B<MUST NOT> be freed.
 
-X509_set_subject_name() sets the issuer name of certificate B<x> to
-B<name>. The B<name> parameter is copied internally and should be freed
+X509_set_subject_name() sets the issuer name of certificate I<x> to
+I<name>. The I<name> parameter is copied internally and should be freed
 up when it is no longer needed.
 
-X509_get_issuer_name() and X509_set_issuer_name() are identical to
-X509_get_subject_name() and X509_set_subject_name() except the get and
-set the issuer name of B<x>.
+X509_subject_name_hash() returns a hash value of the subject name of
+certificate I<x>.
+
+X509_get_issuer_name(), X509_set_issuer_name(), and X509_issuer_name_hash()
+are identical to
+X509_get_subject_name(), X509_set_subject_name(), and X509_subject_name_hash()
+except they relate to the issuer name of I<x>.
 
 Similarly X509_REQ_get_subject_name(), X509_REQ_set_subject_name(),
 X509_CRL_get_issuer() and X509_CRL_set_issuer_name() get or set the subject
@@ -45,9 +67,21 @@ or issuer names of certificate requests of CRLs respectively.
 X509_get_subject_name(), X509_get_issuer_name(), X509_REQ_get_subject_name()
 and X509_CRL_get_issuer() return an B<X509_NAME> pointer.
 
+X509_NAME_hash_ex(), X509_NAME_hash(),
+X509_subject_name_hash() and X509_issuer_name_hash()
+return the first four bytes of the SHA1 hash value,
+converted to B<unsigned long> in little endian order,
+or 0 on failure.
+
 X509_set_subject_name(), X509_set_issuer_name(), X509_REQ_set_subject_name()
 and X509_CRL_set_issuer_name() return 1 for success and 0 for failure.
 
+=head1 BUGS
+
+In case X509_NAME_hash(), X509_subject_name_hash(), or X509_issuer_name_hash()
+returns 0 it remains unclear if this is the real hash value or due to failure.
+Better use X509_NAME_hash_ex() instead.
+
 =head1 SEE ALSO
 
 L<d2i_X509(3)>,
@@ -74,9 +108,11 @@ earlier versions.
 X509_CRL_get_issuer() is a function in OpenSSL 1.1.0. It was previously
 added in OpenSSL 1.0.0 as a macro.
 
+X509_NAME_hash() was turned into a macro and deprecated in OpenSSL 3.0.
+
 =head1 COPYRIGHT
 
-Copyright 2015-2020 The OpenSSL Project Authors. All Rights Reserved.
+Copyright 2015-2021 The OpenSSL Project Authors. All Rights Reserved.
 
 Licensed under the Apache License 2.0 (the "License").  You may not use
 this file except in compliance with the License.  You can obtain a copy