Re-introduce legacy EVP_PKEY types for provided keys

EVP_PKEYs with provider side internal keys got the key type
EVP_PKEY_NONE.  This turned out to be too disruptive, so we try
instead to find a matching EVP_PKEY_ASN1_METHOD and use whatever
EVP_PKEY type it uses.

To make internal coding easier, we introduce a few internal macros to
distinguish what can be expected from a EVP_PKEY:

- evp_pkey_is_blank(), to detect an unassigned EVP_PKEY.
- evp_pkey_is_typed(), to detect that an EVP_PKEY has been assigned a
  type, which may be an old style type number or a EVP_KEYMGMT method.
- evp_pkey_is_assigned(), to detect that an EVP_PKEY has been assigned
  an key value.
- evp_pkey_is_legacy(), to detect that the internal EVP_PKEY key is a
  legacy one, i.e. will be handled via an EVP_PKEY_ASN1_METHOD and an
  EVP_PKEY_METHOD.
- evp_pkey_is_provided(), to detect that the internal EVP_PKEY key is
  a provider side one, i.e. will be handdled via an EVP_KEYMGMT and
  other provider methods.

This also introduces EVP_PKEY_KEYMGMT, to indicate that this EVP_PKEY
contains a provider side key for which there are no known
EVP_PKEY_ASN1_METHODs or EVP_PKEY_METHODs, i.e. these can only be
handled via EVP_KEYMGMT and other provider methods.

Fixes #11823

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/11913)

1 files changed, 18 insertions, 2 deletions

diff --git a/include/crypto/evp.h b/include/crypto/evp.h
index ee4b6221e6..d1756cf183 100644
--- a/include/crypto/evp.h
+++ b/include/crypto/evp.h
@@ -518,9 +518,25 @@ const EVP_CIPHER *EVP_##cname##_ecb(void) { return &cname##_ecb; }
  *     (type != EVP_PKEY_NONE && pkey.ptr != NULL)      ## legacy (libcrypto only)
  *     || (keymgmt != NULL && keydata != NULL)          ## provider side
  *
- * The easiest way to detect a legacy key is:           type != EVP_PKEY_NONE
- * The easiest way to detect a provider side key is:    keymgmt != NULL
+ * The easiest way to detect a legacy key is:
+ *
+ *     keymgmt == NULL && type != EVP_PKEY_NONE
+ *
+ * The easiest way to detect a provider side key is:
+ *
+ *     keymgmt != NULL
  */
+#define evp_pkey_is_blank(pk)                                   \
+    ((pk)->type == EVP_PKEY_NONE && (pk)->keymgmt == NULL)
+#define evp_pkey_is_typed(pk)                                   \
+    ((pk)->type != EVP_PKEY_NONE || (pk)->keymgmt != NULL)
+#define evp_pkey_is_assigned(pk)                                \
+    ((pk)->pkey.ptr != NULL || (pk)->keydata != NULL)
+#define evp_pkey_is_legacy(pk)                                  \
+    ((pk)->type != EVP_PKEY_NONE && (pk)->keymgmt == NULL)
+#define evp_pkey_is_provided(pk)                                \
+    ((pk)->keymgmt != NULL)
+
 struct evp_pkey_st {
     /* == Legacy attributes == */
     int type;