Add KEM (Key encapsulation mechanism) support to providers

SP800-56Br2 requires support for the RSA primitives for RSASVE generate and recover.
As these are simple KEM operations another operation type has been added that can support future extensions.

Added public functions EVP_PKEY_encapsulate_init(), EVP_PKEY_encapsulate(), EVP_PKEY_decapsulate_init() and EVP_PKEY_decapsulate()
Added EVP_KEM_* functions.
Added OSSL_FUNC_kem_* dispatch functions

Added EVP_PKEY_CTX_set_kem_op() so that different types of KEM can be added in the future. This value must currently be set to
"RSASVE" after EVP_PKEY_encapsulate_init() & EVP_PKEY_decapsulate_init() as there is no default value.
This allows the existing RSA key types, keymanagers, and encoders to be used with the encapsulation operations.

The design of the public API's resulted from contributions from @romen & @levitte.

Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/12750)

5 files changed, 295 insertions, 0 deletions

diff --git a/doc/man7/EVP_KEM-RSA.pod b/doc/man7/EVP_KEM-RSA.pod
new file mode 100644
index 0000000000..21dc8ad3dd
--- /dev/null
+++ b/doc/man7/EVP_KEM-RSA.pod
@@ -0,0 +1,66 @@
+=pod
+
+=head1 NAME
+
+EVP_KEM-RSA
+- EVP_KEM RSA keytype and algorithm support
+
+=head1 DESCRIPTION
+
+The B<RSA> keytype and its parameters are described in L<EVP_PKEY-RSA(7)>.
+See L<EVP_PKEY_encapsulate(3)> and L<EVP_PKEY_decapsulate(3)> for more info.
+
+=head2 RSA KEM parameters
+
+=over 4
+
+=item "operation" (B<OSSL_KEM_PARAM_OPERATION>) <UTF8 string>
+
+The OpenSSL RSA Key Encapsulation Mechanism only currently supports the
+following operation
+
+=over 4
+
+=item "RSASVE"
+
+The encapsulate function simply generates a secret using random bytes and then
+encrypts the secret using the RSA public key (with no padding).
+The decapsulate function recovers the secret using the RSA private key.
+
+=back
+
+This can be set using EVP_PKEY_CTX_set_kem_op().
+
+=back
+
+
+=head1 CONFORMING TO
+
+=over 4
+
+=item SP800-56Br2
+
+Section 7.2.1.2 RSASVE Generate Operation (RSASVE.GENERATE).
+Section 7.2.1.3 RSASVE Recovery Operation (RSASVE.RECOVER).
+
+=back
+
+=head1 SEE ALSO
+
+L<EVP_PKEY_CTX_set_kem_op(3)>,
+L<EVP_PKEY_encapsulate(3)>,
+L<EVP_PKEY_decapsulate(3)>
+L<EVP_KEYMGMT(3)>,
+L<EVP_PKEY(3)>,
+L<provider-keymgmt(7)>
+
+=head1 COPYRIGHT
+
+Copyright 2020 The OpenSSL Project Authors. All Rights Reserved.
+
+Licensed under the Apache License 2.0 (the "License").  You may not use
+this file except in compliance with the License.  You can obtain a copy
+in the file LICENSE in the source distribution or at
+L<https://www.openssl.org/source/license.html>.
+
+=cut
diff --git a/doc/man7/OSSL_PROVIDER-FIPS.pod b/doc/man7/OSSL_PROVIDER-FIPS.pod
index 98c6079d72..b802efe215 100644
--- a/doc/man7/OSSL_PROVIDER-FIPS.pod
+++ b/doc/man7/OSSL_PROVIDER-FIPS.pod
@@ -136,6 +136,14 @@ This has the property "provider=fips,fips=no"
 
 =back
 
+=head2 Asymmetric Key Encapsulation
+
+=over 4
+
+=item RSA, see L<EVP_KEM-RSA(7)>
+
+=back
+
 =head2 Asymmetric Key Management
 
 =over 4
diff --git a/doc/man7/OSSL_PROVIDER-default.pod b/doc/man7/OSSL_PROVIDER-default.pod
index a88c0be6e6..848c887b29 100644
--- a/doc/man7/OSSL_PROVIDER-default.pod
+++ b/doc/man7/OSSL_PROVIDER-default.pod
@@ -182,6 +182,14 @@ The OpenSSL default provider supports these operations and algorithms:
 
 =back
 
+=head2 Asymmetric Key Encapsulation
+
+=over 4
+
+=item RSA, see L<EVP_KEM-RSA(7)>
+
+=back
+
 =head2 Asymmetric Key Management
 
 =over 4
diff --git a/doc/man7/provider-kem.pod b/doc/man7/provider-kem.pod
new file mode 100644
index 0000000000..4d16a3e625
--- /dev/null
+++ b/doc/man7/provider-kem.pod
@@ -0,0 +1,207 @@
+=pod
+
+=head1 NAME
+
+provider-kem - The kem library E<lt>-E<gt> provider functions
+
+=head1 SYNOPSIS
+
+=for openssl multiple includes
+
+ #include <openssl/core_dispatch.h>
+ #include <openssl/core_names.h>
+
+ /*
+  * None of these are actual functions, but are displayed like this for
+  * the function signatures for functions that are offered as function
+  * pointers in OSSL_DISPATCH arrays.
+  */
+
+ /* Context management */
+ void *OSSL_FUNC_kem_newctx(void *provctx);
+ void OSSL_FUNC_kem_freectx(void *ctx);
+ void *OSSL_FUNC_kem_dupctx(void *ctx);
+
+ /* Encapsulation */
+ int OSSL_FUNC_kem_encapsulate_init(void *ctx, void *provkey, const char *name);
+ int OSSL_FUNC_kem_encapsulate(void *ctx, unsigned char *out, size_t *outlen,
+                               unsigned char *secret, size_t *secretlen);
+
+ /* Decapsulation */
+ int OSSL_FUNC_kem_decapsulate_init(void *ctx, void *provkey, const char *name);
+ int OSSL_FUNC_kem_decapsulate(void *ctx, unsigned char *out, size_t *outlen,
+                               const unsigned char *in, size_t inlen);
+
+ /* KEM parameters */
+ int OSSL_FUNC_kem_get_ctx_params(void *ctx, OSSL_PARAM params[]);
+ const OSSL_PARAM *OSSL_FUNC_kem_gettable_ctx_params(void *provctx);
+ int OSSL_FUNC_kem_set_ctx_params(void *ctx, const OSSL_PARAM params[]);
+ const OSSL_PARAM *OSSL_FUNC_kem_settable_ctx_params(void *provctx);
+
+=head1 DESCRIPTION
+
+This documentation is primarily aimed at provider authors. See L<provider(7)>
+for further information.
+
+The asymmetric kem (OSSL_OP_KEM) operation enables providers to
+implement asymmetric kem algorithms and make them available to applications
+via the API functions L<EVP_PKEY_encapsulate(3)>,
+L<EVP_PKEY_decapsulate(3)> and other related functions.
+
+All "functions" mentioned here are passed as function pointers between
+F<libcrypto> and the provider in B<OSSL_DISPATCH> arrays via
+B<OSSL_ALGORITHM> arrays that are returned by the provider's
+provider_query_operation() function
+(see L<provider-base(7)/Provider Functions>).
+
+All these "functions" have a corresponding function type definition
+named B<OSSL_{name}_fn>, and a helper function to retrieve the
+function pointer from an B<OSSL_DISPATCH> element named
+B<OSSL_FUNC_{name}>.
+For example, the "function" OSSL_FUNC_kem_newctx() has these:
+
+ typedef void *(OSSL_FUNC_kem_newctx_fn)(void *provctx);
+ static ossl_inline OSSL_FUNC_kem_newctx_fn
+     OSSL_FUNC_kem_newctx(const OSSL_DISPATCH *opf);
+
+B<OSSL_DISPATCH> arrays are indexed by numbers that are provided as
+macros in L<openssl-core_dispatch.h(7)>, as follows:
+
+ OSSL_FUNC_kem_newctx               OSSL_FUNC_KEM_NEWCTX
+ OSSL_FUNC_kem_freectx              OSSL_FUNC_KEM_FREECTX
+ OSSL_FUNC_kem_dupctx               OSSL_FUNC_KEM_DUPCTX
+
+ OSSL_FUNC_kem_encapsulate_init     OSSL_FUNC_KEM_ENCAPSULATE_INIT
+ OSSL_FUNC_kem_encapsulate          OSSL_FUNC_KEM_ENCAPSULATE
+
+ OSSL_FUNC_kem_decapsulate_init     OSSL_FUNC_KEM_DECAPSULATE_INIT
+ OSSL_FUNC_kem_decapsulate          OSSL_FUNC_KEM_DECAPSULATE
+
+ OSSL_FUNC_kem_get_ctx_params       OSSL_FUNC_KEM_GET_CTX_PARAMS
+ OSSL_FUNC_kem_gettable_ctx_params  OSSL_FUNC_KEM_GETTABLE_CTX_PARAMS
+ OSSL_FUNC_kem_set_ctx_params       OSSL_FUNC_KEM_SET_CTX_PARAMS
+ OSSL_FUNC_kem_settable_ctx_params  OSSL_FUNC_KEM_SETTABLE_CTX_PARAMS
+
+An asymmetric kem algorithm implementation may not implement all of these
+functions.
+In order to be a consistent set of functions a provider must implement
+OSSL_FUNC_kem_newctx and OSSL_FUNC_kem_freectx.
+It must also implement both of OSSL_FUNC_kem_encapsulate_init and
+OSSL_FUNC_kem_encapsulate, or both of OSSL_FUNC_kem_decapsulate_init and
+OSSL_FUNC_kem_decapsulate.
+OSSL_FUNC_kem_get_ctx_params is optional but if it is present then so must
+OSSL_FUNC_kem_gettable_ctx_params.
+Similarly, OSSL_FUNC_kem_set_ctx_params is optional but if it is present then
+so must OSSL_FUNC_kem_settable_ctx_params.
+
+An asymmetric kem algorithm must also implement some mechanism for generating,
+loading or importing keys via the key management (OSSL_OP_KEYMGMT) operation.
+See L<provider-keymgmt(7)> for further details.
+
+=head2 Context Management Functions
+
+OSSL_FUNC_kem_newctx() should create and return a pointer to a provider side
+structure for holding context information during an asymmetric kem operation.
+A pointer to this context will be passed back in a number of the other
+asymmetric kem operation function calls.
+The parameter I<provctx> is the provider context generated during provider
+initialisation (see L<provider(7)>).
+
+OSSL_FUNC_kem_freectx() is passed a pointer to the provider side asymmetric
+kem context in the I<ctx> parameter.
+This function should free any resources associated with that context.
+
+OSSL_FUNC_kem_dupctx() should duplicate the provider side asymmetric kem
+context in the I<ctx> parameter and return the duplicate copy.
+
+=head2 Asymmetric Key Encapsulation Functions
+
+OSSL_FUNC_kem_encapsulate_init() initialises a context for an asymmetric
+encapsulation given a provider side asymmetric kem context in the I<ctx>
+parameter, a pointer to a provider key object in the I<provkey> parameter and
+the I<name> of the algorithm.
+The key object should have been previously generated, loaded or imported into
+the provider using the key management (OSSL_OP_KEYMGMT) operation (see
+provider-keymgmt(7)>.
+
+OSSL_FUNC_kem_encapsulate() performs the actual encapsulation itself.
+A previously initialised asymmetric kem context is passed in the I<ctx>
+parameter.
+Unless I<out> is NULL, the data to be encapsulated is internally generated,
+and returned into the the buffer pointed to by the I<secret> parameter and the
+encapsulated data should also be written to the location pointed to by the
+I<out> parameter. The length of the encapsulated data should be written to
+I<*outlen> and the length of the generated secret should be written to
+I<*secretlen>.
+
+If I<out> is NULL then the maximum length of the encapsulated data should be
+written to I<*outlen>, and the maximum length of the generated secret should be
+written to I<*secretlen>.
+
+=head2 Decapsulation Functions
+
+OSSL_FUNC_kem_decapsulate_init() initialises a context for an asymmetric
+decapsulation given a provider side asymmetric kem context in the I<ctx>
+parameter, a pointer to a provider key object in the I<provkey> parameter, and
+a I<name> of the algorithm.
+The key object should have been previously generated, loaded or imported into
+the provider using the key management (OSSL_OP_KEYMGMT) operation (see
+provider-keymgmt(7)>.
+
+OSSL_FUNC_kem_decapsulate() performs the actual decapsulation itself.
+A previously initialised asymmetric kem context is passed in the I<ctx>
+parameter.
+The data to be decapsulated is pointed to by the I<in> parameter which is I<inlen>
+bytes long.
+Unless I<out> is NULL, the decapsulated data should be written to the location
+pointed to by the I<out> parameter.
+The length of the decapsulated data should be written to I<*outlen>.
+If I<out> is NULL then the maximum length of the decapsulated data should be
+written to I<*outlen>.
+
+=head2 Asymmetric Key Encapsulation Parameters
+
+See L<OSSL_PARAM(3)> for further details on the parameters structure used by
+the OSSL_FUNC_kem_get_ctx_params() and OSSL_FUNC_kem_set_ctx_params()
+functions.
+
+OSSL_FUNC_kem_get_ctx_params() gets asymmetric kem parameters associated
+with the given provider side asymmetric kem context I<ctx> and stores them in
+I<params>.
+OSSL_FUNC_kem_set_ctx_params() sets the asymmetric kem parameters associated
+with the given provider side asymmetric kem context I<ctx> to I<params>.
+Any parameter settings are additional to any that were previously set.
+
+No parameters are currently recognised by built-in asymmetric kem algorithms.
+
+OSSL_FUNC_kem_gettable_ctx_params() and OSSL_FUNC_kem_settable_ctx_params()
+get a constant B<OSSL_PARAM> array that describes the gettable and settable
+parameters, i.e. parameters that can be used with OSSL_FUNC_kem_get_ctx_params()
+and OSSL_FUNC_kem_set_ctx_params() respectively.
+See L<OSSL_PARAM(3)> for the use of B<OSSL_PARAM> as parameter descriptor.
+
+=head1 RETURN VALUES
+
+OSSL_FUNC_kem_newctx() and OSSL_FUNC_kem_dupctx() should return the newly
+created provider side asymmetric kem context, or NULL on failure.
+
+All other functions should return 1 for success or 0 on error.
+
+=head1 SEE ALSO
+
+L<provider(7)>
+
+=head1 HISTORY
+
+The provider KEM interface was introduced in OpenSSL 3.0.
+
+=head1 COPYRIGHT
+
+Copyright 2020 The OpenSSL Project Authors. All Rights Reserved.
+
+Licensed under the Apache License 2.0 (the "License").  You may not use
+this file except in compliance with the License.  You can obtain a copy
+in the file LICENSE in the source distribution or at
+L<https://www.openssl.org/source/license.html>.
+
+=cut
diff --git a/doc/man7/provider.pod b/doc/man7/provider.pod
index 2f7f019650..ead37b5769 100644
--- a/doc/man7/provider.pod
+++ b/doc/man7/provider.pod
@@ -154,6 +154,12 @@ The number for this operation is B<OSSL_OP_ASYM_CIPHER>.
 The functions the provider can offer are described in
 L<provider-asym_cipher(7)>
 
+=item Asymmetric Key Encapsulation
+
+In the OpenSSL libraries, the corresponding method object is B<EVP_KEM>.
+The number for this operation is B<OSSL_OP_KEM>.
+The functions the provider can offer are described in L<provider-kem(7)>
+
 =item Encoding
 
 In the OpenSSL libraries, the corresponding method object is