add an additional async notification communication method based on callback

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Paul Yang <yang.yang@baishancloud.com>
Signed-off-by: Ping Yu <ping.yu@intel.com>
Signed-off-by: Steven Linsell <stevenx.linsell@intel.com>

(Merged from https://github.com/openssl/openssl/pull/7573)

3 files changed, 185 insertions, 20 deletions

diff --git a/doc/man3/ASYNC_WAIT_CTX_new.pod b/doc/man3/ASYNC_WAIT_CTX_new.pod
index eeb2777a8b..9076be8171 100644
--- a/doc/man3/ASYNC_WAIT_CTX_new.pod
+++ b/doc/man3/ASYNC_WAIT_CTX_new.pod
@@ -4,13 +4,22 @@
 
 ASYNC_WAIT_CTX_new, ASYNC_WAIT_CTX_free, ASYNC_WAIT_CTX_set_wait_fd,
 ASYNC_WAIT_CTX_get_fd, ASYNC_WAIT_CTX_get_all_fds,
-ASYNC_WAIT_CTX_get_changed_fds, ASYNC_WAIT_CTX_clear_fd - functions to manage
-waiting for asynchronous jobs to complete
+ASYNC_WAIT_CTX_get_changed_fds, ASYNC_WAIT_CTX_clear_fd,
+ASYNC_WAIT_CTX_set_callback, ASYNC_WAIT_CTX_get_callback,
+ASYNC_WAIT_CTX_set_status, ASYNC_WAIT_CTX_get_status, ASYNC_callback_fn,
+ASYNC_STATUS_UNSUPPORTED, ASYNC_STATUS_ERR, ASYNC_STATUS_OK,
+ASYNC_STATUS_EAGAIN
+- functions to manage waiting for asynchronous jobs to complete
 
 =head1 SYNOPSIS
 
  #include <openssl/async.h>
 
+ #define ASYNC_STATUS_UNSUPPORTED    0
+ #define ASYNC_STATUS_ERR            1
+ #define ASYNC_STATUS_OK             2
+ #define ASYNC_STATUS_EAGAIN         3
+ typedef int (*ASYNC_callback_fn)(void *arg);
  ASYNC_WAIT_CTX *ASYNC_WAIT_CTX_new(void);
  void ASYNC_WAIT_CTX_free(ASYNC_WAIT_CTX *ctx);
  int ASYNC_WAIT_CTX_set_wait_fd(ASYNC_WAIT_CTX *ctx, const void *key,
@@ -26,6 +35,14 @@ waiting for asynchronous jobs to complete
                                     size_t *numaddfds, OSSL_ASYNC_FD *delfd,
                                     size_t *numdelfds);
  int ASYNC_WAIT_CTX_clear_fd(ASYNC_WAIT_CTX *ctx, const void *key);
+ int ASYNC_WAIT_CTX_set_callback(ASYNC_WAIT_CTX *ctx,
+                                 ASYNC_callback_fn callback,
+                                 void *callback_arg);
+ int ASYNC_WAIT_CTX_get_callback(ASYNC_WAIT_CTX *ctx,
+                                 ASYNC_callback_fn *callback,
+                                 void **callback_arg);
+ int ASYNC_WAIT_CTX_set_status(ASYNC_WAIT_CTX *ctx, int status);
+ int ASYNC_WAIT_CTX_get_status(ASYNC_WAIT_CTX *ctx);
 
 
 =head1 DESCRIPTION
@@ -103,14 +120,58 @@ code that the job should be resumed by making the wait file descriptor
 "readable". Once resumed the engine should clear the wake signal on the wait
 file descriptor.
 
+As well as a file descriptor, user code may also be notified via a callback. The
+callback and data pointers are stored within the ASYNC_WAIT_CTX along with an
+additional status field that can be used for the notification of retries from an
+engine. This additional method can be used when the user thinks that a file
+descriptor is too costly in terms of CPU cycles or in some context where a file
+descriptor is not appropriate.
+
+ASYNC_WAIT_CTX_set_callback() sets the callback and the callback argument. The
+callback will be called to notify user code when an engine completes a
+cryptography operation. It is a requirement that the callback function is small
+and non-blocking as it will be run in the context of a polling mechanism or an
+interrupt.
+
+ASYNC_WAIT_CTX_get_callback() returns the callback set in the ASYNC_WAIT_CTX
+structure.
+
+ASYNC_WAIT_CTX_set_status() allows an engine to set the current engine status.
+The possible status values are the following:
+ASYNC_STATUS_UNSUPPORTED: The engine does not support the callback mechanism.
+This is the default value. The engine must call ASYNC_WAIT_CTX_set_status() to
+set the status to some value other than ASYNC_STATUS_UNSUPPORTED if it intends
+to enable the callback mechanism.
+ASYNC_STATUS_ERR: The engine has a fatal problem with this request. The user
+code should clean up this session.
+ASYNC_STATUS_OK: The request has been successfully submitted.
+ASYNC_STATUS_EAGAIN: The engine has some problem which will be recovered soon,
+such as a buffer is full, so user code should resume the job.
+
+ASYNC_WAIT_CTX_get_status() allows user code to obtain the current status value.
+If the status is any value other than ASYNC_STATUS_OK then the user code should
+not expect to receive a callback from the engine even if one has been set.
+
+An example of the usage of the callback method might be the following. User
+code would initiate cryptographic operations, and the engine code would dispatch
+this operation to hardware, and if the dispatch is successful, then the engine
+code would call ASYNC_pause_job() to return control to the user code. After
+that, user code can perform other tasks. When the hardware completes the
+operation, normally it is detected by a polling function or an interrupt, as the
+user code set a callback by calling ASYNC_WAIT_CTX_set_callback() previously,
+then the registered callback will be called.
+
 =head1 RETURN VALUES
 
 ASYNC_WAIT_CTX_new() returns a pointer to the newly allocated ASYNC_WAIT_CTX or
 NULL on error.
 
 ASYNC_WAIT_CTX_set_wait_fd, ASYNC_WAIT_CTX_get_fd, ASYNC_WAIT_CTX_get_all_fds,
-ASYNC_WAIT_CTX_get_changed_fds and ASYNC_WAIT_CTX_clear_fd all return 1 on
-success or 0 on error.
+ASYNC_WAIT_CTX_get_changed_fds, ASYNC_WAIT_CTX_clear_fd,
+ASYNC_WAIT_CTX_set_callback, ASYNC_WAIT_CTX_get_callback and
+ASYNC_WAIT_CTX_set_status all return 1 on success or 0 on error.
+ASYNC_WAIT_CTX_get_status() returs the engine status.
+
 
 =head1 NOTES
 
@@ -132,6 +193,10 @@ ASYNC_WAIT_CTX_get_fd(), ASYNC_WAIT_CTX_get_all_fds(),
 ASYNC_WAIT_CTX_get_changed_fds() and ASYNC_WAIT_CTX_clear_fd()
 were added in OpenSSL 1.1.0.
 
+ASYNC_WAIT_CTX_set_callback(), ASYNC_WAIT_CTX_get_callback(),
+ASYNC_WAIT_CTX_set_status(), and ASYNC_WAIT_CTX_get_status()
+were added in OpenSSL 3.0.0.
+
 =head1 COPYRIGHT
 
 Copyright 2016 The OpenSSL Project Authors. All Rights Reserved.
diff --git a/doc/man3/ASYNC_start_job.pod b/doc/man3/ASYNC_start_job.pod
index 95694b636a..5ac368d3ff 100644
--- a/doc/man3/ASYNC_start_job.pod
+++ b/doc/man3/ASYNC_start_job.pod
@@ -107,22 +107,26 @@ ASYNC_pause_job() is called whilst not within the context of a job then no
 action is taken and ASYNC_pause_job() returns immediately.
 
 ASYNC_get_wait_ctx() can be used to get a pointer to the ASYNC_WAIT_CTX
-for the B<job>. ASYNC_WAIT_CTXs can have a "wait" file descriptor associated
-with them. Applications can wait for the file descriptor to be ready for "read"
-using a system function call such as select or poll (being ready for "read"
-indicates that the job should be resumed). If no file descriptor is made
-available then an application will have to periodically "poll" the job by
-attempting to restart it to see if it is ready to continue.
-
-An example of typical usage might be an async capable engine. User code would
-initiate cryptographic operations. The engine would initiate those operations
-asynchronously and then call L<ASYNC_WAIT_CTX_set_wait_fd(3)> followed by
-ASYNC_pause_job() to return control to the user code. The user code can then
-perform other tasks or wait for the job to be ready by calling "select" or other
-similar function on the wait file descriptor. The engine can signal to the user
-code that the job should be resumed by making the wait file descriptor
-"readable". Once resumed the engine should clear the wake signal on the wait
-file descriptor.
+for the B<job>. ASYNC_WAIT_CTXs contain two different ways to notify
+applications that a job is ready to be resumed. One is a "wait" file
+descriptor, and the other is a "callback" mechanism.
+
+The "wait" file descriptor associated with ASYNC_WAIT_CTX is used for
+applications to wait for the file descriptor to be ready for "read" using a
+system function call such as select or poll (being ready for "read" indicates
+that the job should be resumed). If no file descriptor is made available then
+an application will have to periodically "poll" the job by attempting to restart
+it to see if it is ready to continue.
+
+ASYNC_WAIT_CTXs also have a "callback" mechanism to notify applications. The
+callback is set by an application, and it will be automatically called when an
+engine completes a cryptography operation, so that the application can resume
+the paused work flow without polling. An engine could be written to look whether
+the callback has been set. If it has then it would use the callback mechanism
+in preference to the file descriptor notifications. If a callback is not set
+then the engine may use file descriptor based notifications. Please note that
+not all engines may support the callback mechanism, so the callback may not be
+used even if it has been set. See ASYNC_WAIT_CTX_new() for more details.
 
 The ASYNC_block_pause() function will prevent the currently active job from
 pausing. The block will remain in place until a subsequent call to
diff --git a/doc/man3/SSL_set_async_callback.pod b/doc/man3/SSL_set_async_callback.pod
new file mode 100644
index 0000000000..5e7f61eaf1
--- /dev/null
+++ b/doc/man3/SSL_set_async_callback.pod
@@ -0,0 +1,96 @@
+=pod
+
+=head1 NAME
+
+SSL_CTX_set_async_callback,
+SSL_CTX_set_async_callback_arg,
+SSL_set_async_callback,
+SSL_set_async_callback_arg,
+SSL_get_async_status,
+SSL_async_callback_fn
+- manage asynchronous operations
+
+=head1 SYNOPSIS
+
+=for comment multiple includes
+
+ #include <openssl/ssl.h>
+
+ typedef int (*SSL_async_callback_fn)(SSL *s, void *arg);
+ int SSL_CTX_set_async_callback(SSL_CTX *ctx, SSL_async_callback_fn callback);
+ int SSL_CTX_set_async_callback_arg(SSL_CTX *ctx, void *arg);
+ int SSL_set_async_callback(SSL *s, SSL_async_callback_fn callback);
+ int SSL_set_async_callback_arg(SSL *s, void *arg);
+ int SSL_get_async_status(SSL *s, int *status);
+
+=head1 DESCRIPTION
+
+SSL_CTX_set_async_callback() sets an asynchronous callback function. All SSL
+objects generated based on this SSL_CTX will get this callback. If an engine
+supports the callback mechanism, it will be automatically called if
+SSL_MODE_ASYNC has been set and an asynchronous capable engine completes a
+cryptography operation to notify the application to resume the paused work flow.
+
+SSL_CTX_set_async_callback_arg() sets the callback argument.
+
+SSL_set_async_callback() allows an application to set a callback in an
+asynchronous SSL object, so that when an engine completes a cryptography
+operation, the callback will be called to notify the application to resume the
+paused work flow.
+
+SSL_set_async_callback_arg() sets an argument for the SSL object when the above
+callback is called.
+
+SSL_get_async_status() returns the engine status. This function facilitates the
+communication from the engine to the application. During an SSL session,
+cryptographic operations are dispatched to an engine. The engine status is very
+useful for an application to know if the operation has been successfully
+dispatched. If the engine does not support this additional callback method,
+"ASYNC_STATUS_UNSUPPORTED" will be returned. See ASYNC_WAIT_CTX_set_status() for
+a description of all of the status values.
+
+An example of the above functions would be the following.
+1. Application sets the async callback and callback data on an SSL connection
+by calling SSL_set_async_callback().
+2. Application sets SSL_MODE_ASYNC and makes an asynchronous SSL call
+3. OpenSSL submits the asynchronous request to the engine. If a retry occurs at
+this point then the status within the ASYNC_WAIT_CTX would be set and the async
+callback function would be called (goto Step 7).
+4. The OpenSSL engine pauses the current job and returns, so that the
+application can continue processing other connections.
+5. At a future point in time (probably via a polling mechanism or via an
+interrupt) the engine will become aware that the asynchronous request has
+finished processing.
+6. The engine will call the application's callback passing the callback data as
+a parameter.
+7. The callback function should then run. Note: it is a requirement that the
+callback function is small and non-blocking as it will be run in the context of
+a polling mechanism or an interrupt.
+8. It is the application's responsibility via the callback function to schedule
+recalling the OpenSSL asynchronous function and to continue processing.
+9. The callback function has the option to check the status returned via
+SSL_get_async_status() to determine whether a retry happened instead of the
+request being submitted, allowing different processing if required.
+
+=head1 RETURN VALUES
+
+SSL_CTX_set_async_callback(), SSL_set_async_callback(),
+SSL_CTX_set_async_callback_arg(), SSL_CTX_set_async_callback_arg() and
+SSL_get_async_status() return 1 on success or 0 on error.
+
+=head1 HISTORY
+
+SSL_CTX_set_async_callback(), SSL_CTX_set_async_callback_arg(),
+SSL_set_async_callback(), SSL_set_async_callback_arg() and
+SSL_get_async_status() were first added to OpenSSL 3.0.0.
+
+=head1 COPYRIGHT
+
+Copyright 2019 The OpenSSL Project Authors. All Rights Reserved.
+
+Licensed under the OpenSSL license (the "License").  You may not use
+this file except in compliance with the License.  You can obtain a copy
+in the file LICENSE in the source distribution or at
+L<https://www.openssl.org/source/license.html>.
+
+=cut