If an engine comes up explicitely, it must also come down explicitely

In apps/apps.c, one can set up an engine with setup_engine().
However, we freed the structural reference immediately, which means
that for engines that don't already have a structural reference
somewhere else (because it has registered at least one cipher or digest
algorithm method, and therefore gets a functional reference through the
ENGINE_set_default() call), we end up returning an invalid reference.

Instead, the function release_engine() is added, and called at the end
of the routines that call setup_engine().

Originally, the ENGINE API wasn't designed for this to happen, an
engine had to register at least one algorithm method, and was
especially expected to register the algorithms corresponding to the
key types that could be stored and hidden in hardware.  However, it
turns out that some engines will not register those algorithms with
the ENGINE_set_{algo}, ENGINE_set_cipher or ENGINE_set_digest
functions, as they only want the methods to be used for keys, not as
general crypto accelerator methods.  That may cause ENGINE_set_default()
to do nothing, and no functional reference is therefore made, leading
to a premature deallocation of the engine and it thereby becoming
unavailable when trying to fetch a key.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1644)

1 files changed, 4 insertions, 0 deletions

diff --git a/apps/genrsa.c b/apps/genrsa.c
index 91e6550a57..cd4490c25e 100644
--- a/apps/genrsa.c
+++ b/apps/genrsa.c
@@ -314,6 +314,10 @@ int MAIN(int argc, char **argv)
         RSA_free(rsa);
     if (out)
         BIO_free_all(out);
+# ifndef OPENSSL_NO_ENGINE
+    if (e != NULL)
+        release_engine(e);
+# endif
     if (passout)
         OPENSSL_free(passout);
     if (ret != 0)