cms encrypt, better OBJ_nid2obj() return check

Fixes #22225

In OBJ_nid2obj(), if the NID does not have an OID, then a pointer to
the special "undefined" ASN1_OBJECT is returned.  Check for the
undefined-ASN1_OBJECT and return an error.  Also, add a test for this
in 80-test_cms.t.

Testing:

  #!/bin/bash -x

  shopt -s expand_aliases

  alias openssl="LD_LIBRARY_PATH=~/git/openssl ~/git/openssl/apps/openssl"

  echo "This is a confidential message.  It should be encrypted." > msg.txt

  ## this should fail b/c there is no OID for aes-256-ctr
  openssl cms -encrypt -in msg.txt -aes-256-ctr -out msg.txt.cms -recip demos/cms/signer.pem
  echo $?

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/22392)

1 files changed, 12 insertions, 1 deletions

diff --git a/test/recipes/80-test_cms.t b/test/recipes/80-test_cms.t
index 3857916105..0e1ebc50cd 100644
--- a/test/recipes/80-test_cms.t
+++ b/test/recipes/80-test_cms.t
@@ -50,7 +50,7 @@ my ($no_des, $no_dh, $no_dsa, $no_ec, $no_ec2m, $no_rc2, $no_zlib)
 
 $no_rc2 = 1 if disabled("legacy");
 
-plan tests => 21;
+plan tests => 22;
 
 ok(run(test(["pkcs7_test"])), "test pkcs7");
 
@@ -1154,3 +1154,14 @@ with({ exit_checker => sub { return shift == 3; } },
             "issue#21986");
         }
     });
+
+# Test for problem reported in #22225
+with({ exit_checker => sub { return shift == 3; } },
+    sub {
+	ok(run(app(['openssl', 'cms', '-encrypt',
+		    '-in', srctop_file("test", "smcont.txt"),
+		    '-aes-256-ctr', '-recip',
+		    catfile($smdir, "smec1.pem"),
+		   ])),
+	   "Check for failure when cipher does not have an assigned OID (issue#22225)");
+     });