OSSL_HTTP_parse_url(): Handle any userinfo, query, and fragment components

Now handle [http[s]://][userinfo@]host[:port][/path][?query][#frag]
by optionally providing any userinfo, query, and frag components.

All usages of this function, which are client-only,
silently ignore userinfo and frag components,
while the query component is taken as part of the path.
Update and extend the unit tests and all affected documentation.
Document and deprecat OCSP_parse_url().

Fixes an issue that came up when discussing FR #14001.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/14009)

5 files changed, 106 insertions, 44 deletions

diff --git a/doc/man1/openssl-cmp.pod.in b/doc/man1/openssl-cmp.pod.in
index dcb3ceedac..640505e4fb 100644
--- a/doc/man1/openssl-cmp.pod.in
+++ b/doc/man1/openssl-cmp.pod.in
@@ -47,9 +47,9 @@ Certificate enrollment and revocation options:
 
 Message transfer options:
 
-[B<-server> I<[http[s]://]address[:port][/path]>]
+[B<-server> I<[http[s]://][userinfo@]host[:port][/path][?query][#fragment]>]
 [B<-path> I<remote_path>]
-[B<-proxy> I<[http[s]://]address[:port][/path]>]
+[B<-proxy> I<[http[s]://][userinfo@]host[:port][/path][?query][#fragment]>]
 [B<-no_proxy> I<addresses>]
 [B<-msg_timeout> I<seconds>]
 [B<-total_timeout> I<seconds>]
@@ -429,11 +429,13 @@ Reason numbers defined in RFC 5280 are:
 
 =over 4
 
-=item B<-server> I<[http[s]://]address[:port][/path]>
+=item B<-server> I<[http[s]://][userinfo@]host[:port][/path][?query][#fragment]>
 
 The IP address or DNS hostname and optionally port (defaulting to 80 or 443)
 of the CMP server to connect to using HTTP(S) transport.
-The optional I<http://> or I<https://> prefix is ignored.
+The scheme I<https> may be given only if the B<tls_used> option is used.
+The optional userinfo and fragment components are ignored.
+Any given query component is handled as part of the path component.
 If a path is included it provides the default value for the B<-path> option.
 
 =item B<-path> I<remote_path>
@@ -441,11 +443,13 @@ If a path is included it provides the default value for the B<-path> option.
 HTTP path at the CMP server (aka CMP alias) to use for POST requests.
 Defaults to any path given with B<-server>, else C<"/">.
 
-=item B<-proxy> I<[http[s]://]address[:port][/path]>
+=item B<-proxy> I<[http[s]://][userinfo@]host[:port] [/path][?query][#fragment]>
 
 The HTTP(S) proxy server to use for reaching the CMP server unless B<no_proxy>
 applies, see below.
-The optional I<http://> or I<https://> prefix and any trailing path are ignored.
+The optional I<http://> or I<https://> prefix is ignored (note that TLS may be
+selected by B<tls_used>), as well as any path, userinfo, and query, and fragment
+components.
 Defaults to the environment variable C<http_proxy> if set, else C<HTTP_PROXY>
 in case no TLS is used, otherwise C<https_proxy> if set, else C<HTTPS_PROXY>.
 
diff --git a/doc/man1/openssl-ocsp.pod.in b/doc/man1/openssl-ocsp.pod.in
index 8cd9c7de19..96fe6acbc9 100644
--- a/doc/man1/openssl-ocsp.pod.in
+++ b/doc/man1/openssl-ocsp.pod.in
@@ -157,6 +157,8 @@ with B<-serial>, B<-cert> and B<-host> options).
 =item B<-url> I<responder_url>
 
 Specify the responder URL. Both HTTP and HTTPS (SSL/TLS) URLs can be specified.
+The optional userinfo and fragment components are ignored.
+Any given query component is handled as part of the path component.
 
 =item B<-host> I<hostname>:I<port>, B<-path> I<pathname>
 
diff --git a/doc/man1/openssl-s_server.pod.in b/doc/man1/openssl-s_server.pod.in
index cb6a1378a0..2bc307dca1 100644
--- a/doc/man1/openssl-s_server.pod.in
+++ b/doc/man1/openssl-s_server.pod.in
@@ -470,6 +470,8 @@ Sets the timeout for OCSP response to I<int> seconds.
 Sets a fallback responder URL to use if no responder URL is present in the
 server certificate. Without this option an error is returned if the server
 certificate does not contain a responder address.
+The optional userinfo and fragment URL components are ignored.
+Any given query component is handled as part of the path component.
 
 =item B<-status_file> I<infile>
 
diff --git a/doc/man3/OSSL_HTTP_parse_url.pod b/doc/man3/OSSL_HTTP_parse_url.pod
new file mode 100644
index 0000000000..5933e954da
--- /dev/null
+++ b/doc/man3/OSSL_HTTP_parse_url.pod
@@ -0,0 +1,73 @@
+=pod
+
+=head1 NAME
+
+OSSL_HTTP_parse_url,
+OCSP_parse_url
+- http utility functions
+
+=head1 SYNOPSIS
+
+ #include <openssl/http.h>
+
+ int OSSL_HTTP_parse_url(const char *url,
+                         int *pssl, char **puser, char **phost,
+                         char **pport, int *pport_num,
+                         char **ppath, char **pquery, char **pfrag);
+
+Deprecated since OpenSSL 3.0, can be hidden entirely by defining
+B<OPENSSL_API_COMPAT> with a suitable version value, see
+L<openssl_user_macros(7)>:
+
+ int OCSP_parse_url(const char *url, char **phost, char **pport, char **ppath,
+                    int *pssl);
+
+=head1 DESCRIPTION
+
+OSSL_HTTP_parse_url() parses its input string I<url> as a URL of the form
+C<[http[s]://][userinfo@]host[:port][/path][?query][#fragment]> and splits it up
+into userinfo, host, port, path, query, and fragment components
+and a flag indicating whether it begins with C<https>.
+The host component may be a DNS name or an IP address
+where IPv6 addresses should be enclosed in square brackets C<[> and C<]>.
+The port component is optional and defaults to "443" for HTTPS, else "80".
+If given, it must be in decimal form.  If the I<pport_num> argument is not NULL
+the integer value of the port number is assigned to I<*pport_num> on success.
+The path component is also optional and defaults to C</>.
+If I<pssl> is not NULL, I<*pssl> is assigned 1 in case parsing was successful
+and the schema part is present and is C<https>, else 0.
+Each non-NULL result pointer argument I<puser>, I<phost>, I<pport>, I<ppath>,
+I<pquery>, and I<pfrag>, is assigned the respective url component.
+On success, they are guaranteed to contain non-NULL string pointers, else NULL.
+It is the reponsibility of the caller to free them using L<OPENSSL_free(3)>.
+If I<pquery> is NULL, any given query component is handled as part of the path.
+A string returned via I<*ppath> is guaranteed to begin with a C</> character.
+For absent userinfo, query, and fragment components an empty string is given.
+
+Calling the deprecated fucntion OCSP_parse_url(url, host, port, path, ssl) is
+equivalent to OSSL_HTTP_parse_url(url, ssl, NULL, host, port, NULL, path, NULL, NULL).
+
+=head1 RETURN VALUES
+
+OSSL_HTTP_parse_url() and OCSP_parse_url()
+return 1 on success, 0 on error.
+
+=head1 SEE ALSO
+
+L<OSSL_HTTP_transfer(3)>
+
+=head1 HISTORY
+
+OOSSL_HTTP_parse_url() was added in OpenSSL 3.0.
+OCSP_parse_url() was deprecated in OpenSSL 3.0.
+
+=head1 COPYRIGHT
+
+Copyright 2019-2021 The OpenSSL Project Authors. All Rights Reserved.
+
+Licensed under the Apache License 2.0 (the "License").  You may not use
+this file except in compliance with the License.  You can obtain a copy
+in the file LICENSE in the source distribution or at
+L<https://www.openssl.org/source/license.html>.
+
+=cut
diff --git a/doc/man3/OSSL_HTTP_transfer.pod b/doc/man3/OSSL_HTTP_transfer.pod
index cb38d0124f..7de213670d 100644
--- a/doc/man3/OSSL_HTTP_transfer.pod
+++ b/doc/man3/OSSL_HTTP_transfer.pod
@@ -7,8 +7,7 @@ OSSL_HTTP_get_asn1,
 OSSL_HTTP_post_asn1,
 OSSL_HTTP_transfer,
 OSSL_HTTP_bio_cb_t,
-OSSL_HTTP_proxy_connect,
-OSSL_HTTP_parse_url
+OSSL_HTTP_proxy_connect
 - http client functions
 
 =head1 SYNOPSIS
@@ -22,15 +21,15 @@ OSSL_HTTP_parse_url
                     OSSL_HTTP_bio_cb_t bio_update_fn, void *arg,
                     const STACK_OF(CONF_VALUE) *headers,
                     int maxline, unsigned long max_resp_len, int timeout,
-                    const char *expected_content_type, int expect_asn1);
+                    const char *expected_ct, int expect_asn1);
  ASN1_VALUE *OSSL_HTTP_get_asn1(const char *url,
                                 const char *proxy, const char *no_proxy,
                                 BIO *bio, BIO *rbio,
                                 OSSL_HTTP_bio_cb_t bio_update_fn, void *arg,
                                 const STACK_OF(CONF_VALUE) *headers,
                                 int maxline, unsigned long max_resp_len,
-                                int timeout, const char *expected_content_type,
-                                const ASN1_ITEM *it);
+                                int timeout, const char *expected_ct,
+                                const ASN1_ITEM *rsp_it);
  ASN1_VALUE *OSSL_HTTP_post_asn1(const char *server, const char *port,
                                  const char *path, int use_ssl,
                                  const char *proxy, const char *no_proxy,
@@ -54,27 +53,26 @@ OSSL_HTTP_parse_url
  int OSSL_HTTP_proxy_connect(BIO *bio, const char *server, const char *port,
                              const char *proxyuser, const char *proxypass,
                              int timeout, BIO *bio_err, const char *prog);
- int OSSL_HTTP_parse_url(const char *url, char **phost, char **pport,
-                         int *pport_num, char **ppath, int *pssl);
 
 =head1 DESCRIPTION
 
 OSSL_HTTP_get() uses HTTP GET to obtain data (of any type) from the given I<url>
 and returns it as a memory BIO.
+If the schema component of the I<url> is C<https> a TLS connection is requested
+and the I<bio_update_fn> parameter, described below, must be provided.
+Any userinfo and fragment components in the I<url> are ignored.
+Any query component is handled as part of the path component.
 
-OSSL_HTTP_get_asn1() uses HTTP GET to obtain an ASN.1-encoded value
-(e.g., an X.509 certificate) with the expected structure specified by I<it>
-(e.g., I<ASN1_ITEM_rptr(X509)>) from the given I<url>
+OSSL_HTTP_get_asn1() is like OSSL_HTTP_get() but in addition
+parses the received contents (e.g., an X.509 certificate)
+as an ASN.1 DER encoded value with the expected structure specified by I<rsp_it>
 and returns it on success as a pointer to I<ASN1_VALUE>.
 
-OSSL_HTTP_post_asn1() uses the HTTP POST method to send a request I<req>
-with the ASN.1 structure defined in I<req_it> and the given I<content_type> to
-the given I<server> and optional I<port> and I<path>.
+OSSL_HTTP_post_asn1() is like OSSL_HTTP_get_asn1() but uses the HTTP POST method
+to send a request I<req> with the ASN.1 structure defined in I<req_it> and the
+given I<content_type> to the given I<server> and optional I<port> and I<path>.
 If I<use_ssl> is nonzero a TLS connection is requested and the I<bio_update_fn>
 parameter, described below, must be provided.
-The optional list I<headers> may contain additional custom HTTP header lines.
-The expected structure of the response is specified by I<rsp_it>.
-On success it returns the response as a pointer to B<ASN1_VALUE>.
 
 OSSL_HTTP_transfer() exchanges any form of HTTP request and response.
 It implements the core of the functions described above.
@@ -137,7 +135,7 @@ an ASN.1-encoded response is expected, which should include a total length,
 the length indications received are checked for consistency
 and for not exceeding the maximum response length.
 
-If the parameter I<expected_content_type> (or I<expected_ct>, respectively)
+If the parameter I<expected_ct>
 is not NULL then the HTTP client checks that the given content type string
 is included in the HTTP header of the response and returns an error if not.
 
@@ -192,24 +190,6 @@ Since this function is typically called by applications such as
 L<openssl-s_client(1)> it uses the I<bio_err> and I<prog> parameters (unless
 NULL) to print additional diagnostic information in a user-oriented way.
 
-OSSL_HTTP_parse_url() parses its input string I<url> as a URL
-of the form C<[http[s]://]address[:port][/path]> and splits it up into host,
-port, and path components and a flag indicating whether it begins with 'https'.
-The host component may be a DNS name or an IP address
-where IPv6 addresses should be enclosed in square brackets C<[> and C<]>.
-The port component is optional and defaults to "443" for HTTPS, else "80".
-If the I<pport_num> argument is NULL the port specification
-can be in mnemonic form such as "http" like with L<BIO_set_conn_port(3)>, else
-it must be in numerical form and its integer value is assigned to I<*pport_num>.
-The path component is also optional and defaults to "/".
-On success the function assigns via each non-NULL result pointer argument
-I<phost>, I<pport>, I<pport_num>, I<ppath>, and I<pssl>
-the respective url component.
-On error, I<*phost>, I<*pport>, and I<*ppath> are assigned to NULL,
-else they are guaranteed to contain non-NULL string pointers.
-It is the reponsibility of the caller to free them using L<OPENSSL_free(3)>.
-A string returned via I<*ppath> is guaranteed to begin with a C</> character.
-
 =head1 NOTES
 
 The names of the environment variables used by this implementation:
@@ -224,17 +204,18 @@ OSSL_HTTP_transfer() return a memory BIO containing the data received via HTTP.
 This must be freed by the caller. On failure, NULL is returned.
 Failure conditions include connection/transfer timeout, parse errors, etc.
 
-OSSL_HTTP_proxy_connect() and OSSL_HTTP_parse_url()
-return 1 on success, 0 on error.
+OSSL_HTTP_proxy_connect() returns 1 on success, 0 on error.
 
 =head1 SEE ALSO
 
+L<OSSL_HTTP_parse_url(3)>
 L<BIO_set_conn_port(3)>
 
 =head1 HISTORY
 
 OSSL_HTTP_get(), OSSL_HTTP_get_asn1(), OSSL_HTTP_post_asn1(),
-OSSL_HTTP_proxy_connect(), and OSSL_HTTP_parse_url() were added in OpenSSL 3.0.
+OSSL_HTTP_transfer(), and OSSL_HTTP_proxy_connect()
+were added in OpenSSL 3.0.
 
 =head1 COPYRIGHT