HTTP client: make server/proxy and port params more consistent; minor other improvements

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: David von Oheimb <david.von.oheimb@siemens.com>
(Merged from https://github.com/openssl/openssl/pull/11404)

3 files changed, 86 insertions, 55 deletions

diff --git a/doc/man3/OSSL_CMP_CTX_new.pod b/doc/man3/OSSL_CMP_CTX_new.pod
index 2c1ad59750..d070a59b0b 100644
--- a/doc/man3/OSSL_CMP_CTX_new.pod
+++ b/doc/man3/OSSL_CMP_CTX_new.pod
@@ -11,11 +11,10 @@ OSSL_CMP_CTX_set_log_cb,
 OSSL_CMP_CTX_set_log_verbosity,
 OSSL_CMP_CTX_print_errors,
 OSSL_CMP_CTX_set1_serverPath,
-OSSL_CMP_CTX_set1_serverName,
+OSSL_CMP_CTX_set1_server,
 OSSL_CMP_CTX_set_serverPort,
 OSSL_CMP_CTX_set1_proxy,
 OSSL_CMP_CTX_set1_no_proxy,
-OSSL_CMP_DEFAULT_PORT,
 OSSL_CMP_CTX_set_http_cb,
 OSSL_CMP_CTX_set_http_cb_arg,
 OSSL_CMP_CTX_get_http_cb_arg,
@@ -78,11 +77,10 @@ OSSL_CMP_CTX_set1_senderNonce
 
  /* message transfer: */
  int OSSL_CMP_CTX_set1_serverPath(OSSL_CMP_CTX *ctx, const char *path);
- int OSSL_CMP_CTX_set1_serverName(OSSL_CMP_CTX *ctx, const char *name);
+ int OSSL_CMP_CTX_set1_server(OSSL_CMP_CTX *ctx, const char *address);
  int OSSL_CMP_CTX_set_serverPort(OSSL_CMP_CTX *ctx, int port);
  int OSSL_CMP_CTX_set1_proxy(OSSL_CMP_CTX *ctx, const char *name);
  int OSSL_CMP_CTX_set1_no_proxy(OSSL_CMP_CTX *ctx, const char *names);
- #define OSSL_CMP_DEFAULT_PORT 80
  int OSSL_CMP_CTX_set_http_cb(OSSL_CMP_CTX *ctx, HTTP_bio_cb_t cb);
  int OSSL_CMP_CTX_set_http_cb_arg(OSSL_CMP_CTX *ctx, void *arg);
  void *OSSL_CMP_CTX_get_http_cb_arg(const OSSL_CMP_CTX *ctx);
@@ -306,19 +304,26 @@ It is similar to B<ERR_print_errors_cb()> but uses the CMP log callback function
 if set in the C<ctx> for uniformity with CMP logging if given. Otherwise it uses
 B<ERR_print_errors(3)> to print to STDERR (unless OPENSSL_NO_STDIO is defined).
 
-OSSL_CMP_CTX_set1_serverPath() sets the HTTP path of the CMP server on the host.
+OSSL_CMP_CTX_set1_serverPath() sets the HTTP path of the CMP server on the host,
+also known as "CMP alias".
+The default is "/".
 
-OSSL_CMP_CTX_set1_serverName() sets the given server Address (as IP or name)
-in the given OSSL_CMP_CTX structure.
+OSSL_CMP_CTX_set1_server() sets the given server B<address>
+(which may be a hostname or IP address or NULL) in the given B<ctx>.
 
 OSSL_CMP_CTX_set_serverPort() sets the port of the CMP server to connect to.
-Port defaults to OSSL_CMP_DEFAULT_PORT = 80 if not set explicitly.
-
-OSSL_CMP_CTX_set1_proxy() sets the HTTP proxy to be used for connecting to the
-CMP server.
-Defaults to the environment variable B<http_proxy> if set, else B<HTTP_PROXY>
-in case no TLS is used, otherwise B<https_proxy> if set, else B<HTTPS_PROXY>.
-The format is [http://]address[:port][/path] where the optional path is ignored.
+If not used or the B<port> argument is 0
+the default port applies, which is 80 for HTTP and 443 for HTTPS.
+
+OSSL_CMP_CTX_set1_proxy() sets the HTTP proxy to be used for connecting to
+the given CMP server unless overruled by any "no_proxy" settings (see below).
+If TLS is not used this defaults to the value of
+the environment variable B<http_proxy> if set, else B<HTTP_PROXY>.
+Otherwise defaults to the value of B<https_proxy> if set, else B<HTTPS_PROXY>.
+An empty proxy string specifies not to use a proxy.
+Else the format is I<[http[s]://]address[:port][/path]>,
+where any path given is ignored.
+The default port number is 80, or 443 in case "https:" is given.
 
 OSSL_CMP_CTX_set1_no_proxy() sets the list of server hostnames not to use
 an HTTP proxy for. The names may be separated by commas and/or whitespace.
@@ -606,7 +611,7 @@ All other functions return 1 on success, 0 on error.
 The following code does an Initialization Request:
 
         cmp_ctx = OSSL_CMP_CTX_new();
-        OSSL_CMP_CTX_set1_serverName(cmp_ctx, opt_serverName);
+        OSSL_CMP_CTX_set1_server(cmp_ctx, address);
         OSSL_CMP_CTX_set1_referenceValue(cmp_ctx, ref, ref_len);
         OSSL_CMP_CTX_set1_secretValue(cmp_ctx, sec, sec_len);
         OSSL_CMP_CTX_set0_newPkey(cmp_ctx, new_pkey, 1);
@@ -618,7 +623,7 @@ The following code does an Initialization Request using an
 external identity certificate (RFC 4210, Appendix E.7):
 
         cmp_ctx = OSSL_CMP_CTX_new();
-        OSSL_CMP_CTX_set1_serverName(cmp_ctx, sname);
+        OSSL_CMP_CTX_set1_server(cmp_ctx, sname);
         OSSL_CMP_CTX_set1_clCert(cmp_ctx, cl_cert);
         OSSL_CMP_CTX_set1_pkey(cmp_ctx, pkey);
         OSSL_CMP_CTX_set0_newPkey(cmp_ctx, new_pkey, 1);
@@ -633,7 +638,7 @@ which is trusted by the current CA the code will connect to.
 The following code does a Key Update Request:
 
         cmp_ctx = OSSL_CMP_CTX_new();
-        OSSL_CMP_CTX_set1_serverName(cmp_ctx, sname);
+        OSSL_CMP_CTX_set1_server(cmp_ctx, url);
         OSSL_CMP_CTX_set1_pkey(cmp_ctx, pkey);
         OSSL_CMP_CTX_set0_newPkey(cmp_ctx, new_pkey, 1);
         OSSL_CMP_CTX_set1_clCert(cmp_ctx, cl_cert);
@@ -646,7 +651,7 @@ including, as an example, the id-it-signKeyPairTypes OID and prints info on
 the General Response contents.
 
     cmp_ctx = OSSL_CMP_CTX_new();
-    OSSL_CMP_CTX_set1_serverName(cmp_ctx, sname);
+    OSSL_CMP_CTX_set1_server(cmp_ctx, sname);
     OSSL_CMP_CTX_set1_referenceValue(cmp_ctx, ref, ref_len);
     OSSL_CMP_CTX_set1_secretValue(cmp_ctx, sec, sec_len);
 
diff --git a/doc/man3/OSSL_CMP_MSG_http_perform.pod b/doc/man3/OSSL_CMP_MSG_http_perform.pod
index 92f6000867..6582d4413b 100644
--- a/doc/man3/OSSL_CMP_MSG_http_perform.pod
+++ b/doc/man3/OSSL_CMP_MSG_http_perform.pod
@@ -3,35 +3,46 @@
 =head1 NAME
 
 OSSL_CMP_MSG_http_perform
-- implementation of HTTP transfer for CMP messages
+- client-side HTTP(S) transfer of a CMP request-response pair
 
 =head1 SYNOPSIS
 
  #include <openssl/cmp.h>
 
- SSL_CMP_MSG *OSSL_CMP_MSG_http_perform(OSSL_CMP_CTX *ctx,
-                                        const OSSL_CMP_MSG *req);
+ OSSL_CMP_MSG *OSSL_CMP_MSG_http_perform(OSSL_CMP_CTX *ctx,
+                                         const OSSL_CMP_MSG *req);
 
 =head1 DESCRIPTION
 
-This is the API for creating a BIO for CMP (Certificate Management
-Protocol) over HTTP(S) with OpenSSL.
-
-OSSL_CMP_MSG_http_perform() sends the given PKIMessage req to the CMP server
-specified in ctx. On success it returns the server's response.
+OSSL_CMP_MSG_http_perform() sends the given PKIMessage B<req>
+to the CMP server specified in B<ctx> via L<OSSL_CMP_CTX_set1_server(3)>
+and optionally L<OSSL_CMP_CTX_set_serverPort(3)>, using
+any "CMP alias" optionally specified via L<OSSL_CMP_CTX_set1_serverPath(3)>.
+The default port is 80 for HTTP and 443 for HTTPS; the default path is "/".
+On success the function returns the server's response PKIMessage.
+
+The function makes use of any HTTP callback function
+set via L<OSSL_CMP_CTX_set_http_cb(3)>.
+It respects any timeout value set via L<OSSL_CMP_CTX_set_option(3)>
+with an B<OSSL_CMP_OPT_MSG_TIMEOUT> argument.
+It also respects any HTTP(S) proxy options set via L<OSSL_CMP_CTX_set1_proxy(3)>
+and L<OSSL_CMP_CTX_set1_no_proxy(3)> and the respective environment variables.
+Proxying plain HTTP is supported directly,
+while using a proxy for HTTPS connections requires a suitable callback function
+such as L<OSSL_HTTP_proxy_connect(3)>.
 
 =head1 NOTES
 
-CMP is defined in RFC 4210 (and CRMF in RFC 4211).
+CMP is defined in RFC 4210.
+HTTP transfer for CMP is defined in RFC 6712.
 
 =head1 RETURN VALUES
 
-OSSL_CMP_MSG_http_perform() returns a message on success or else NULL.
-It uses ctx->http_cb if set and respects ctx->msgTimeOut.
+OSSL_CMP_MSG_http_perform() returns a CMP message on success, else NULL.
 
 =head1 SEE ALSO
 
-L<OSSL_CMP_CTX_new(3)>, L<OSSL_CMP_exec_IR_ses(3)>
+L<OSSL_CMP_CTX_new(3)>, L<OSSL_HTTP_proxy_connect(3)>.
 
 =head1 HISTORY
 
diff --git a/doc/man3/OSSL_HTTP_transfer.pod b/doc/man3/OSSL_HTTP_transfer.pod
index 27f438114e..632f48dbe8 100644
--- a/doc/man3/OSSL_HTTP_transfer.pod
+++ b/doc/man3/OSSL_HTTP_transfer.pod
@@ -69,17 +69,18 @@ and returns it on success as a pointer to I<ASN1_VALUE>.
 
 OSSL_HTTP_post_asn1() uses the HTTP POST method to send a request B<req>
 with the ASN.1 structure defined in B<req_it> and the given B<content_type> to
-the given B<server> and optional B<port> and B<path>, which defaults to "/".
+the given B<server> and optional B<port> and B<path>.
 If B<use_ssl> is nonzero a TLS connection is requested and the B<bio_update_fn>
 parameter, described below, must be provided.
 The optional list B<headers> may contain additional custom HTTP header lines.
 The expected structure of the response is specified by B<rsp_it>.
 On success it returns the response as a pointer to B<ASN1_VALUE>.
 
-OSSL_HTTP_transfer() exchanges an HTTP request and response with
-the given B<server> and optional B<port> and B<path>, which defaults to "/".
-If B<use_ssl> is nonzero a TLS connection is requested and the B<bio_update_fn>
-parameter, described below, must be provided.
+OSSL_HTTP_transfer() exchanges any form of HTTP request and response.
+It implements the core of the functions described above.
+If B<path> parameter is NULL it defaults to "/".
+If B<use_ssl> is nonzero a TLS connection is requested
+and the B<bio_update_fn> parameter, described below, must be provided.
 If B<req_mem> is NULL it uses the HTTP GET method, else it uses HTTP POST to
 send a request with the contents of the memory BIO and optional B<content_type>.
 The optional list B<headers> may contain additional custom HTTP header lines.
@@ -91,30 +92,36 @@ L<OPENSSL_free(3)>.
 
 The above functions have the following parameters in common.
 
-If the optional B<proxy> parameter contains a non-empty string or
-it is NULL and one of the environment variables B<http_proxy> and B<HTTP_PROXY>
-(or B<https_proxy> and B<HTTPS_PROXY>, respectively, in case B<use_ssl> != 0)
-is set and contains a non-empty string this is used as the candidate address
-of HTTP(S) proxy to use.
-The address may include a port specification separated by ':'.
-Any prefix "http://" and any trailing string starting with '/' is ignored.
-The HTTP client functions connect via the given proxy unless the B<server>
-is found in the optional list B<no_proxy> of proxy hostnames (if it is not NULL,
-else in the environment variable B<no_proxy> if set or else in B<HTTP_PROXY>).
-Proxying plain HTTP is supported directly,
-while using a proxy for HTTPS connections requires a suitable callback function
-such as OSSL_HTTP_proxy_connect(), described below.
-
-Typically the B<bio> and B<rbio> parameters are NULL and the client creates a
-network BIO internally for connecting to the given server and port (optionally
-via a proxy and its port), and uses it for exchanging the request and response.
-If B<bio> is given and B<rbio> is NULL then the client uses this BIO instead.
+Typically the OpenSSL build supports sockets
+and the B<bio> and B<rbio> parameters are both NULL.
+In this case the client creates a network BIO internally
+for connecting to the given B<server>
+at the specified B<port> (if any, defaulting to 80 for HTTP or 443 for HTTPS),
+optionally via a B<proxy> (respecting B<no_proxy>) as described below.
+Then the client uses this internal BIO for exchanging the request and response.
+If B<bio> is given and B<rbio> is NULL then the client uses this B<bio> instead.
 If both B<bio> and B<rbio> are given (which may be memory BIOs for instance)
 then no explicit connection is attempted,
 B<bio> is used for writing the request, and B<rbio> for reading the response.
 As soon as the client has flushed B<bio> the server must be ready to provide
 a response or indicate a waiting condition via B<rbio>.
 
+The optional B<proxy> parameter can be used to set the address of the an
+HTTP(S) proxy to use (unless overridden by "no_proxy" settings).
+If TLS is not used this defaults to the environment variable B<http_proxy>
+if set, else B<HTTP_PROXY>.
+If B<use_ssl> != 0 it defaults to B<https_proxy> if set, else B<HTTPS_PROXY>.
+An empty proxy string specifies not to use a proxy.
+Else the format is I<[http[s]://]address[:port][/path]>,
+where any path given is ignored.
+The default proxy port number is 80, or 443 in case "https:" is given.
+The HTTP client functions connect via the given proxy unless the B<server>
+is found in the optional list B<no_proxy> of proxy hostnames (if not NULL;
+default is the environment variable B<no_proxy> if set, else B<NO_PROXY>).
+Proxying plain HTTP is supported directly,
+while using a proxy for HTTPS connections requires a suitable callback function
+such as B<OSSL_HTTP_proxy_connect()>, described below.
+
 The B<maxline> parameter specifies the response header maximum line length,
 where 0 indicates the default value, which currently is 4k.
 The B<max_resp_len> parameter specifies the maximum response length,
@@ -123,7 +130,7 @@ where 0 indicates the default value, which currently is 100k.
 An ASN.1-encoded response is expected by OSSL_HTTP_get_asn1() and
 OSSL_HTTP_post_asn1(), while for OSSL_HTTP_get() or OSSL_HTTP_transfer()
 this is only the case if the B<expect_asn1> parameter is nonzero.
-If the response header contains one or more Content-Length header lines and/or
+If the response header contains one or more "Content-Length" header lines and/or
 an ASN.1-encoded response is expected, which should include a total length,
 the length indications received are checked for consistency
 and for not exceeding the maximum response length.
@@ -172,11 +179,12 @@ Here is a simple example that supports TLS connections (but not via a proxy):
 After disconnect the modified BIO will be deallocated using BIO_free_all().
 
 OSSL_HTTP_proxy_connect() may be used by an above BIO connect callback function
-to set up an SSL/TLS connection via an HTTP proxy.
+to set up an SSL/TLS connection via an HTTPS proxy.
 It promotes the given BIO B<bio> representing a connection
 pre-established with a TLS proxy using the HTTP CONNECT method,
 optionally using proxy client credentials B<proxyuser> and B<proxypass>,
 to connect with TLS protection ultimately to B<server> and B<port>.
+If the B<port> argument is NULL or the empty string it defaults to "443".
 The B<timeout> parameter is used as described above.
 Since this function is typically called by appplications such as
 L<openssl-s_client(1)> it uses the B<bio_err> and B<prog> parameters (unless
@@ -192,6 +200,13 @@ them copies of the respective string components.
 The strings returned this way must be deallocated by the caller using
 L<OPENSSL_free(3)> unless they are NULL, which is their default value on error.
 
+=head1 NOTES
+
+The names of the environment variables used by this implementation:
+B<http_proxy>, B<HTTP_PROXY>, B<https_proxy>, B<HTTPS_PROXY>, B<no_proxy>, and
+B<NO_PROXY>, have been chosen for maximal compatibility with
+other HTTP client implementations such as wget, curl, and git.
+
 =head1 RETURN VALUES
 
 OSSL_HTTP_get(), OSSL_HTTP_get_asn1(), OSSL_HTTP_post_asn1(), and