HTTP client: Allow streaming of response data (with possibly indefinite length)

Also clean up max_resp_len and add OSSL_HTTP_REQ_CTX_get_resp_len().

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15053)

2 files changed, 12 insertions, 11 deletions

diff --git a/doc/man3/OSSL_HTTP_REQ_CTX.pod b/doc/man3/OSSL_HTTP_REQ_CTX.pod
index f5f70584df..99396dfe7e 100644
--- a/doc/man3/OSSL_HTTP_REQ_CTX.pod
+++ b/doc/man3/OSSL_HTTP_REQ_CTX.pod
@@ -87,16 +87,19 @@ For example, to add a C<Host> header for C<example.com> you would call:
  OSSL_HTTP_REQ_CTX_add1_header(ctx, "Host", "example.com");
 
 OSSL_HTTP_REQ_CTX_set_expected() optionally sets in I<rctx> some expectations
-of the HTTP client.
+of the HTTP client on the response.
 Due to the structure of an HTTP request, if the I<keep_alive> argument is
 nonzero the function must be used before calling OSSL_HTTP_REQ_CTX_set1_req().
 If the I<content_type> parameter
 is not NULL then the client will check that the given content type string
 is included in the HTTP header of the response and return an error if not.
 If the I<asn1> parameter is nonzero a structure in ASN.1 encoding will be
-expected as the response content.  This means that an ASN.1 sequence header
-is required and the its length field is checked.
+expected as the response content and input streaming is disabled.  This means
+that an ASN.1 sequence header is required, its length field is checked, and
+OSSL_HTTP_REQ_CTX_get0_mem_bio() should be used to get the buffered response.
 Else any form of input is allowed without length checks, which is the default.
+In this case the BIO given as I<rbio> argument to OSSL_HTTP_REQ_CTX_new() should
+be used directly to read the response contents, which may support streaming.
 If the I<timeout> parameter is > 0 this indicates the maximum number of seconds
 the subsequent HTTP transfer (sending the request and receiving a response)
 is allowed to take.
@@ -121,9 +124,6 @@ All of this ends up in the internal memory B<BIO>.
 OSSL_HTTP_REQ_CTX_nbio() attempts to send the request prepared in I<rctx>
 and to gather the response via HTTP, using the I<wbio> and I<rbio>
 that were given when calling OSSL_HTTP_REQ_CTX_new().
-If successful, the contents of the internal memory B<BIO> contains
-the contents of the HTTP response, without the response headers.
-This does not support streaming.
 The function may need to be called again if its result is -1, which indicates
 L<BIO_should_retry(3)>.  In such a case it is advisable to sleep a little in
 between, using L<BIO_wait(3)> on the read BIO to prevent a busy loop.
diff --git a/doc/man3/OSSL_HTTP_transfer.pod b/doc/man3/OSSL_HTTP_transfer.pod
index 6760c515e7..0133122558 100644
--- a/doc/man3/OSSL_HTTP_transfer.pod
+++ b/doc/man3/OSSL_HTTP_transfer.pod
@@ -30,7 +30,7 @@ OSSL_HTTP_close
                            const STACK_OF(CONF_VALUE) *headers,
                            const char *content_type, BIO *req,
                            const char *expected_content_type, int expect_asn1,
-                           int timeout, int keep_alive);
+                           size_t max_resp_len, int timeout, int keep_alive);
  BIO *OSSL_HTTP_exchange(OSSL_HTTP_REQ_CTX *rctx, char **redirection_url);
  BIO *OSSL_HTTP_get(const char *url, const char *proxy, const char *no_proxy,
                     BIO *bio, BIO *rbio,
@@ -126,8 +126,6 @@ The I<buf_size> parameter specifies the response header maximum line length.
 A value <= 0 indicates that
 the B<HTTP_DEFAULT_MAX_LINE_LENGTH> of 4KiB should be used.
 This length is also used as the number of content bytes that are read at a time.
-The I<max_resp_len> specifies the maximum allowed response content length.
-The value 0 indicates B<HTTP_DEFAULT_MAX_RESP_LEN>, which currently is 100 KiB.
 
 If the I<overall_timeout> parameter is > 0 this indicates the maximum number of
 seconds the overall HTTP transfer (i.e., connection setup if needed,
@@ -161,6 +159,8 @@ is not NULL then the client will check that the given content type string
 is included in the HTTP header of the response and return an error if not.
 If the I<expect_asn1> parameter is nonzero,
 a structure in ASN.1 encoding will be expected as response content.
+The I<max_resp_len> parameter specifies the maximum allowed
+response content length, where the value 0 indicates no limit.
 If the I<timeout> parameter is > 0 this indicates the maximum number of seconds
 the subsequent HTTP transfer (sending the request and receiving a response)
 is allowed to take.
@@ -234,8 +234,9 @@ OSSL_HTTP_proxy_connect() and OSSL_HTTP_set_request()
 return 1 on success, 0 on error.
 
 On success, OSSL_HTTP_exchange(), OSSL_HTTP_get(), and OSSL_HTTP_transfer()
-return a memory BIO containing the data received.
-This must be freed by the caller.
+return a memory BIO containing the data received if an ASN.1-encoded response
+is expected, else a BIO that may support streaming.
+The BIO must be freed by the caller.
 On failure, they return NULL.
 Failure conditions include connection/transfer timeout, parse errors, etc.