QUIC: Server API Design Document

Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/21795)

1 files changed, 521 insertions, 0 deletions

diff --git a/doc/designs/quic-design/server/quic-server-api.md b/doc/designs/quic-design/server/quic-server-api.md
new file mode 100644
index 0000000000..3dfefa0b43
--- /dev/null
+++ b/doc/designs/quic-design/server/quic-server-api.md
@@ -0,0 +1,521 @@
+QUIC Server API Design
+======================
+
+Requirements
+------------
+
+Essential:
+
+- Support server mode.
+- Support multiple clients on one UDP socket/BIO.
+- Accept incoming clients and get a `SSL *` object which works just like a
+  client QCSO.
+
+Maybe:
+
+- Binding multiple BIOs (UDP sockets) to one listener?
+
+Implied:
+
+- Support connection migration and preferred addressing in the future.
+- Maximise reuse of existing APIs.
+
+Approach
+--------
+
+Two design sketches are shown below, their relative merits are evaluated, and
+one is chosen.
+
+Sketch A: `SSL_LISTENER`
+------------------------
+
+A new type `SSL_LISTENER` is introduced to the public API. This represents an
+entity which listens for incoming connections. It is **not** intended to be QUIC
+specific and provides an abstract interface which could be adapted to other
+cases in the future. For example, it could be used for TLS over TCP to make
+accepting incoming TCP connections easy.
+
+Internally, an `SSL_LISTENER` has a common header in much the same way that
+`SSL` objects now do. For example, if called using a QUIC `SSL_CTX`, the
+internal structure might be a `QUIC_LISTENER`, etc. If called using a TLS
+`SSL_CTX`, this might be a `TCP_LISTENER` doing something very ordinary. We can
+support DTLS in the future in the same way as QUIC, etc.
+
+Internally, an `SSL_LISTENER` stores any common state that needs to be held
+collectively for all connections created by that listener.
+
+`SSL_LISTENER` objects are refcounted and referenced by any connection which
+descends from them. Thus, an application may free the `SSL_LISTENER`, but
+it will be retained until it is no longer depended on by any connection.
+
+The operation of an `SSL_LISTENER` is simple: call `SSL_LISTENER_accept_conn`.
+For QUIC, this returns a new QCSO, which is considered a child of the listener.
+
+```c
+SSL_LISTENER *SSL_LISTENER_new(SSL_CTX *ctx, BIO *net_rbio, BIO *net_wbio);
+int SSL_LISTENER_up_ref(SSL_LISTENER *l);
+int SSL_LISTENER_free(SSL_LISTENER *l);
+
+BIO *SSL_LISTENER_get0_net_rbio(SSL_LISTENER *l);
+BIO *SSL_LISTENER_get0_net_wbio(SSL_LISTENER *l);
+int SSL_LISTENER_set1_net_rbio(SSL_LISTENER *l, BIO *net_rbio);
+int SSL_LISTENER_set1_net_wbio(SSL_LISTENER *l, BIO *net_wbio);
+
+SSL *SSL_LISTENER_accept_conn(SSL_LISTENER *l, uint64_t flags);
+size_t SSL_LISTENER_get_accept_queue_len(SSL_LISTENER *l);
+```
+
+### Discussion
+
+However, there are a number of disadvantages to introducing a new object type.
+The concept of a listener shares many of the same concepts inherent to
+supporting non-blocking network I/O which our SSL API for QUIC connections has
+been carefully designed to support:
+
+- Poll descriptors (`SSL_get_[rw]poll_descriptor`);
+- `SSL_net_(read|write)_desired`;
+- `SSL_handle_events`;
+- `SSL_get_event_timeout`;
+- BIO configuration (`SSL_set_[rw]bio`);
+- Refcounting (`SSL_up_ref`, `SSL_free`).
+
+We would need to duplicate all of these APIs. Moreover application developers
+would need to use both and the distinction between the two would become
+confusing.
+
+Sketch B: New SSL Object Type: Listener
+---------------------------------------
+
+Instead of the above, the definition of a new SSL object subtype — a Listener —
+is proposed. A QLSO (QUIC Listener SSL Object) is defined in much the same way
+that QCSOs and QSSOs are currently defined, using the common SSL structure
+header.
+
+Many existing APIs around the handling of non-blocking I/O can then be supported
+and work in exactly the same way.
+
+```c
+/*
+ * SSL_new_listener
+ * ----------------
+ *
+ * Creates a QLSO, assuming a QUIC SSL_CTX.
+ *
+ * TCP and DTLS can be supported in the future; not supported for non-QUIC
+ * SSL_CTXs for now.
+ */
+SSL *SSL_new_listener(SSL_CTX *ctx, uint64_t flags);
+
+/*
+ * A QLSO supports the following existing APIs:
+ *
+ *   - Poll descriptors (`SSL_get_[rw]poll_descriptor`)
+ *   - `SSL_net_(read|write)_desired`
+ *   - `SSL_handle_events`
+ *   - `SSL_get_event_timeout`
+ *   - BIO configuration (`SSL_set_[rw]bio`)
+ *   - Refcounting (`SSL_up_ref`, `SSL_free`)
+ */
+
+/*
+ * New API for QLSOs: SSL_accept_conn
+ * ----------------------------------
+ *
+ * These APIs are exactly the same as  SSL_accept_stream, but made a separate
+ * function to avoid confusion. Supported on QLSOs only.
+ */
+SSL *SSL_accept_conn(SSL *ssl, uint64_t flags);
+size_t SSL_get_accept_conn_queue_len(SSL *ssl);
+
+/*
+ * New API for QLSOs: SSL_listen
+ * -----------------------------
+ *
+ * This is not usually needed as SSL_accept_conn will implicitly trigger this if
+ * it has not been called yet. It is only needed if a server wishes to ensure it
+ * has started to accept incoming connections but does not wish to actually call
+ * SSL_accept_conn yet. In this regard it is similar to listen(2).
+ */
+int SSL_listen(SSL *ssl);
+
+/*
+ * New API for QCSOs: SSL_get_peer_addr
+ * ------------------------------------
+ *
+ * An API to retrieve a L4 peer address. Supported on QCSOs only.
+ */
+int SSL_get_peer_addr(SSL *ssl, BIO_ADDR *peer_addr);
+
+/* SSL_get_local_addr may be introduced once it is relevant (migration etc.) */
+
+/* Returns 1 for QLSOs (or future TLS/DTLS listeners) only. */
+int SSL_is_listener(SSL *ssl);
+
+/* SSL_is_quic returns 1 */
+```
+
+Selection of Design
+-------------------
+
+We could also consider using an `SSL_CTX` as the listener type. Since a
+`SSL_CTX` is intended to span the lifespan of multiple TLS connections there is
+a degree of logic here. However, this does not match up naturally because an
+`SSL_CTX` is not designed to hold network resources or manage network I/O. It
+does not have the facilities for polling and event handling that the `SSL`
+object API has been enhanced with. It would also be a fairly major paradigm
+shift for existing server applications and make things difficult if a server
+wanted to use different `SSL_CTX` instances to use different e.g. TLS settings
+within the same listener.
+
+Based on the merits of the above, Sketch B is chosen for adoption.
+
+Discussion of Relevant Issues
+-----------------------------
+
+### Notes on Internal Refactoring to Support Multiple Connections
+
+Currently a client connection is a QCSO
+(`QUIC_CONNECTION`) which contains a `QUIC_CHANNEL`. The `QUIC_CONNECTION`
+provides API Personality Layer (APL) implementation and wraps the APL-agnostic
+`QUIC_CHANNEL`. We currently have two APLs, the libssl API (“the” APL) and
+`QUIC_TSERVER` for internal testing use. `QUIC_XSO` is the APL object
+representing a QUIC stream and wraps our APL-agnostic channel-owned
+`QUIC_STREAM` object. There is no specific object in the `QUIC_TSERVER` APL for
+a stream as they are referenced in that API by stream ID only.
+
+`QUIC_CHANNEL` corresponds exactly to a QUIC connection. In a similar manner,
+`QUIC_STREAM` corresponds exactly to a QUIC stream. In server operation,
+multiple `QUIC_CHANNEL` instances will exist, one per accepted connection.
+
+Above we introduce the notion of a QUIC Listener SSL Object (QLSO), which
+internally will be represented by the type `QUIC_LISTENER`. This forms a new
+object in the libssl APL.
+
+With server operation, multiple connections will share a single UDP
+socket/network BIO. Certain resources currently tied 1:1 into a `QUIC_CHANNEL`
+will be broken out, such as the network BIO pointers and `QUIC_DEMUX` instance.
+A `QUIC_CHANNEL` continues to own e.g. the QRX/QTX as these are
+connection-specific.
+
+Because preserving the separation of the APL(s) and the QUIC core is desirable,
+the concept of a `QUIC_PORT` is introduced. This is the internal object tracking
+a demux and network BIOs which a `QUIC_CHANNEL` will belong to. In other words,
+the new situation is as follows:
+
+| Abbr. | libssl APL Object | QUIC Core Object |
+|-------|-------------------|------------------|
+| QLSO  | `QUIC_LISTENER`   | `QUIC_PORT`      |
+| QCSO  | `QUIC_CONNECTION` | `QUIC_CHANNEL`   |
+| QSSO  | `QUIC_XSO`        | `QUIC_STREAM`    |
+
+### Listener Flexibility
+
+A listener allows multiple connections to be received on a UDP socket. However,
+there is no intrinsic requirement in QUIC that a socket only be used for client
+use, or only for server use. Therefore, it should be possible for an application
+to create a QLSO, use it to accept connections, and also use it to initiate
+outgoing connections.
+
+This can be supported in a simple manner:
+
+```c
+SSL *SSL_new_from_listener(SSL *ssl);
+```
+
+This creates an unconnected (idle) QCSO from a QLSO. Use of
+`SSL_set1_initial_peer_addr` is mandatory in this case as peer address
+autodetection is inhibited.
+
+The created QCSO otherwise works in a completely normal fashion.
+
+### Client-Only Listeners
+
+In fact, the realisation of this hybrid capability is simply an emergent
+consequence of our internal refactor. Since a QUIC server will be split into an
+internal `QUIC_PORT` and zero or more `QUIC_CONNECTION` instances, so too will a
+client connection become simply a degenerate case where there is a `QUIC_PORT`
+and only one `QUIC_CONNECTION`. So the client case basically just becomes a
+simplified API where we spin up a `QUIC_PORT`/listener automatically to support
+a single connection.
+
+It is potentially desirable for a client to be able to use the listener
+API even if it is only initiating connections as a client and never accepting
+connections. There are several reasons why an application might want to do this:
+
+- it allows the application to centralise its I/O polling and networking arrangements;
+- it allows the application to handle the TERMINATING phase of connections in a
+  better (and more RFC-compliant) way.
+
+For the latter reason, consider how connection termination works. Once a QUIC
+Immediate Close is initiated using a `CONNECTION_CLOSE` frame, a connection
+enters the `TERMINATING` state and is expected to stay there for some period of
+time until the possibility of various contingencies (such as loss of a
+`CONNECTION_CLOSE` frame and its necessary retransmission) has been accounted
+for. However, many applications may not wish to hang around for potentially
+several seconds or longer while tearing down a QUIC connection.
+
+Allowing clients to use a listener resolves this problem, if the lifetime of the
+application exceeds the lifetime of most client connections. The listener can
+continue to hold the `QUIC_PORT` internally and respond to enquiries for
+terminating and terminated connections, even if (as far as the application is
+concerned) no client connections currently exist and all prior client
+connections have been freed using `SSL_free`. The internal `QUIC_CHANNEL` can be
+kept around by the `QUIC_PORT` until it reaches the `TERMINATED` state.
+
+A client-only listener can be created one of two ways:
+
+1. by calling `SSL_new_listener` while passing this flag:
+
+    ```
+    #define SSL_LISTENER_FLAG_NO_ACCEPT  (1U << 0)
+    ```
+
+2. by using a client `SSL_METHOD`, which implies `NO_ACCEPT`.
+
+This flag simply indicates that this listener is not to be used to accept
+incoming connections. A client can then use `SSL_new_from_listener` as usual to
+create an outgoing QCSO.
+
+### CID Length Determination
+
+Currently a client connection uses a zero-length local CID. Since we want to
+support multiple outgoing client connections on one socket these will need to
+support a non-zero-length local CID. Equally servers will need to support CIDs.
+
+By default when a QLSO is used, we will use a real, non-zero-length CID. If the
+existing QUIC client API is used (with no QLSO) we will likely spin up a
+`QUIC_PORT` internally but specially choose to use a zero-length local CID since
+we know this port will never be used for more than one connection.
+
+If an application creates a QLSO explicitly but knows it will not need to
+create more than one connection, it can pass the following flag when creating
+the listener:
+
+```c
+#define SSL_LISTENER_FLAG_SINGLE_CONN   (1U << 1)
+```
+
+Failure to do so is nonfatal and simply increases packet overheads.
+
+### Future Compatibility: TLS over TCP Support
+
+Assuming we choose to use the SSL object type as the basis of a new listener
+object type, let us consider how this API could also be used to support TLS over
+TCP, at least in a server role:
+
+```c
+/* Blocking example: */
+{
+    SSL *l, *conn;
+    BIO *tcp_listener = get_bio_sock_for_listening_tcp_socket();
+    SSL_CTX *ctx = create_ssl_ctx(TLS_server_method());
+    BIO_ADDR addr;
+
+    l = SSL_new_listener(ctx);
+    SSL_set_bio(l, tcp_listener, tcp_listener);
+
+    for (;;) {
+        conn = SSL_accept_conn(l, 0); /* standard SSL_CONNECTION is returned */
+
+        spawn_thread_to_process_conn(conn, &addr);
+    }
+}
+```
+
+Essentially, when `SSL_new_listener` is given a `SSL_CTX` which is using a TLS
+method, it returns some different kind of object, say a `TCP_LISTENER`, which
+also is a kind of SSL object. `SSL_accept_conn` calls `accept(2)` via some BIO
+abstraction on the provided BIO, and constructs a new BIO on the resulting
+socket, wrapping it via standard `SSL_new` using the same `SSL_CTX`.
+
+Thus, this API should be compatible with any future adaptation to also support
+TCP.
+
+### Future Compatibility: DTLS over UDP Support
+
+The relevant concerns regarding DTLS are likely to be highly similar to QUIC,
+so this API should also be adaptable to support DTLS in the future, providing
+a unified server-side API. `SSL_new_listener` will return e.g. a
+`DTLS_LISTENER`.
+
+### Supporting Multiple Sockets
+
+There are three means that we can provide to support a server which wants to
+listen on multiple UDP sockets:
+
+1. Allow a QLSO to bind multiple BIOs;
+2. Create multiple QLSOs and use some polling mechanism to service all of them;
+3. Create some higher level object which can aggregate multiple QLSOs.
+
+(1) would require more extensive API changes and should only be done if a
+compelling reason to do so arises. (3) is a more advanced design which is
+theoretically possible in future and is discussed in the next section.
+For now, we will support only (2) (which we can support via our planned polling
+API) until a compelling need arises.
+
+### Roles and Domains: Concepts and Future Evolution
+
+A client can use QUIC using a single QCSO without ever creating a QLSO.
+Conversely a client or server can use QUIC by creating a QLSO and then one or
+more subsidiary QCSOs. This implies that, at least in terms of the publicly
+visible API, the roles of certain essential aspects of QUIC can belong to
+different publicly visible API objects.
+
+The following concepts are introduced:
+
+- **Port Leader:** A Port Leader is the APL object which caused the creation
+  of a `QUIC_PORT`, which is responsible for servicing a particular pair
+  of network BIOs. Currently maps to one UDP socket.
+
+  In the simple client API, the QCSO is the port leader.
+  In a client using a QLSO, the QLSO is the port leader.
+  In a server, the QLSO is the port leader.
+
+- **Event Leader:** An Event Leader is the APL object which handles event
+  processing. Calls to SSL_handle_events() for any subsidiary object is
+  equivalent to a call to SSL_handle_events() on the Event Leader, which handles
+  state machine processing for that object and all subsidiary objects.
+
+  The event leader is the highest-level APL object in a QUIC APL object
+  hierarchy:
+
+  In the simple client API, the QCSO is the event leader.
+  In a client using a QLSO, the QLSO is the event leader.
+  In a server, the QLSO is the event leader.
+
+- **Domain:** A QUIC domain is a set of APL objects rooted in an Event Leader.
+  In other words, requests to handle events on subsidiary objects in a domain
+  are channeled up to the Event Leader and processed there. A QUIC domain is a
+  set of logically grouped QUIC objects which share a single I/O reactor and
+  event processing context.
+
+  A domain could contain multiple QLSOs (i.e., multiple UDP sockets) in future.
+  In this case QLSOs may cease to be the most senior APL object type and there
+  may be some new hypothetical SSL_new_quic_domain() call.
+
+  However, there is no plan to implement domains at this time; for the time
+  being, this is purely conceptual and ensures our architecture can grow to
+  supporting multiple sockets and listeners in a given QUIC context if need be.
+
+  If ever implemented, a domain will be reified by a `QUIC_DOMAIN` structure as
+  part of the APL and a `QUIC_ENGINE` structure as part of the QUIC core. The
+  SSL object type will be abbreviated QDSO (QUIC Domain SSL Object).
+
+The introduction of the concept of a domain demonstrates the value of the Port
+Leader and Domain Leader concepts going forward:
+
+Example 1 — Explicitly Created Domain:
+
+```text
+QDSO (QUIC_DOMAIN/QUIC_ENGINE)              Event Leader
+    QLSO (QUIC_LISTENER/QUIC_PORT)          Port Leader
+        QCSO (QUIC_CONNECTION/QUIC_CHANNEL)
+            QSSO (QUIC_XSO/QUIC_STREAM)
+```
+
+Example 2 - Explicitly Created QLSO, Implicitly Created Domain:
+
+```text
+QLSO (QUIC_LISTENER/QUIC_PORT)              Event Leader, Port Leader
+    [QUIC_ENGINE]
+    QCSO (QUIC_CONNECTION/QUIC_CHANNEL)
+        QSSO (QUIC_XSO/QUIC_STREAM)
+```
+
+Here we see that the QLSO now has the Event Leader role, as it is the most
+senior object visible publicly as a constructed API object. Note that a
+QUIC_ENGINE will be created internally to allow the QLSO to service the Event
+Leader role, but the corresponding QUIC_DOMAIN APL object will not exist, as
+this is not API-visible.
+
+Example 3 - Explicitly Created Single Client QCSO:
+
+```text
+QCSO (QUIC_CONNECTION/QUIC_CHANNEL)         Event Leader, Port Leader
+    [QUIC_ENGINE]
+    [QUIC_PORT]
+    QSSO (QUIC_XSO/QUIC_STREAM)
+```
+
+Here we see that the QCSO now has the Port Leader role. Note that a QUIC_PORT
+will be created internally to allow the QCSO to service the Port Leader role,
+but the corresponding QUIC_LISTENER APL object will not exist, as this is not
+API-visible. Similarly a QUIC_ENGINE is spun up to handle the Event Leader role,
+as above.
+
+It should be emphasised that everything in this section is largely academic and
+intended to demonstrate potential future directions and the adaptability of the
+design to future evolution. There is no immediate plan to implement a QDSO
+object type in the APL. However it will be viable to do so if we want to support
+multiple sockets per domain someday.
+
+| Abbr. | libssl APL Object | QUIC Core Object | Seniority   | Potential Roles |
+|-------|-------------------|------------------|-------------|-----------------|
+| QDSO  | `QUIC_DOMAIN`     | `QUIC_ENGINE`    | 1 (Highest) | EL              |
+| QLSO  | `QUIC_LISTENER`   | `QUIC_PORT`      | 2           | EL PL           |
+| QCSO  | `QUIC_CONNECTION` | `QUIC_CHANNEL`   | 3           | EL PL           |
+| QSSO  | `QUIC_XSO`        | `QUIC_STREAM`    | 4           |                 |
+
+### Polling
+
+We will support polling all QUIC object types (QLSOs, QCSOs, QSSOs) using a new
+polling API. This is the subject of its own design and will be discussed in more
+detail in a separate design document.
+
+### Impact of `SSL_METHOD` selection
+
+The interaction of our various QCSO and QLSO construction APIs in the APL,
+and the choice of the `SSL_METHOD` for a `SSL_CTX`, must also be considered.
+The following interactions are chosen:
+
+```text
+SSL_new(QUIC_client_method)             → QCSO(Client)
+SSL_new(QUIC_client_thread_method)      → QCSO(Client)
+SSL_new(QUIC_server_method)             → (not supported; error)
+
+SSL_new_listener(QUIC_client_method)        → QLSO(Client)   (implicit NO_ACCEPT)
+SSL_new_listener(QUIC_client_thread_method) → QLSO(Client)   (implicit NO_ACCEPT)
+SSL_new_listener(QUIC_server_method)        → QLSO(Server + Client)
+
+SSL_new_from_listener(QUIC_client_method)           → QCSO (Client)
+SSL_new_from_listener(QUIC_client_thread_method)    → QCSO (Client)
+SSL_new_from_listener(QUIC_server_method)           → QCSO (Client)
+```
+
+Observations:
+
+- It makes no sense to try and make a single QCSO proactively as a server,
+  so this just results in an error.
+
+- A server QLSO can also make outgoing client connections.
+
+- By contrast, if a client `SSL_METHOD` is specified when creating a listener,
+  it is assumed to be uninterested in accepting connections and this implies the
+  `NO_ACCEPT` flag. In this regard the functionality of a server is a superset
+  of the functionality of a client.
+
+- `SSL_new_from_listener` always creates a client QCSO.
+
+- `SINGLE_CONN` is implied automatically only when calling `SSL_new` and not
+  using a QLSO.
+
+Usage Example
+-------------
+
+```c
+{
+    SSL *l, *conn;
+    SSL_CTX *ctx = create_ssl_ctx(QUIC_server_method());
+    BIO *dgram_bio = get_dgram_bio();
+
+    l = SSL_new_listener(ctx);
+    SSL_set_bio(l, dgram_bio, dgram_bio);
+
+    for (;;) {
+        /* automatically starts and calls SSL_listen on first call */
+        conn = SSL_accept_conn(l, 0); /* standard QCSO is returned */
+
+        spawn_thread_to_process_conn(conn);
+    }
+}
+```