GH787: Fix ALPN

* Perform ALPN after the SNI callback; the SSL_CTX may change due to
  that processing
* Add flags to indicate that we actually sent ALPN, to properly error
  out if unexpectedly received.
* clean up ssl3_free() no need to explicitly clear when doing memset
* document ALPN functions

Signed-off-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Emilia Käsper <emilia@openssl.org>

1 files changed, 126 insertions, 0 deletions

diff --git a/doc/ssl/SSL_CTX_set_alpn_select_cb.pod b/doc/ssl/SSL_CTX_set_alpn_select_cb.pod
new file mode 100644
index 0000000000..974ca8618b
--- /dev/null
+++ b/doc/ssl/SSL_CTX_set_alpn_select_cb.pod
@@ -0,0 +1,126 @@
+=pod
+
+=head1 NAME
+
+SSL_CTX_set_alpn_select_cb, SSL_CTX_set_alpn_protos, SSL_set_alpn_protos,
+SSL_get0_alpn_selected, SSL_select_next_proto - handle application layer
+protocol negotiation (ALPN)
+
+=head1 SYNOPSIS
+
+ #include <openssl/ssl.h>
+
+ int SSL_CTX_set_alpn_protos(SSL_CTX *ctx, const unsigned char *protos,
+                             unsigned int protos_len);
+ int SSL_set_alpn_protos(SSL *ssl, const unsigned char *protos,
+                         unsigned int protos_len);
+ void SSL_CTX_set_alpn_select_cb(SSL_CTX *ctx,
+                                 int (*cb) (SSL *ssl,
+                                            const unsigned char **out,
+                                            unsigned char *outlen,
+                                            const unsigned char *in,
+                                            unsigned int inlen,
+                                            void *arg), void *arg);
+ int SSL_select_next_proto(unsigned char **out, unsigned char *outlen,
+                           const unsigned char *server,
+                           unsigned int server_len,
+                           const unsigned char *client,
+                           unsigned int client_len)
+ void SSL_get0_alpn_selected(const SSL *ssl, const unsigned char **data,
+                             unsigned int *len);
+
+=head1 DESCRIPTION
+
+SSL_CTX_set_alpn_protos() and SSL_set_alpn_protos() are used by the client to
+set the list of protocols available to be negotiated. The B<protos> must be in
+protocol-list format, described below. The length of B<protos> is specified in
+B<protos_len>.
+
+SSL_CTX_set_alpn_select_cb() sets the application callback B<cb> used by a
+server to select which protocol to use for the incoming connection. When B<cb>
+is NULL, no ALPN is not used. The B<arg> value is pointer which is passed to
+the application callback.
+
+B<cb> is the application defined callback. The B<in>, B<inlen> parameters are a
+vector in protocol-list format. The value of the B<out>, B<outlen> vector
+should be set to the value of a single protocol contained with in the B<in>,
+B<inlen> vector. The B<arg> parameter is the pointer set via
+SSL_CTX_set_alpn_select_cb().
+
+SSL_select_next_proto() is a helper function used to select protocols. It
+implements the standard protocol selection. It is expected that this function
+is called from the application callback B<cb>. The protocol data in B<server>,
+B<server_len> and B<client>, B<client_len> must be in protocol-list format
+described below. The first item in the B<server>, B<server_len> list that
+matches an item in the B<client>, B<client_len> list is selected, and returned
+in B<out>, B<outlen>. The B<out> value will point into either B<server> or
+B<client>, so it should be copied immediately. If no match is found, the first
+item in B<client>, B<client_len> is returned in B<out>, B<outlen>. This
+function can also be used in the NPN callback.
+
+SSL_get0_alpn_selected() returns a pointer to the selected protocol in B<data>
+with length B<len>. It is not NUL-terminated. B<data> is set to NULL and B<len>
+is set to 0 if no protocol has been selected. B<data> value must not be freed.
+
+=head1 NOTES
+
+The protocol-lists must be in wire-format, which is defined as a vector of
+non-empty, 8-bit length-prefixed, byte strings. The length-prefix byte is not
+included in the length. Each string is limited to 255 bytes. A byte-string
+length of 0 is invalid. A truncated byte-string is invalid. The length of the
+vector is not in the vector itself, but in a separate variable.
+
+Example:
+
+ unsigned char vector[] = {
+     6, 's', 'p', 'd', 'y', '/', '1',
+     8, 'h', 't', 't', 'p', '/', '1', '.', '1'
+ };
+ unsigned int length = sizeof(vector);
+
+The ALPN callback is executed after the servername callback; as that servername
+callback may update the SSL_CTX, and subsequently, the ALPN callback.
+
+If there is no ALPN proposed in the ClientHello, the ALPN callback is not
+invoked.
+
+=head1 RETURN VALUES
+
+SSL_CTX_set_alpn_protos() and SSL_set_alpn_protos() return 0 on success, and
+non-0 on failure. WARNING: these functions reverse the return value convention.
+
+SSL_select_next_proto() returns one of the following:
+
+=over 4
+
+=item OPENSSL_NPN_NEGOTIATED
+
+A match was found and is returned in B<out>, B<outlen>.
+
+=item OPENSSL_NPN_NO_OVERLAP
+
+No match was found. The first item in B<client>, B<client_len> is returned in
+B<out>, B<outlen>.
+
+=back
+
+The ALPN select callback B<cb>, must return one of the following:
+
+=over 4
+
+=item SSL_TLSEXT_ERR_OK
+
+ALPN protocol selected.
+
+=item SSL_TLSEXT_ERR_NOACK
+
+ALPN protocol not selected.
+
+=back
+
+=head1 SEE ALSO
+
+L<ssl(3)>, L<SSL_CTX_set_tlsext_servername_callback(3)>,
+L<SSL_CTX_set_tlsext_servername_arg(3)>
+
+=cut