DTLSv1_listen rewrite

The existing implementation of DTLSv1_listen() is fundamentally flawed. This
function is used in DTLS solutions to listen for new incoming connections
from DTLS clients. A client will send an initial ClientHello. The server
will respond with a HelloVerifyRequest containing a unique cookie. The
client the responds with a second ClientHello - which this time contains the
cookie.

Once the cookie has been verified then DTLSv1_listen() returns to user code,
which is typically expected to continue the handshake with a call to (for
example) SSL_accept().

Whilst listening for incoming ClientHellos, the underlying BIO is usually in
an unconnected state. Therefore ClientHellos can come in from *any* peer.
The arrival of the first ClientHello without the cookie, and the second one
with it, could be interspersed with other intervening messages from
different clients.

The whole purpose of this mechanism is as a defence against DoS attacks. The
idea is to avoid allocating state on the server until the client has
verified that it is capable of receiving messages at the address it claims
to come from. However the existing DTLSv1_listen() implementation completely
fails to do this. It attempts to super-impose itself on the standard state
machine and reuses all of this code. However the standard state machine
expects to operate in a stateful manner with a single client, and this can
cause various problems.

A second more minor issue is that the return codes from this function are
quite confused, with no distinction made between fatal and non-fatal errors.
Most user code treats all errors as non-fatal, and simply retries the call
to DTLSv1_listen().

This commit completely rewrites the implementation of DTLSv1_listen() and
provides a stand alone implementation that does not rely on the existing
state machine. It also provides more consistent return codes.

Reviewed-by: Andy Polyakov <appro@openssl.org>

1 files changed, 5 insertions, 0 deletions

diff --git a/include/openssl/ssl.h b/include/openssl/ssl.h
index 1334eb6161..192640e1db 100644
--- a/include/openssl/ssl.h
+++ b/include/openssl/ssl.h
@@ -1913,6 +1913,7 @@ void ERR_load_SSL_strings(void);
 # define SSL_F_DTLS1_GET_RECORD                           254
 # define SSL_F_DTLS1_HANDLE_TIMEOUT                       297
 # define SSL_F_DTLS1_HEARTBEAT                            305
+# define SSL_F_DTLS1_LISTEN                               350
 # define SSL_F_DTLS1_OUTPUT_CERT_CHAIN                    255
 # define SSL_F_DTLS1_PREPROCESS_FRAGMENT                  288
 # define SSL_F_DTLS1_PROCESS_OUT_OF_SEQ_MESSAGE           256
@@ -2166,6 +2167,7 @@ void ERR_load_SSL_strings(void);
 # define SSL_R_COMPRESSION_ID_NOT_WITHIN_PRIVATE_RANGE    307
 # define SSL_R_COMPRESSION_LIBRARY_ERROR                  142
 # define SSL_R_CONNECTION_TYPE_NOT_SET                    144
+# define SSL_R_COOKIE_GEN_CALLBACK_FAILURE                400
 # define SSL_R_COOKIE_MISMATCH                            308
 # define SSL_R_DATA_BETWEEN_CCS_AND_FINISHED              145
 # define SSL_R_DATA_LENGTH_TOO_LONG                       146
@@ -2189,6 +2191,7 @@ void ERR_load_SSL_strings(void);
 # define SSL_R_ERROR_IN_RECEIVED_CIPHER_LIST              151
 # define SSL_R_EXCESSIVE_MESSAGE_SIZE                     152
 # define SSL_R_EXTRA_DATA_IN_MESSAGE                      153
+# define SSL_R_FRAGMENTED_CLIENT_HELLO                    401
 # define SSL_R_GOT_A_FIN_BEFORE_A_CCS                     154
 # define SSL_R_GOT_NEXT_PROTO_BEFORE_A_CCS                355
 # define SSL_R_GOT_NEXT_PROTO_WITHOUT_EXTENSION           356
@@ -2201,6 +2204,7 @@ void ERR_load_SSL_strings(void);
 # define SSL_R_INVALID_COMPRESSION_ALGORITHM              341
 # define SSL_R_INVALID_NULL_CMD_NAME                      385
 # define SSL_R_INVALID_PURPOSE                            278
+# define SSL_R_INVALID_SEQUENCE_NUMBER                    402
 # define SSL_R_INVALID_SERVERINFO_DATA                    388
 # define SSL_R_INVALID_SRP_USERNAME                       357
 # define SSL_R_INVALID_STATUS_RESPONSE                    328
@@ -2250,6 +2254,7 @@ void ERR_load_SSL_strings(void);
 # define SSL_R_NO_SHARED_SIGATURE_ALGORITHMS              376
 # define SSL_R_NO_SRTP_PROFILES                           359
 # define SSL_R_NO_VERIFY_CALLBACK                         194
+# define SSL_R_NO_VERIFY_COOKIE_CALLBACK                  403
 # define SSL_R_NULL_SSL_CTX                               195
 # define SSL_R_NULL_SSL_METHOD_PASSED                     196
 # define SSL_R_OLD_SESSION_CIPHER_NOT_RETURNED            197