path: root/Documentation/networking/rxrpc.rst

.. SPDX-License-Identifier: GPL-2.0

======================
RxRPC Network Protocol
======================

The RxRPC protocol driver provides a reliable two-phase transport on top of UDP
that can be used to perform RxRPC remote operations.  This is done over sockets
of AF_RXRPC family, using sendmsg() and recvmsg() with control data to send and
receive data, aborts and errors.

Contents of this document:

 (#) Overview.

 (#) RxRPC protocol summary.

 (#) AF_RXRPC driver model.

 (#) Control messages.

 (#) Socket options.

 (#) Security.

 (#) Example client usage.

 (#) Example server usage.

 (#) AF_RXRPC kernel interface.

 (#) Configurable parameters.


Overview
========

RxRPC is a two-layer protocol.  There is a session layer which provides
reliable virtual connections using UDP over IPv4 (or IPv6) as the transport
layer, but implements a real network protocol; and there's the presentation
layer which renders structured data to binary blobs and back again using XDR
(as does SunRPC)::

		+-------------+
		| Application |
		+-------------+
		|     XDR     |		Presentation
		+-------------+
		|    RxRPC    |		Session
		+-------------+
		|     UDP     |		Transport
		+-------------+


AF_RXRPC provides:

 (1) Part of an RxRPC facility for both kernel and userspace applications by
     making the session part of it a Linux network protocol (AF_RXRPC).

 (2) A two-phase protocol.  The client transmits a blob (the request) and then
     receives a blob (the reply), and the server receives the request and then
     transmits the reply.

 (3) Retention of the reusable bits of the transport system set up for one call
     to speed up subsequent calls.

 (4) A secure protocol, using the Linux kernel's key retention facility to
     manage security on the client end.  The server end must of necessity be
     more active in security negotiations.

AF_RXRPC does not provide XDR marshalling/presentation facilities.  That is
left to the application.  AF_RXRPC only deals in blobs.  Even the operation ID
is just the first four bytes of the request blob, and as such is beyond the
kernel's interest.


Sockets of AF_RXRPC family are:

 (1) created as type SOCK_DGRAM;

 (2) provided with a protocol of the type of underlying transport they're going
     to use - currently only PF_INET is supported.


The Andrew File System (AFS) is an example of an application that uses this and
that has both kernel (filesystem) and userspace (utility) components.


RxRPC Protocol Summary
======================

An overview of the RxRPC protocol:

 (#) RxRPC sits on top of another networking protocol (UDP is the only option
     currently), and uses this to provide network transport.  UDP ports, for
     example, provide transport endpoints.

 (#) RxRPC supports multiple virtual "connections" from any given transport
     endpoint, thus allowing the endpoints to be shared, even to the same
     remote endpoint.

 (#) Each connection goes to a particular "service".  A connection may not go
     to multiple services.  A service may be considered the RxRPC equivalent of
     a port number.  AF_RXRPC permits multiple services to share an endpoint.

 (#) Client-originating packets are marked, thus a transport endpoint can be
     shared between client and server connections (connections have a
     direction).

 (#) Up to a billion connections may be supported concurrently between one
     local transport endpoint and one service on one remote endpoint.  An RxRPC
     connection is described by seven numbers::

	Local address	}
	Local port	} Transport (UDP) address
	Remote address	}
	Remote port	}
	Direction
	Connection ID
	Service ID

 (#) Each RxRPC operation is a "call".  A connection may make up to four
     billion calls, but only up to four calls may be in progress on a
     connection at any one time.

 (#) Calls are two-phase and asymmetric: the client sends its request data,
     which the service receives; then the service transmits the reply data
     which the client receives.

 (#) The data blobs are of indefinite size, the end of a phase is marked with a
     flag in the packet.  The number of packets of data making up one blob may
     not exceed 4 billion, however, as this would cause the sequence number to
     wrap.

 (#) The first four bytes of the request data are the service operation ID.

 (#) Security is negotiated on a per-connection basis.  The connection is
     initiated by the first data packet on it arriving.  If security is
     requested, the server then issues a "challenge" and then the client
     replies with a "response".  If the response is successful, the security is
     set for the lifetime of that connection, and all subsequent calls made
     upon it use that same security.  In the event that the server lets a
     connection lapse before the client, the security will be renegotiated if
     the client uses the connection again.

 (#) Calls use ACK packets to handle reliability.  Data packets are also
     explicitly sequenced per call.

 (#) There are two types of positive acknowledgment: hard-ACKs and soft-ACKs.
     A hard-ACK indicates to the far side that all the data received to a point
     has been received and processed; a soft-ACK indicates that the data has
     been received but may yet be discarded and re-requested.  The sender may
     not discard any transmittable packets until they've been hard-ACK'd.

 (#) Reception of a reply data packet implicitly hard-ACK's all the data
     packets that make up the request.

 (#) An call is complete when the request has been sent, the reply has been
     received and the final hard-ACK on the last packet of the reply has
     reached the server.

 (#) An call may be aborted by either end at any time up to its completion.


AF_RXRPC Driver Model
=====================

About the AF_RXRPC driver:

 (#) The AF_RXRPC protocol transparently uses internal sockets of the transport
     protocol to represent transport endpoints.

 (#) AF_RXRPC sockets map onto RxRPC connection bundles.  Actual RxRPC
     connections are handled transparently.  One client socket may be used to
     make multiple simultaneous calls to the same service.  One server socket
     may handle calls from many clients.

 (#) Additional parallel client connections will be initiated to support extra
     concurrent calls, up to a tunable limit.

 (#) Each connection is retained for a certain amount of time [tunable] after
     the last call currently using it has completed in case a new call is made
     that could reuse it.

 (#) Each internal UDP socket is retained [tunable] for a certain amount of
     time [tunable] after the last connection using it discarded, in case a new
     connection is made that could use it.

 (#) A client-side connection is only shared between calls if they have
     the same key struct describing their security (and assuming the calls
     would otherwise share the connection).  Non-secured calls would also be
     able to share connections with each other.

 (#) A server-side connection is shared if the client says it is.

 (#) ACK'ing is handled by the protocol driver automatically, including ping
     replying.

 (#) SO_KEEPALIVE automatically pings the other side to keep the connection
     alive [TODO].

 (#) If an ICMP error is received, all calls affected by that error will be
     aborted with an appropriate network error passed through recvmsg().


Interaction with the user of the RxRPC socket:

 (#) A socket is made into a server socket by binding an address with a
     non-zero service ID.

 (#) In the client, sending a request is achieved with one or more sendmsgs,
     followed by the reply being received with one or more recvmsgs.

 (#) The first sendmsg for a request to be sent from a client contains a tag to
     be used in all other sendmsgs or recvmsgs associated with that call.  The
     tag is carried in the control data.

 (#) connect() is used to supply a default destination address for a client
     socket.  This may be overridden by supplying an alternate address to the
     first sendmsg() of a call (struct msghdr::msg_name).

 (#) If connect() is called on an unbound client, a random local port will
     bound before the operation takes place.

 (#) A server socket may also be used to make client calls.  To do this, the
     first sendmsg() of the call must specify the target address.  The server's
     transport endpoint is used to send the packets.

 (#) Once the application has received the last message associated with a call,
     the tag is guaranteed not to be seen again, and so it can be used to pin
     client resources.  A new call can then be initiated with the same tag
     without fear of interference.

 (#) In the server, a request is received with one or more recvmsgs, then the
     the reply is transmitted with one or more sendmsgs, and then the final ACK
     is received with a last recvmsg.

 (#) When sending data for a call, sendmsg is given MSG_MORE if there's more
     data to come on that call.

 (#) When receiving data for a call, recvmsg flags MSG_MORE if there's more
     data to come for that call.

 (#) When receiving data or messages for a call, MSG_EOR is flagged by recvmsg
     to indicate the terminal message for that call.

 (#) A call may be aborted by adding an abort control message to the control
     data.  Issuing an abort terminates the kernel's use of that call's tag.
     Any messages waiting in the receive queue for that call will be discarded.

 (#) Aborts, busy notifications and challenge packets are delivered by recvmsg,
     and control data messages will be set to indicate the context.  Receiving
     an abort or a busy message terminates the kernel's use of that call's tag.

 (#) The control data part of the msghdr struct is used for a number of things:

     (#) The tag of the intended or affected call.

     (#) Sending or receiving errors, aborts and busy notifications.

     (#) Notifications of incoming calls.

     (#) Sending debug requests and receiving debug replies [TODO].

 (#) When the kernel has received and set up an incoming call, it sends a
     message to server application to let it know there's a new call awaiting
     its acceptance [recvmsg reports a special control message].  The server
     application then uses sendmsg to assign a tag to the new call.  Once that
     is done, the first part of the request data will be delivered by recvmsg.

 (#) The server application has to provide the server socket with a keyring of
     secret keys corresponding to the security types it permits.  When a secure
     connection is being set up, the kernel looks up the appropriate secret key
     in the keyring and then sends a challenge packet to the client and
     receives a response packet.  The kernel then checks the authorisation of
     the packet and either aborts the connection or sets up the security.

 (#) The name of the key a client will use to secure its communications is
     nominated by a socket option.


Notes on sendmsg:

 (#) MSG_WAITALL can be set to tell sendmsg to ignore signals if the peer is
     making progress at accepting packets within a reasonable time such that we
     manage to queue up all the data for transmission.  This requires the
     client to accept at least one packet per 2*RTT time period.

     If this isn't set, sendmsg() will return immediately, either returning
     EINTR/ERESTARTSYS if nothing was consumed or returning the amount of data
     consumed.


Notes on recvmsg:

 (#) If there's a sequence of data messages belonging to a particular call on
     the receive queue, then recvmsg will keep working through them until:

     (a) it meets the end of that call's received data,

     (b) it meets a non-data message,

     (c) it meets a message belonging to a different call, or

     (d) it fills the user buffer.

     If recvmsg is called in blocking mode, it will keep sleeping, awaiting the
     reception of further data, until one of the above four conditions is met.

 (2) MSG_PEEK operates similarly, but will return immediately if it has put any
     data in the buffer rather than sleeping until it can fill the buffer.

 (3) If a data message is only partially consumed in filling a user buffer,
     then the remainder of that message will be left on the front of the queue
     for the next taker.  MSG_TRUNC will never be flagged.

 (4) If there is more data to be had on a call (it hasn't copied the last byte
     of the last data message in that phase yet), then MSG_MORE will be
     flagged.


Control Messages
================

AF_RXRPC makes use of control messages in sendmsg() and recvmsg() to multiplex
calls, to invoke certain actions and to report certain conditions.  These are:

	=======================	=== ===========	===============================
	MESSAGE ID		SRT DATA	MEANING
	=======================	=== ===========	===============================
	RXRPC_USER_CALL_ID	sr- User ID	App's call specifier
	RXRPC_ABORT		srt Abort code	Abort code to issue/received
	RXRPC_ACK		-rt n/a		Final ACK received
	RXRPC_NET_ERROR		-rt error num	Network error on call
	RXRPC_BUSY		-rt n/a		Call rejected (server busy)
	RXRPC_LOCAL_ERROR	-rt error num	Local error encountered
	RXRPC_NEW_CALL		-r- n/a		New call received
	RXRPC_ACCEPT		s-- n/a		Accept new call
	RXRPC_EXCLUSIVE_CALL	s-- n/a		Make an exclusive client call
	RXRPC_UPGRADE_SERVICE	s-- n/a		Client call can be upgraded
	RXRPC_TX_LENGTH		s-- data len	Total length of Tx data
	=======================	=== ===========	===============================

	(SRT = usable in Sendmsg / delivered by Recvmsg / Terminal message)

 (#) RXRPC_USER_CALL_ID

     This is used to indicate the application's call ID.  It's an unsigned long
     that the app specifies in the client by attaching it to the first data
     message or in