Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Other considerations can be found in this white paper.

Quick guide

TCP fastopen [I-D.ietf-tcpm-fastopen] (TFO) allows data to be carried in the SYN packet. It also saves up to one RTT compared to standard TCP. TFO clients request a server cookie in the initial SYN packet at the
start of a new connection. The server returns a cookie in its SYN-ACK. The client caches the cookie and reuses it when opening subsequent connections to the same server. The cookie is stored by the client's TCP stack (kernel) and persists if either the client or server processes are restarted. TFO also falls back to a regular TCP handshake gracefully. The implementation in the Linux kernel also caches negative responses from servers and disables further attempts to use TFO temporarily on that specific connection. 

 

...

Message flow

Client side

Code Block
titleClient
linenumberstrue
// Send first packet with sendto instead of connect
bytes_written = sendto(sockd, pkt_data, i, MSG_FASTOPEN, (struct sockaddr *) &serv_addr, sizeof (serv_addr));
// If the fastopen fails then it appears to fallback automagicly to regular 3 way handshake

Server side

Code Block
titleServer
linenumberstrue
// between bind and listen add
setsockopt(tcp, IPPROTO_TCP, TCP_FASTOPEN, &backlog, sizeof(backlog))

Kernel parameter

To enable TFO on the server machine, the kernel parameters need to be configured to support TCP_FASTOPEN in server mode. To enable it, run the following command as root:

Code Block
titleKernel parameter to support TFO
sudo sysctl -w net.ipv4.tcp_fastopen=2

To act in pure server mode, set it to 2. If also running a TFO client on the machine, set it to 3.

More details from Documentation/networking/ip-sysctl.txt:

...

 

Requesting Fast Open Cookie in connection 1:

   TCP A (Client)                                       TCP B(Server)
   ______________                                       _____________
   CLOSED                                                      LISTEN

   #1 SYN-SENT       ----- <SYN,CookieOpt=NIL>  ---------->  SYN-RCVD
   #2 ESTABLISHED    <---- <SYN,ACK,CookieOpt=C> ----------  SYN-RCVD
   (caches cookie C)
   Performing TCP Fast Open in connection 2:

   TCP A (Client)                                       TCP B(Server)
   ______________                                       _____________
   CLOSED                                                      LISTEN

   #1 SYN-SENT       ----- <SYN=x,CookieOpt=C,DATA_A> ---->  SYN-RCVD
   #2 ESTABLISHED    <---- <SYN=y,ACK=x+len(DATA_A)+1> ----  SYN-RCVD
   #3 ESTABLISHED    <---- <ACK=x+len(DATA_A)+1,DATA_B>----  SYN-RCVD
   #4 ESTABLISHED    ----- <ACK=y+1>--------------------> ESTABLISHED
   #5 ESTABLISHED    --- <ACK=y+len(DATA_B)+1>----------> ESTABLISHED


Implementation

 TFO is currently only available on Linux.

Adding support for this to existing name server implementations is relatively easy, but does require source code modifications. It is also controlled via the kernel parameter net.ipv4.tcp_fastopen, which is set to 1 by default which enables only client mode TFO. The changes required to support TFO are:

Client side

  • On the client, the call to connect() is replaced with a call to sendmsg() or sendto() with the flags parameter set to MSG_FASTOPEN
    • For blocking sockets this performs both the handshake and sends the data before returning.
    • For non-blocking socket it returns the number of bytes buffered and sent in the SYN packet. If the cookie is not available locally, it returns -1 with errno EINPROGRESS, and sends a SYN with TFO cookie request automatically. The caller needs to write the data again when the socket is connected with a call to send() or similar.
  • Subsequent writes on this socket must also be done with send() or similar. A subsequent call with sendto() will return the error EISCONN.

Server side

  • On the server hosting the nameserver, TFO must be switched into server mode by changing the kernel parameter net.ipv4.tcp_fastopen to enable the server bit. To act in pure server mode, set the integer value to 2. To enable both client and server mode, set it to 3. In our implementation we have chosen to leave this as a manual act to be performed on the server, rather than attempting this programatically. The change can be made with the following command: 

    Code Block
    sysctl -w net.ipv4.tcp_fastopen=2

     

     

  • Also the TCP_FASTOPEN socket option must be set between the bind() and listen() calls.