The transport layer is one of the key layers of the Internet protocol stack. It enrichs the network layer service to make it suitable for applications. Almost 40 years after its initial design, TCP remains the most widely used transport protocol. In the early 2000s, SCTP was proposed as an alternative to TCP. Despite a clean and extensible design and many useful features, it did not reach wide deployment. This failure is mainly caused by middleboxes. We'll describe their operation and explain why Multipath TCP, which is a backward compatible evolution to TCP, has better chances of being deployed. We'll explain the main principles behind Multipath TCP and the lessons that can be drawn from its design. We'll then analyse why Internet giants like Google and Microsoft now consider application-layer solutions like QUIC to replace standard protocols like TCP.

1. Beyond TCP: The evolution of
Internet transport protocols
Olivier Bonaventure
UCL
http://inl.info.ucl.ac.be
CNSM'15, November10th, 2015

2. Agenda
• Internet transport protocols
– TCP
– SCTP
• Multipath TCP
– Basic principles
– Use cases
• What's next ?
– QUIC

3. The origins of TCP
Source : http://spectrum.ieee.org/computing/software/the-strange-birth-and-long-life-of-unix

4. The Unix pipe model
echo wc
1234 abbsbbbs

5. The TCP bytestream model
Client Server
ABCDEF...111232
0988989 ... XYZZ
IP:1.2.3.4 IP:4.5.6.7

6. TCP
More than 30 years old!

7. Congestion collapse
JACOBSON, V. Congestion avoidance and control. In Proceedings of SIGCOMM ’88 (Stanford,
CA, Aug. 1988), ACM.

8. Performance issues
• TCP considered to be too complex by many
– Software implementation cannot cope with
increasing network bandwidth
• For high performance, transport should be
implemented in hardware
– Transputers
– Simpler transport protocols

9. More limitations of TCP
• Issues with the TCP pipe model
– Only supports a single bytestream
• Some applications need several streams with priorities
– No support for messages
– Connections are attached to one IP address on
client and one IP address on server
• No failover even if hosts have multiple interfaces
• No support for mobility
• No load balancing for multihomed hosts

10. SCTP : An alternative to TCP

11. SCTP in two slides
• Modern transport protocol
– Cleaner connection establishment
• Four-way handshake to counter SYN flooding attacks
– Cleaner protocol
• Flexible TLV packet format that is easy to extend
• Selective acknowledgements from the start
– Richer semantics
• Messages, multiple streams, unreliable delivery
• Advanced API to replace socket API
– Failover support
• Connection can move from one IP address to another one

12. SCTP connection establishment
INIT,Itag=1234
INIT-ACK,cookie,ITag=5678
COOKIE-Echo,Vtag=5678, cookie
COOKIE-ACK,Vtag=1234
Encrypt state in cookie,
Does not store it
Decrypts cookie,
Recover info to create state

13. What went wrong with SCTP ?
• Replacing a transport protocol
Physical
Datalink
Network
TCP
Application
SCTP
Applications must be
rewritten with new API
IP protocol=132
For SCTP packets

14. Deploying SCTP
• Applications developers will invest in SCTP as
soon as SCTP is implemented on
– Clients
– Servers

15. The Internet architecture
that we explain to our students
Physical
Datalink
Network
Transport
Application
O. Bonaventure, Computer networking : Principles, Protocols and Practice, open ebook, http://inl.info.ucl.ac.be/cnp3
Physical
Physical
Datalink
Physical
Datalink
Network

16. In reality
– almost as many middleboxes as routers
– various types of middleboxes are deployed
Sherry, Justine, et al. "Making middleboxes someone else's problem: Network processing as a cloud service."
Proceedings of the ACM SIGCOMM 2012 conference. ACM, 2012.

17. Internet devices according to Cisco
http://www.cisco.com/web/about/ac50/ac47/2.html
Web Security
Appliance
NAC Appliance
ACE XML
Gateway
Streamer
VPN Concentrator
SSL
Terminator
Cisco IOS Firewall
IP Telephony
Router
PIX Firewall
Right and Left
Voice
GatewayVVVV
Content
Engine
NAT

18. Middleboxes in the architecture
• In the official architecture, they do not exist
• In reality...
Physical
Datalink
Network
Transport
Application
Physical
Datalink
Network
Transport
Application
Physical
Datalink
Network
TCP
Physical
Datalink
Network
Transport
Application

19. TCP segments processed by a router
Source port Destination port
Checksum Urgent pointer
THL Reserved Flags
Acknowledgment number
Sequence number
Window
Ver IHL ToS Total length
ChecksumTTL Protocol
Flags Frag. Offset
Source IP address
Identification
Destination IP address
Payload
Options
Source port Destination port
Checksum Urgent pointer
THL Reserved Flags
Acknowledgment number
Sequence number
Window
Ver IHL ToS Total length
ChecksumTTL Protocol
Flags Frag. Offset
Source IP address
Identification
Destination IP address
Payload
Options
IP
TCP

20. TCP segments processed by a NAT
Source port Destination port
Checksum Urgent pointer
THL Reserved Flags
Acknowledgment number
Sequence number
Window
Ver IHL ToS Total length
ChecksumTTL Protocol
Flags Frag. Offset
Source IP address
Identification
Destination IP address
Payload
Options
Source port Destination port
Checksum Urgent pointer
THL Reserved Flags
Acknowledgment number
Sequence number
Window
Ver IHL ToS Total length
ChecksumTTL Protocol
Flags Frag. Offset
Source IP address
Identification
Destination IP address
Payload
Options

21. © O. Bonaventure, 2011
How transparent is the Internet ?
• 25th September 2010
to 30th April 2011
• 142 access networks
• 24 countries
• Sent specific TCP
segments from client
to a server in Japan
Honda, Michio, et al. "Is it still possible to extend TCP?" Proceedings of the 2011 ACM
SIGCOMM conference on Internet measurement conference. ACM, 2011.

22. End-to-end transparency today
Source port Destination port
Checksum Urgent pointer
THL Reserved Flags
Acknowledgment number
Sequence number
Window
Ver IHL ToS Total length
ChecksumTTL Protocol
Flags Frag. Offset
Source IP address
Identification
Destination IP address
Payload
Options
Source port Destination port
Checksum Urgent pointer
THL Reserved Flags
Acknowledgment number
Sequence number
Window
Ver IHL ToS Total length
ChecksumTTL Protocol
Flags Frag. Offset
Source IP address
Identification
Destination IP address
Payload
Options
Middleboxes don't change
the Protocol field, but
some discard packets with a
Protocol field different than
TCP or UDP

23. Agenda
• Internet transport protocols
– TCP
– SCTP
• Multipath TCP
– Basic principles
– Use cases
• What's next ?
– QUIC

24. Multipath TCP
• How can we efficiently use the multiple
interfaces that are available on today's hosts?

25. Design objectives
• Multipath TCP is an evolution of TCP
• Design objectives
– Support unmodified applications
– Work over today’s networks (IPv4 and IPv6)
– Works in all networks where regular TCP works

26. The Multipath TCP bytestream model
29
Client Server
ABCDEF...111232
0988989 ... XYZZ
IP:1.2.3.4
IP:4.5.6.7
IP:2.3.4.5 IP:6.7.8.9
BCD A

27. The Multipath TCP protocol
• Control plane
– How to manage a Multipath TCP connection that
uses several paths ?
• Data plane
– How to transport data ?
• Congestion control
– How to control congestion over multiple paths ?

28. A naïve Multipath TCP
SYN+ACK+Option
ACK
seq=123, "abc"
seq=126, "def"
SYN+Option

29. A naïve Multipath TCP
In today's Internet ?
SYN+Option
SYN+ACK+Option
ACK
seq=123, "abc"
seq=126, "def"
There is no
corresponding
TCP connection

30. Design decision
– A Multipath TCP connection is composed of one or
more regular TCP subflows that are combined
• Each host maintains state that glues the TCP subflows
that compose a Multipath TCP connection together
• Each TCP subflow is sent over a single path and appears
like a regular TCP connection along this path

31. Multipath TCP and the architecture
Physical
Datalink
Network
Transport
Application Multipath TCP
TCP1
socket
TCP2 TCPn...
Application
A. Ford, C. Raiciu, M. Handley, S. Barre, and J. Iyengar, “Architectural guidelines for multipath TCP
development", RFC6182 2011.
No modification
to ease deployment
Multiple subflows
to cope with
middleboxes

32. A regular TCP connection
• What is a regular TCP connection ?
– It starts with a three-way handshake
• SYN segments may contain special options
– All data segments are sent in sequence
• There is no gap in the sequence numbers
– It is terminated by using FIN or RST

33. Multipath TCP
SYN+Option
SYN+ACK+Option
ACK
SYN+OtherOption
SYN+ACK+OtherOption
ACK

34. How to combine two TCP subflows ?
SYN+Option
SYN+ACK+Option
ACK
SYN+OtherOption
SYN+ACK+OtherOption
ACK
How to link with
blue subflow ?

35. TCP 101
Identification of a TCP connection
Four tuple
– IPsource
– IPdest
– Portsource
– Portdest
All TCP segments
contain the four
tuple
Source port Destination port
Checksum Urgent pointer
THL Reserved Flags
Acknowledgment number
Sequence number
Window
Ver IHL ToS Total length
ChecksumTTL Protocol
Flags Frag. Offset
Source IP address
Identification
Destination IP address
Payload
Options
IP
TCP

36. How to link TCP subflows ?
SYN, Portsrc=1234,Portdst=80+Option
SYN+ACK[...]
ACK
SYN, Portsrc=1235,Portdst=80
+Option[link Portsrc=1234,Portdst=80]
A NAT could change
addresses and
port numbers

37. How to link TCP subflows ?
SYN, Portsrc=1234,Portdst=80
+Option[Token=5678]
SYN+ACK+Option[Token=6543]
ACK
SYN, Portsrc=1235,Portdst=80
+Option[Token=6543]
MyToken=5678
YourToken=6543
MyToken=6543
YourToken=5678

38. TCP subflows in practice
• Multipath TCP supports subflow agility
– Client/server can add subflows at any time
– Client/server can remove subflows at any time

39. The Multipath TCP protocol
• Control plane
– How to manage a Multipath TCP connection that
uses several paths ?
• Data plane
– How to transport data ?
• Congestion control
– How to control congestion over multiple paths ?

40. How to transfer data ?
seq=123,"a"
seq=124,"b"
seq=125,"c"
seq=126,"d"
ack=124
ack=126
ack=125
ack=127

41. How to transfer data
in today's Internet ?
seq=123,"a"
seq=124,"b"
seq=125,"c"
ack=124
ack=126
ack=125
Gap in sequence numbering space
Some DPI will not allow this !

42. Multipath TCP Data transfer
• Two levels of sequence numbers
Multipath TCP
TCP1
socket
TCP2
Multipath TCP
TCP1
socket
TCP2
ABCDEF
Data sequence #
TCP1 sequence #
TCP2 sequence #

43. Multipath TCP
Data transfer
Dseq=0,seq=123,"a"
DSeq=1, seq=456,"b"
DSeq=2, seq=124,"c"
DAck=1,ack=124
DAck=3, ack=125
DAck=2,ack=457

44. Multipath TCP
How to deal with losses ?
• Data losses over one TCP subflow
– Fast retransmit and timeout as in regular TCP
Dseq=0,seq=123,"a"
DAck=1,ack=12
4Dseq=0,seq=123,"a"
DAck=1,ack=124

45. Multipath TCP
• What happens when a TCP subflow fails ?
Dseq=0,seq=123,"a"
DSeq=1, seq=456,"b"
DAck=0,ack=457
Dseq=0,seq=457,"a"
DAck=2,ack=458

46. The Multipath TCP protocol
• Control plane
– How to manage a Multipath TCP connection that uses
several paths ?
• Data plane
– How to transport data ?
• Congestion control
– How to control congestion over multiple paths ?
– Congestion windows on subflows MUST be coupled to
ensure that TCP remains fair with regular TCP

47. Multipath TCP use cases
The beast

48. Multipath TCP use cases
Low latency for Siri
• Long-lived TLS connections
WiFi
3G/LTE
Voice samples
Voice samples

49. Multipath TCP use cases
High bandwidth on smartphones
• Koreans want 800+ Mbps on smartphones
WiFi
4G/LTE
Multipath TCP Regular TCP
SOCKS

50. Faster broadband networks ?

51. Multipath TCP use cases
Hybrid Access Networks
DSL
4G/LTE
Multipath TCP Regular TCP
Hybrid Access
Gateway
TCP
TCP

52. Agenda
• Internet transport protocols
– TCP
– SCTP
• Multipath TCP
– Basic principles
– Use cases
• What's next ?
– QUIC

53. Issues with the current stack
Physical
Datalink
IPv4/IPv6
TCP
HTTP1.1
ASCII difficult to
parse, no priority
Unsecure
Wait for three way
handshake before
data transfer
Physical
Datalink
IPv4/IPv6
TCP
HTTP/2
TLS
Secure,
But adds more delay
Physical
Datalink
IPv4/IPv6
UDP
QUICFirst bytes
After 2 RTTs
First bytes
After 3-4 RTTs First bytes
After 0 RTT

54. QUIC in a nutshell
• First connection attempt
CHLO [SNI, VER]
CHLO[Token, Crypto info]
ServerName and Version
Rejected
REJ[Config, Token, Certificate]
DATA[Encrypted]
SHLO[Config, Token, Certificate]
DATA[Encrypted]

55. QUIC features
• Congestion control
– Leverages TCP's long history (CUBIC)
• Retransmissions
– Better than with regular TCP
– Each segment has a different seqnum
• Avoids retransmission ambiguities
• Selective acknowledgements
– Cleaner than in TCP

56. QUIC usage at google
QUIC handshakes fail when RTTs are greater than 2.5 seconds or
when UDP is blocked
Source : J. Iyengar, QUIC Overview, IETF93, July 2015, Prague

57. Why running QUIC over UDP ?
• Simplest transport protocol
– Supported correctly by all operating systems
– Supported correctly by all middleboxes
• Application can entirely control everything
– Same version of QUIC runs on all platforms
– QUIC can be upgraded as frequently as the application
– Application developer does not need to coordinate
with IETF or anyone

58. How to cope with middleboxes ?
• Very few middleboxes interfere with UDP
– Some middleboxes drop UDP segments
• Applications will detect and fallback to TCP
– Some middleboxes rate limit UDP
• Applications will detect and fallback to TCP
• What about middleboxes optimising QUIC/UDP
– Nightmare for google
– Everything in QUIC (payload and headers) is
encrypted

59. Internet transport layer
• Still lots of innovation for an old layer…
– TCP extensions
• Initial window, TCP Fast Open, …
– Multipath TCP is getting deployed
• RFC6824 was published in January 2013
– But Middleboxes have ossified the Internet
• Other protocols
– QUIC
• Pushed by google for web applications
– TCPINC
• Support encryption inside transport layer
– TLS 1.3
• Faster handshake and lower delays