Two Hotnets papers

Hotnets is a yearly SIGCOMM workshop that focuses on new and innovative research ideas that could be start new research directions.

During Hotnets’20, we present two papers that propose new ideas for two of the key Internet protocols: BGP and TCP.

The first paper is entitled xBGP: When You Can’t Wait for the IETF and Vendors. It was written by Thomas Wirtgen, Quentin De Coninck, Randy Bush, Laurent Vanbever and Olivier Bonaventure. Since the early days of BGP, network operators have proposed changes and improvements to the protocol and its implementation. This has resulted in many discussions within the IETF and among network vendors to identify the extensions that need to be implemented and standardized. In this paper, we propose a very different approach. Since BGP implementations will evolve, we consider each BGP implementation as a kind of micro-kernel that exposes a vendor-neutral API which can be used by network operators to extend the protocol and develop protocol extensions. We implement this new idea on two different BGP implementations. To enable network operators to develop a single extension that runs on any implementation, we embed an eBPF virtual machine inside each BGP implementation. This virtual machine executes the protocol extensions that we call plugins. We use plugins to implement four very different BGP extensions.

The second paper is entitled TCPLS: Closely Integrating TCP and TLS. It was written by Florentin Rochet, Emery Assogba and Olivier Bonaventure. During the last years, the IETF has devoted a lot of effort to standardize QUIC: a new transport protocol that combines the main functions of the TLS and the transport layer (reliability, congestion control, …) but runs above UDP so that it can be implemented inside userspace libraries. Despite the growing interest in QUIC, TCP remains the most popular Internet transport protocol. In this paper, we should that by combining TCP and TLS together, it is possible to design and implement a transport protocol that has much better properties than TCP alone, but still benefits from TCP’s advantages compared to QUIC. We implement a TCPLS prototype and use it to demonstrate the benefits that TCPLS could bring to current and future Internet applications.

These two articles are part of a broader effort that aims developing a new methodology to design and implement Internet protocols. See for additional information and pointers to our current prototypes.

Bringing Multipath capabilities in QUIC

During the last years, the use cases for Multipath TCP have continued to grow. Multipath TCP is used on all iPhones to provide seamless handovers and improve performance for Siri, Apple Music and other applications. This successful deployment has encouraged 3GPP to adopt it for the ATSSS service that will future 5G smartphones to seamlessly use Wi-Fi and cellular networks. Multipath TCP is also used by network operators to deploy hybrid access networks that combine cellular and xDSL networks.

In parallel with this deployment, the IETF has finalised the specification of the QUIC protocol. In a nutshell, QUIC combines the functions that are typically found in the transport and TLS layer in a single protocol that runs above UDP so that it can be implemented as a userspace library. When the QUIC working group was chartered, we argued for the inclusion of multipath capabilities in this new protocol. This item was added to the charter and we proposed a first design for Multipath QUIC.

The QUIC working group spent more time than expected on designing the protocol and we adapted the design of Multipath QUIC to the different changes to the protocol. The latest version of Multipath QUIC is cleaner and better adapted to QUIC. The multipath requirements remain and QUIC version only provides a poor man’s solution with its connection migration capabilities that have not yet been evaluated in the field.

On October 22nd, 2020, the chairs of the QUIC working group organised an interim meeting to better understand the multipath requirements and how they could be included in QUIC. Key presentations during this interim meeting include :

The recordings are also available on

SIGCOMM’20 tutorial on Multipath Transport Protocols

During ACM SIGCOMM’20, I gave with Quentin De Coninck a three hours tutorial on Multipath Transport Protocols. During this tutorial, we summarised the most important features of two multipath transport protocols that we co-designed: Multipath TCP and Multipath QUIC. Here are the materials that we prepared for this tutorial:


The vagrant box contains both a Multipath TCP kernel and a recent version of pquic including the multipath plugin. You can use both to experiment with Multipath TCP and Multipath QUIC.

Multipath TCP proxies

Network operators want to leverage the unique capabilities of Multipath TCP for two main types of use cases:
  • aggregate the bandwidth of heterogenous paths such as xDSL and cellular to support Hybrid Access Networks
  • seamless handovers and cellular/Wi-Fi offload for smartphones and mobile devices

The Broadband Forum and 3GPP have standardised solutions architectures that rarely on Multipath TCP proxies to support those service since servers rarely support Multipath TCP. We had identified this problem several years ago and proposed efficient Multipath TCP proxies in Multipath in the Middle(Box) in 2013.

To enable network operators to deploy such proxies, it was important to document them in IETF RFCs. Despite the simplicity of the idea, it took several years of effort to convince the IETF of documenting this approach in the 0-RTT TCP Convert Protocol that is now published as RFC 8803.

Open-source networking ebook

For more than a decade and with the help of several Ph.D. and Master students, we have developed a series of open educational resources to help training the next generation of networking students. A recent summary of our effort appears in the paper Open Educational Resources for Computer Networking published in the July 2020 issue of SIGCOMM’s CCR. The CCR Editor summarises our contribution as What distinguishes this textbook from traditional ones is that it not only is it free and available for anyone in the world to use, but also, it is also interactive. Therefore, this goes way beyond what a textbook usually offers: it is an interactive learning platform for computer networking.

Innovating with Multipath TCP

There is a huge difference between research and innovation. Research often means finding new and original results. In the networking community, this often means inventing new protocols, finding new alogorithms, collecting measurements, … Innovation is different. According to the business dictionnary, innovation is “The process of translating an idea or invention into a good or service that creates value or for which customers will pay.”

Multipath TCP is an interesting example of a research result that drives innovation. In 2013, Apple started to use it to improve the performance of the SiRi voice recognition application. Two years later, several researchers from the IP Networking Lab created the Tessares to develop hybrid access networks that rely on Multipath TCP to efficiently combine xDSL and cellular networks. The main target of these hybrid networks are rural areas where existing xDSL networks do not provide enough bandwidth to support the current needs of rural users. The xDSL network is then complemented with the LTE network to boost the bandwidth when required.

This technology has been standardised by the BBF as TR-348 Hybrid Access Broadband Architecture. It is now deployed in multiple countries where rural users benefit from improved Internet access.

A recent paper published in IEEE Communications Standards Magazine and entitled Increasing Broadband Reach with Hybrid Access Networks provides additional technical details about this Multipath TCP use case.

Contributing to Multipath TCP in six Ph.D. theses

Multipath TCP is a major extension TCP extension that has attracted a lot of interest from both academia, with more than one thousand citations for RFC 6824 and industry with deployments by Apple, Samsung, Huawei, LG, Tessares, … It received the 2019 ACM SIGCOMM’s Networking Systems award. The initial ideas on Multipath TCP emerged within the FP7 Trilogy funded by the European Commission and a large part of the work was carried out within UCLouvain’s IP Networking Lab During the last decade, six Ph.D. theses were granted with results that contributed to the different aspects of Multipath TCP.

The history of Multipath TCP

During the last decade, we heavily contributed to the design, implementation and the deployment of Multipath TCP. I was interviewed by Russ White as part of the History of Networking podcast series to summarize the history of Multipath TCP. You can download this podcast from

Multipath TCP : the first ten years

The research on Multipath TCP started a bit more than ten years ago with the launch of the FP7 Trilogy project. During this decade, Multipath TCP has evolved a lot. It has also generated a lot of interest within the scientific community with several hundreds of articles that use, extend or reference Multipath TCP. As an illustration of the scientific impact of Multipath TCP, the figure below shows the cumulative number of citations for the sequence of internet drafts that became RFC 6824 according to Google Scholar.


The industrial impact of Multipath TCP is also very important as Apple uses it on all iPhones and several network operators use it to create Hybrid Access Networks that combine xDSL and LTE to provide faster Internet services in rural areas.

On all the remaining days until Christmas, a new post will appear on the Multipath TCP blog to illustrate one particular aspect of Multipath TCP with pointers to relevant scientific papers, commercial deployments, … This series of blog posts will constitute a simple advent calendar that could be useful for network engineers and researchers who want to understand how this new protocol works and why it is becoming more and more important in today’s Internet.


How can universities contribute to future Internet protocols ? Our experience with Segment Routing

The Internet continues to evolve. Given its commercial importance, a large fraction of this evolution is driven by large telecommunications and cloud companies with input from various stakeholders such as network operators. In this growingly commercial Internet, some of my colleagues wondered the role that University researchers could play ? Different researchers have different strategies. Within the IP Networking Lab, we focus our research on protocols and techniques which can improve the Internet in the medium to long term. As most of the researchers of the group are Ph.D. students, it is important for them to address research problems that will remain relevant at the end of their thesis, typically after four years. The selection of a research topic is a strategic decision for any academic lab. While many labs focus on a single topic for decades and explore every of its aspects, I tend to switch the focus of the group every 5-7 years. During my Ph.D., I explored the interactions between TCP and ATM, but when I became professor, I did not consider that the topic for still relevant enough to encourage new Ph.D. students to continue to work on it. Looking at the evolution of the field, I decided to focus our work on traffic engineering techniques and the Border Gateway Protocol. This lead to very successful projects and Ph.D. theses. In 2008, the Trilogy project gave us the opportunity to work on new future Internet protocols. With a group a very talented Ph.D. students, we played a key role in the design, development and deployment of Multipath TCP. Its successes have exceeded our initial expectations.

However, despite the benefits of Multipath TCP, betting the future of the entire research group on a single protocol did not seem to be the right approach. In early 2013, I was impressed by a presentation that Clarence Filfils gave at NANOG. This presentation completely changed my view on MPLS. MPLS emerged in the 1990s from the early work on IP switching and tag switching. One of the early motivations for MPLS was the ability to reuse the ATM and frame relay switching fabrics that were, at that time, more powerful than their IP counterparts. When the MPLS shim header appeared, the IETF required MPLS to be agnostic of the underlying routing protocol and for this reason designed the LDP and RSVP-TE signalling protocols. Over the years, as MPLS networks grew, these two protocols became operational concerns.

Segment Routing was initially presented as a drastic simplification of the networking architecture. Instead of requiring the utilisation of specific signalling protocols, it relies on the existing link state routing protocols such as OSPF and IS-IS to distribute the MPLS labels. I saw that as a major breakthrough for future MPLS networks. Beyond the expected impact on networking protocols, Segment Routing brought a fundamental change to the way paths are computed in a network. A unique feature of Segment Routing compared to all the other networking technologies is that with Segment Routing a path between a source and a destination node is composed as a succession of shortest paths between intermediate nodes. With the MPLS variant of Segment Routing, these paths are identified by their MPLS label that is placed inside each packet. With the IPv6 variant of Segment Routing, these paths are encoded as a source route inside the IPv6 Segment Routing Header. This contrasts with popular networking architectures such as plain IP that uses a single shortest path between the source and the destination while MPLS with RSVP-TE can be configured to use any path. These different types of paths have lead to very different traffic engineering techniques. In pure IP networks, a popular technique is to tune the weights of the link-state routing protocol. With Segment Routing, the traffic engineering problem can be solved by using very different techniques. During the last years, we have proposed several innovative solutions to optimise the traffic flows in large networks.

This is illustrated in the figure below. The numbers associated to the links are the IGP weights. With pure IP routing, the path from node a to node f is the shortest one, i.e. the one via node a. With RSVP-TE, any path can be constructed between node a and node f, e.g. a-g-b-c-e-f, but this requires state on all intermediate nodes. With Segment Routing, we trade the state in the routers with labels in the packets. A path is now a succession of shortest paths. For example, the figure below shows the a-c-f paths. To send packets along those paths, node a sends packets that contain two labels: (1) the label to reach node c and (2) the label to reach node f. The packets are first forwarded according to node c’s label and there are two shortest paths of equal cost between a and c. When they reach node c, it pops the top label and then packets are forwarded along the shortest path to reach node f.


A few research labs, including the IP Networking Lab have actively participated to the development of Segment Routing. Our research started almost at the same time as the initial work within the Spring IETF working group. Despite the visibility of this working group, we decided to not actively participate to the standardisation of the MPLS variant of Segment Routing. Instead, we focused our work on two different but very important aspects of Segment Routing. The first one is the design of innovative optimisation techniques that can be applied by network operators to leverage the unique characteristics of Segment Routing. The second one is the IPv6 variant of Segment Routing. Both problems were important and they were not the initial focus of IETF working group. This gave us enough time to carry research whose results could have an impact on the development of Segment Routing.

Let us start with the optimisation techniques. This work was carried out in collaboration with two colleagues: Yves Deville and Pierre Schaus. Our first approach to solve this problem was presented at SIGCOMM’15 in A Declarative and Expressive Approach to Control Forwarding Paths in Carrier-Grade Networks. This was the first important traffic engineering paper that leverages the unique features of Segment Routing. Renaud Hartert, the Ph.D. student who initiated this traffic engineering work, presented at INFOCOM’17 a faster solution in Expect the unexpected: Sub-second optimisation for segment routing. His Ph.D. thesis, Fast and scalable optimisation for segment routing, contains other unpublished techniques.

Another Ph.D. student, Francois Aubry explored other use cases than the classical traffic engineering problem. In SCMon: Leveraging Segment Routing to Improve Network Monitoring, presented at INFOCOM’16, he proposed a new technique to create efficient cycles that a monitoring node can used to verify the performance of a live network. His most recent paper that will be presented at Conext’18, Robustly Disjoint Paths with Segment Routing demonstrates that it is possible with Segment Routing to create disjoint paths that remain disjoints even after a link failure. He is currently preparing his Ph.D. thesis.

David Lebrun explored the networking aspects of Segment Routing during his Ph.D. He started his Ph.D. at the same time as the initial thinkings about the IPv6 variant of Segment Routing and has proposed several important contributions. He was the first to implement IPv6 Segment Routing in the Linux kernel. His implementation has heavily influenced several of the design choices that have shaped the specification of the IPv6 Segment Routing Header. His implementation has been described in Implementing IPv6 Segment Routing in the Linux Kernel and in his Ph.D. thesis. It has been included in the mainline Linux kernel since version 4.14. This implies that any Linux host can now use IPv6 Segment Routing. Besides this kernel implementation, David Lebrun has demonstrated in a paper that was presented at SOSR’18 how enterprise networks could leverage IPv6 Segment Routing.

Our most recent work has contributed to the utilisation of IPv6 Segment Routing to support Network Function Virtualisation or Service Function Chaining. The IPv6 variant of Segment Routing enables a very nice feature that is called network programming. With network programming, a router can expose network functions as IPv6 addresses and the packets that are sent towards those addresses are processed by a specific function on the router before being forwarded. This idea looks nice on paper and reminds older researchers of active networks that were a popular research topic around 2000. Within his Master thesis, Mathieu Xhonneux proposed to use eBPF to implement such network functions on Linux. His architecture is described in more details in Leveraging eBPF for programmable network functions with IPv6 Segment Routing with several use cases. It has also been accepted in the mainline Linux kernel. Another use case is described in Flexible failure detection and fast reroute using eBPF and SRv6.

Looking at out last five years of research on Segment Routing, I think that there are two important lessons that would be valid for other research groups willing to have impact on Internet protocols. First, it is important to have a critical mass of 3-4 Ph.D. students who can collaborate together and develop different aspects in their own thesis. The second lesson is the importance of releasing the artefacts associated to our research results. These artefacts encourage other researcher to expand our work. Our implementations that are now included in the official Linux kernel go beyond the simple reproducibility of our research results since anyone will be able to use our code.

This research on Segment Routing has been funded by the ARC-SDN project, FRIA Ph.D. fellowships and a URP grant from Cisco. It continues with a facebook grant.