With the growing adoption of disaggregated storage, NVMe over Fabrics over TCP has emerged as a practical remote storage solution for enterprise and cloud environments, avoiding the managed lossless networks typically required by RDMA-based deployments. However, neither NVMe/TCP nor NVMe/RDMA provides an efficient built-in security mechanism for protecting storage traffic against eavesdropping in modern data centers and WAN deployments. While NVMe/TCP can be secured using TLS, layering TLS over TCP increases CPU overhead and is subject to TCP head-of-line blocking under lossy or high-latency conditions.

This paper presents NVMe-over-QUIC, a redesign of NVMe-oF around a secure, message-oriented, multiplexed transport. Our key insight is to map each NVMe I/O request onto an independent QUIC stream, eliminating redundant framing and enabling true per-command isolation without cross-command head-of-line blocking. By integrating transport and encryption within QUIC, our design also avoids the inefficiencies of TLS over TCP. To preserve scalability on multi-core systems, we design a UDP-based core-affinity architecture aligned with high-performance storage processing models. Evaluation results show up to 4× higher throughput and 10× lower latency under lossy network conditions.

Paper ; Code

The post NVMe-over-QUIC at ICNP’26 appeared first on Tom Barbette.

Most of my work happens on remote Linux servers, through VS Code Remote-SSH, and more and more with Claude Code running inside that remote session. My MacBook sits on a dock with an Ethernet cable most of the day. Every time I unplug it to walk to a meeting, everything freezes: the terminal, the VS Code window, and Claude Code in the middle of a task. Nothing comes back until the dead TCP connection times out and VS Code reconnects. And the Claude session did not advance while I was away.

The reason is TCP itself. A TCP connection is bound to one source address, and when the Ethernet address disappears, the connection has nowhere to go. Multipath TCP (RFC 8684, introduce by my beloved colleague Olivier Bonaventure) fixes exactly this: one connection runs over several subflows, one per interface, and traffic moves from the Ethernet subflow to the WiFi one without the application noticing. Linux supports it since kernel 5.6, and macOS has an MPTCP client in its kernel. But it’s nearly hidden !

So the question was simple: how do I make ssh on macOS open MPTCP connections? The answer took longer than expected. It ends with ssh-mptcp, a 300-line ProxyCommand that works with any ssh client, VS Code included. And yes, Claude did the 300 lines by himself. Thanks Claude.

Demo

TL;DR

On the Mac, with the Xcode command line tools installed:

git clone https://github.com/UCLouvain-ENSG/ssh-mptcp.git
cd ssh-mptcp
make && make install   # installs ~/.local/bin/ssh-mptcp
make check             # prints "You are using MPTCP." if your Mac can do MPTCP

Then, in ~/.ssh/config, for the servers you care about:

Host *
    ProxyCommand ~/.local/bin/ssh-mptcp %h %p

That is all on the client side. VS Code Remote-SSH reads the same file, so it needs no extra setting. The server must speak MPTCP too: it’s as easy as installing mptcpize and running “mptpize enable sshd”.

The search

OpenSSH does not want it

The obvious first stop is a recent OpenSSH with an MPTCP option. There is none. Native support was proposed twice, in PR #335 and PR #547, and the maintainers declined both: they do not want a protocol that is only available on some of the platforms OpenSSH supports. Linux is reaching nearly 100% of top computer, and nearly all servers. iOS and MacOS supports it. In Android it’s just not activated. On Desktop, only Windows is left but we also have a solution. So yes, I’m a bit pissed off

The Linux tricks do not work on macOS

macOS has no IPPROTO_MPTCP. The kernel does have an AF_MULTIPATH socket family, used through connectx(2), but it is undocumented and, according to mptcp.dev, it creates MPTCP connections without the extra subflows that make them useful. The supported way in is Network.framework: nw_parameters_set_multipath_service(), available since macOS 10.14, as the mptcp.dev macOS guide explains. So whatever opens the connection has to go through Network.framework, not through a socket.

Apple already patched ssh, sort of

This is where it gets interesting. Apple ships its own fork of OpenSSH, and its sshconnect.c already connects through Network.framework. It also has a hidden, undocumented option, AppleMultipath, present since Apple’s OpenSSH-281.40.2 release (July 2022). When it is set, ssh asks for nw_multipath_service_handover through a private call, nw_parameters_create_legacy_tcp_socket(), and extracts the kernel socket from the connection, so there is no relay process.

On paper, AppleMultipath yes in ~/.ssh/config is all you need. In practice, it is fragile:

• Homebrew’s ssh refuses the option.
• Apple’s code skips the Network.framework path, and multipath with it, whenever ConnectTimeout, ConnectionAttempts, ProxyJump, ProxyCommand or hostname canonicalization is set, or when the destination is an IP address.

In my tests, with Apple’s ssh and the option accepted, tcpdump still showed a SYN with no MPTCP option at all, to a server that does speak MPTCP. I did not find out why: the private call is closed source.

The result: ssh-mptcp

Instead of patching ssh, I moved the connection out of it. ssh already knows how to delegate its transport: ProxyCommand starts a program and speaks SSH over its stdin and stdout. ssh-mptcp is that program, about 300 lines of C on the public Network.framework API:

1. ssh starts ssh-mptcp host port.
2. ssh-mptcp creates an nw_connection to the server with plain TCP parameters, TCP_NODELAY, and nw_parameters_set_multipath_service(). Network.framework resolves the name, races IPv4 and IPv6, and asks the kernel for an MPTCP connection.
3. Once connected, two loops relay the bytes: stdin to nw_connection_send(), and nw_connection_receive() to stdout.
4. When the Ethernet cable goes, the kernel moves the connection to the WiFi subflow. ssh, and everything running over it, never sees it.

https://github.com/UCLouvain-ENSG/ssh-mptcp

ssh only talks to a pipe. The MPTCP connection and its subflows live in the kernel, so losing the Ethernet subflow changes nothing above it.

Network.framework never hands out the underlying socket, so ssh-mptcp has to stay in the middle and copy the bytes, rather than pass a file descriptor back to ssh with ProxyUseFdpass. At SSH speeds, the extra process and the extra copy do not matter.

The -m option picks one of the three modes Apple defines:

• handover only uses another interface once the primary one is lost.
• interactive, the default, also switches when the primary interface suffers loss or delay. It picks the lowest-latency path, which is what an interactive SSH session wants.
• aggregate spreads the traffic over all interfaces. macOS refuses it to third-party programs unless you run sudo sysctl net.inet.mptcp.allow_aggregate=1.

With -v, ssh-mptcp logs the interfaces in use and every path change to stderr, which is the easiest way to watch a handover happen. Because it is a plain ProxyCommand, it works with Apple’s ssh, Homebrew’s ssh, git, rsync, and VS Code Remote-SSH alike.

Server side

macOS only does MPTCP as a client, so the other end must be a Linux server (kernel 5.6 or later, net.mptcp.enabled = 1) whose sshd listens on an MPTCP socket. If it does not, the connection falls back to plain TCP: ssh still works, but the session will not survive the switch.

That’s actually easy, just install mptcpize and run “sudo mptcpize enable sshd” and the sshd service will be prefixed by a preloader that will change TCP by MPTCP. Don’t worry, MPTCP always fallbacks to TCP.

SSH gateway

Do you need to do this on all your server? No ! SSH ProxyCommands can be changed. Probably if your work has a bit of sense about security, you already have to hop through an SSH gateway.

That will enable to hop throu a SSH gateway (a machine just justed to hop through SSH) :

Host *_via_proxy
    User XXX
    ProxyCommand ssh -o StrictHostKeyChecking=no -A -X -Y -q [gateway address] -W $(basename %h _via_proxy):%p

That will use ssh-mptcp as a hop for all destination :

Host *
    ControlMaster auto
    ControlPath /tmp/.ssh.%h
    ServerAliveInterval 1
    ServerAliveCountMax 300
    ProxyCommand ~/workspace/openssh-mptcp/ssh-mptcp -m interactive %h %p

The controlsocket is unrelated but allows to reuse a single connection to multiplex all client connections. More efficient and the server does not thing it’s DDOSed when you have many many connection.

With that, using ssh google.com_via_proxy will connect to google.com, via MPTCP, via your proxy.

Limitations

ssh-mptcp only covers what goes through ssh. Anything else on the Mac, including the Claude app’s own connections, still uses plain TCP, and MPTCP would also need the remote end to support it. The server must run MPTCP, and a middlebox on the path can strip the options, in which case the connection quietly falls back to TCP. macOS cannot accept MPTCP connections, so this does not help with a Mac acting as a server.

Conclusion

MPTCP on macOS has been one API call away for years, but nothing in the ssh toolchain makes that call reliably, and the one place Apple did it is hidden and fragile. A small ProxyCommand on the public API is enough: my remote sessions, VS Code, and the Claude Code agent running inside them now survive leaving the dock. The code, README and build instructions are on GitHub, under the MIT license. If you are on Windows, my older post on MPTCP with WSL2 covers the same use case there.

Sources

• ssh-mptcp on GitHub
• RFC 8684, TCP Extensions for Multipath Operation with Multiple Addresses
• mptcp.dev FAQ: MPTCP with OpenSSH, supported operating systems
• mptcp.dev: MPTCP on macOS with Network.framework
• openssh-portable PR #335 and PR #547
• Apple’s OpenSSH, sshconnect.c
• XNU sources (bsd/netinet/mptcp_usrreq.c, tcp_cache.c)
• Apple documentation: nw_multipath_service_t
• mptcp-check, the server behind check.mptcp.dev

The post Tired of SSH and Claude getting frozen when you switch to WiFi? Use MPTCP on macOS! appeared first on Tom Barbette.

Among many findings in the paper, we show the importance of the uncore, the high potential for GPU only systems and ARM complete systems like standalong DPUs that are nearly on par with x86 counter-part.

The post xPUBench at PAM’26 appeared first on Tom Barbette.

Today Maxime Vanliefde presented xPUBench: Scalable and Energy-Efficient GPU and DPU-Accelerated Network Functions at #PAM2026

This is a collaboration with Nikita tyunyayev Clément Delzotti, Romain Van Hauwaert and Elena Agostini (NVIDIA). We benchmark network packet processing on GPU, DPU, and CPU and a combination of them, hence the name, xPU (Bench).

We also take a novel angle on energy efficiency, with techniques to use less power for GPU (reducing uncore frequency), and show that DPU are highly efficient… but limited in term of processing power. When they match, they’re best. For complex workloads, GPU are the way to go and are even more energy efficient than CPUs.

Many more findings in the paper!

paper

The post xPUBench: Scalable and Energy-Efficient GPU and DPU-Accelerated Network Functions appeared first on Tom Barbette.

This week I am going to present PAMO, a modified version of the industry-grade Suricata IDS to support offloading pattern matching to the RegEx engine of the BlueField 2.

We first review and analyses the internals of IDS, focusing on Suricata with the help of one of its maintainers, Lukas Sisimis who did a 6 month exchange with us at UCLouvain (and continued to work on it as the job was much bigger than initially envisioned).

We then benchmarked what the RegEx engine was capable of.

The answer is : 51Gbps with big packets and not too many rules (we employed the widely used Emerging Threats ruleset). Still, the RXP engine used from the ARM cores of the BlueField provides a huge help and alleviate 6 or 7 x86 cores.

But the reality is that an IDS is far from being just about rules matching. And industry-grade IDS have complex processing to decide which rules should be evaluated.

After many challenges we evaluate PAMO using a real trace and a window-based mechanism to accelerate it using parallel traces, leaving temporal features untouched.

While we reduce the payload prefilter CPU processing time to peanuts, the RXP has a cost that bring the improvement to 6X. As we can’t beat Amdahl’s law, we get a 40% performance increase (this is with one core).

Perhaps an interesting result is how PAMO improves the performance of Suricata on the BlueField 2 itself. In that mode the IDS runs entierly on the NIC. As the ARM cores are weaker, the improvement reaches 70%.

Come say hello at Middleware’25 in Nashville, or check out our paper !

The post PAMO at Middleware’25 appeared first on Tom Barbette.

Université catholique de Louvain

Institute of Information and Communication Technologies, Electronics and Applied Mathematics (ICTEAM)

RESEARCH FIELD

Networking, Systems, High-speed packet processing, SmartNICs, P4, NFV

MINIMUM REQUIRED QUALIFICATIONS

The candidate must have obtained a PhD in computer science before the start date.

APPLICATION DEADLINE

Open (continuous screening process), starting ASAP

LOCATION

Belgium › Louvain-la-Neuve

TYPE OF CONTRACT

Post-Doc fellowship for 1+1 year (yearly contract)

2 years starting 1/1/2026

JOB STATUS

Full-time

HOURS PER WEEK

38

OFFER DESCRIPTION

The applicant will join the ENSG to work on leading-edge research topics in crossroad of networking and systems to build a sustainable IT infrastructure. The group is composed of the PI and 5 PhD students, and 2 post-docs.

The post-doc will join the PI’s group to conduct research on thematics that fit the PI’s area of expertise. The applicant will also be expected to advise master students and take a leading role with PhD students. The exact project is open to discussion, but the following project is proposed and describes well the overall direction of the lab.

Internet was designed at a time when computers were monolithic devices, transferring data over a network of routers and switches. This paradigm does not match the reality of today’s devices, which are composed of several elements of different nature (CPU cores, RAM, storage, NICs, GPU, etc).  You will join the study of the fundamental challenges for the Internet to catch up with this shift of paradigm. Much like atoms were later refined into a set of particles, the communications of hosts must be reconsidered to enable the next leap in the Internet evolution.

We will examine the newfound programmability of the network, i.e., P4 switches and Smart NICs, to enable sub-atomic communications over the Internet by delegating the intelligence out of the end “hosts”.

The Smart NIC of a host may essentially act as a transparent multiplexer for the sub-devices of the host, bypassing unneeded CPU transfers, and saving time and energy. The Smart NIC will be aware of power-conservative strategies when assigning requests to cores. The group is also conducting research on efficient CPU-aware software pipelines using dynamic compilation to avoid cache and branch misses.

Programmable switches will similarly act as coordinators of the streams through the edge. They will also lead the transfers toward the right particles among the increasingly disaggregated datacenter’s resources that are serving a provider’s content. To overcome ossification, they may expose information and negotiate a behavior for each particle’s streams to reach one possible servicing entity through the best paths. It will be possible without going back to the “ends”, and therefore enable particle-to-particle encryption as well as network efficiency.

The resulting low latency communication will enable future use cases such as cloud gaming and latency-critical workloads, connecting nearby particles that are getting virtually closer thanks to 5G and fiber connectivity. In the long-term, the vision enforced by the group will bring back competitiveness to the Internet by standardizing the means for such next-generation sub-atomic communication.

SKILLS/QUALIFICATION

• Published in known conference in the field (CoNEXT, NSDI, SIGCOMM, IMC, PAM, …) or journals (ToN, SIGCOMM CCR, TNSM, …)
• Good comprehension of computer systems, operating systems. Knowledge of techniques like eBPF, kernel-bypass, Xilinx FPGA ecosystem, DPDK, … is a plus
• Ease with low-level programming in C and/or Rust

SUBMISSION

Please send to tom.barbette@uclouvain.be:

(a) Curriculum vitae;
(b) A letter of motivation;
(c) Links to Masters and PhD thesis (If already defended);
(d) List of publications and links to PDF (not behind a paywall);
(e) If applicable, links to examples of personal software contributions.

REQUIRED LANGUAGES

ENGLISH: at least B2 (upper intermediate)

French is not required

Other positions

See the range of possibilities at the Efficiency of Networked Systems Group

The post 2-years post-doc position – CLOSED appeared first on Tom Barbette.

El Mehdi Makhroute, Quentin De Coninck, Tom Barbette

The Internet has shifted from an end-to-end paradigm to an end-to-ends paradigm, i.e. to serve a provider’s content (e.g. a web page, an app feed, …) the client must connect to many ends to fetch various types of resources like web documents, pictures, scripts, news feed, videos, etc… However, even the recent QUIC transport protocol failed to catch up with this change of paradigm, and the user is forced to waste many round-trip times to fetch the entire content from multiple servers. In this paper, we propose Multi-End QUIC, an extension of Multipath QUIC that enables the establishment of sub-streams directly to backend servers or third-party servers. Multi-End QUIC alleviates the need for proxies in the edge that re-encode and delay those sub-streams, that is able to bypass the relays entirely. This leads to a ~50% latency improvement in a preliminary experiment.

acm

The post Multi-End QUIC: A Transport Protocol to Enable One-to-Many Communications appeared first on Tom Barbette.

In OpenDesc, we propose to use P4 as an interface to define packet descriptors. OpenDesc enables to expose NIC capabilities and match them with application intents. We propose a prototype compiler to generate accessors that can directly extract metadata from a negotiated NIC descriptor without the need for intermediate data structures like sk_buff, xdp_sock, rte_mbuf & cie.

Read more below!

The post OpenDesc at HotNets’25 ! appeared first on Tom Barbette.

In short: Flexicast QUIC brings multicast back to the Internet by blending it with unicast, all within QUIC. It offers scalability where multicast works, and robustness where it doesn’t — making it a practical transport for the next generation of large-scale applications.

acm ; code

In long:

When distributing live video, software updates, or cloud gaming streams, today’s Internet almost exclusively relies on unicast: each receiver gets its own copy of the data. This is simple and robust but highly inefficient, especially when thousands of receivers consume the same content. The cost is felt both at the sender — which must generate and encrypt per-receiver packets — and in the network, which carries redundant traffic.

Multicast was designed decades ago to solve exactly this problem: a source transmits once, and routers replicate packets along a multicast tree. But despite its promise, multicast never became mainstream on the global Internet. It is difficult to deploy across ISPs, hard to monetize, and fragile — applications always need to fall back to unicast anyway. Most content providers gave up and built massive unicast infrastructures instead.

With the wide deployment of QUIC, a modern transport protocol running above UDP, there is a chance to revisit multicast — not at the IP layer, but directly at the transport layer. This is where Flexicast QUIC comes in.

The Idea of Flexicast QUIC

Flexicast QUIC, presented in our SIGCOMM CCR 2025 paper, extends Multipath QUIC to combine the efficiency of multicast with the reliability of unicast. The idea is to make multicast flexible:

• Each receiver gets a unicast QUIC path for control and fallback.
• The sender also establishes a flexicast flow: a shared, unidirectional path encrypted with a common key and intended for all receivers.
• If multicast routing is available, this flow is carried efficiently through the network. If not, the sender can still replicate the packets itself and deliver them over unicast.

Receivers can join or leave the flexicast flow at any time. If multicast fails for one receiver, it automatically falls back to unicast — without affecting others. All this happens within the same QUIC connection, so applications don’t need to juggle two protocols.

Key Design Points

Flexicast QUIC builds on QUIC’s extensible design:

• Per-path keys: unlike regular Multipath QUIC, each unicast path and the flexicast flow use distinct encryption keys.
• Reliability: acknowledgments are sent over unicast paths, aggregated at the sender to avoid the classic ack implosion problem.
• Congestion control: the sender maintains per-receiver congestion states. If one receiver drags the group down, it can be removed from the flexicast flow and served over unicast.

Implementation and Results

We implemented Flexicast QUIC in Cloudflare’s quiche library (Rust), adding ~10,000 lines of code and 5,000 lines of tests. The evaluation was run on CloudLab and emulated networks.

Scalability

• Unicast QUIC: saturates at ~200 receivers (~20 Gbps). CPU is the bottleneck, as every packet must be encrypted per receiver.
• Flexicast QUIC: supports 1000 receivers and delivers >80 Gbps, over 4× higher throughput than unicast QUIC, with acceptable CPU usage.
• With a small acknowledgment delay (5 ms), Flexicast QUIC perfectly matches the ideal UDP baseline.

Even without multicast in the network, Flexicast QUIC still helps: the sender can replicate encrypted packets using sendmmsg, which is far cheaper than generating per-receiver packets.

Robustness

We tested Flexicast QUIC with a 5 Mbps live video stream under failing multicast trees. Some receivers randomly lost multicast connectivity, forcing fallback to unicast. Results:

• Other receivers were unaffected.
• Video quality stayed excellent (SSIM > 0.99 for 99.4% of frames).
• Latency remained low, with only small tail increases during recovery.

This shows Flexicast QUIC provides seamless continuity even when multicast is unreliable.

Why It Matters

Flexicast QUIC makes multicast practical again:

• Efficient: one packet can serve thousands of receivers.
• Robust: unicast fallback is built-in, so failures don’t break the stream.
• Practical: works today, even without multicast routers.
• Deployable: it’s just QUIC — already used by major Internet services.

This makes it promising for content delivery networks, software updates, and live streaming at scale.

What’s Next?

Our future work will explore:

• Forward Erasure Correction to improve reliability.
• Smarter flow control for heterogeneous receivers.
• Source authentication to defend against spoofed multicast traffic.
• Multiple flexicast flows (e.g., different video bitrates).
• Dynamic key rotation for large, changing groups.
• Inter-domain deployment using AMT and TreeDN.
• New use cases like software updates or gaming.

The source code and experiment scripts are open-source: Flexicast QUIC on GitHub.

The post Flexicast QUIC: Rethinking Multicast for the QUIC Era in SIGCOMM CCR appeared first on Tom Barbette.

In recent years, bypassing the Linux kernel networking stack has become a common strategy to accelerate packet processing. Techniques like DPDK eliminate kernel overhead, enabling network functions to run at much higher speeds. However, these solutions do not fundamentally change how packets are exchanged between the Network Interface Controller (NIC) and the CPU.

At the core of this communication are descriptors: NIC-specific metadata structures that reference memory buffers and encapsulate information such as VLAN tags, flow IDs, tunnel IDs, timestamps, and L3/L4 protocol information. Because descriptor formats vary across NIC vendors, both the Linux networking stack and user-space drivers like DPDK must translate these proprietary formats into a generic representation. This translation consumes substantial CPU resources, even though most of the metadata is typically unused by applications. Additionally, applications often perform further metadata transformations themselves. X-Change addresses some of those inefficiencies by proposing a unified model that merges driver-level and application-level metadata.

Ensō introduced a streaming interface that eliminates pointer indirection by delivering packets as a continuous array. While this improves performance through better memory locality and reduced overhead, it introduces challenges when packets need to be processed out of order, requiring costly copying operations. 

Introducing ASNI: Application-Specific Network Interface 

In our CoNEXT 2025 paper, we present ASNI, a new approach that builds upon, rather than replaces, the traditional packetized interface. ASNI delivers packets in large, contiguous buffers, each capable of holding multiple packets and their metadata, organized in a format tailored to the specific needs of the application. By offloading the majority of the driver datapath from the CPU to the NIC, ASNI significantly reduces CPU overhead. 

Our design brings three key benefits 

•     Improved PCIe efficiency through better buffer utilization 

•     Higher packet and metadata locality, reducing cache misses 

•     Application-specific metadata layouts and content, avoiding unnecessary transformations 

In NFV scenarios, ASNI outperforms DPDK, the dominant kernel-bypass solution, by serving 2.2× more traffic, demonstrating its effectiveness in high-throughput, low-latency environments. 

paper ; code ; acm

The post ASNI: Rethinking Packet I/O for High-Performance Networking appeared first on Tom Barbette.