Unified TUN I/O Engine Roadmap

Status: Implemented prototype + Linux netns throughput validated; external Windows/protocol validation pending

Last reviewed: 2026-06-22

Purpose

NexusNet 的 L3 runtime 同時有 QUIC REALITY DATAGRAM 和 BoringTun userspace WireGuard。兩者都把 inner IP packet 承載在 UDP-family datagram path 上。萬兆網路 下，單包 TUN read/write 會把 CPU 消耗集中在 syscall、checksum、segmentation、 allocation 和跨 worker copy。

本 roadmap 定義一個共享 TUN I/O engine：

OS TUN / Wintun / future helper ring
  -> TunIoEngine::recv_batch(&mut PacketBatch)
  -> SR header processor / GSO-GRO fallback / memory pool
  -> protocol workers: QUIC DATAGRAM or BoringTun
  -> TunIoEngine::send_batch(&PacketBatch)

Tailscale Source Notes

本設計參考兩組 Tailscale code path：

Reviewed upstream snapshots:

• github.com/tailscale/wireguard-go at 493cf277aedbf90efef3d5d59d6ca8e7d09de079
• github.com/tailscale/tailscale at 59159d9180bd56fb6c2ae58c426692d7edc47490

Source	用途	NexusNet 採用點
tailscale/wireguard-go/tun/tun_linux.go	Linux TUN 建立、IFF_VNET_HDR、TUN offload enable、read/write flow。	建立 TUN 時使用 `IFF_TUN
tailscale/wireguard-go/tun/offload_linux.go	10-byte virtio_net_hdr、GSO metadata、TCP/UDP GRO coalescing、writev scatter-gather。	Rust VirtioNetHdr 對齊 legacy Linux ABI，GsoMetadata 作為跨 OS metadata。
tailscale/wireguard-go/tun/offload.go	GSOSplit userspace fallback。	Rust FallbackOffloadEngine::split_gso_packet 已實作 TCP/UDP IPv4/IPv6 segmentation。
tailscale/tailscale/net/batching/conn_linux.go	UDP socket sendmmsg/recvmmsg、UDP_GSO/GRO control message。	NexusNet 的 UDP/QUIC socket path 可復用同樣策略；TUN char fd 不應誤認為 UDP socket。
tailscale/tailscale/net/tstun/wrap.go	gVisor/netstack GSO metadata 轉 TUN GSO options。	Protocol worker 不直接知道 OS adapter，只處理 batch 裡的 packet + metadata。

Important correction: Tailscale 的 Linux TUN VNET path 使用 TUN char fd read/writev 搭配 virtio header，不是 recvmmsg/sendmmsg。recvmmsg 和 sendmmsg 是 UDP socket batching。NexusNet prototype 提供了 mmsg adapter shape， 但真 TUN adapter 需要保留 read/writev fallback，否則部分 kernel 會回 ENOTSOCK。

Detailed source audit:

• offload_linux.go defines the Linux legacy 10-byte virtioNetHdr as flags, gsoType, hdrLen, gsoSize, csumStart, and csumOffset. Decode/encode are raw ABI copies, and toGSOOptions maps Linux VIRTIO_NET_HDR_GSO_NONE, TCPV4, TCPV6, and UDP_L4 into a platform-neutral metadata struct.
• tun_linux.go reads the VNET header first, converts it into GSOOptions, then normalizes HdrLen instead of trusting the kernel-provided value: UDP uses CsumStart + 8; TCP parses the TCP data-offset nibble and accepts only 20..60 bytes. LinuxVnetTunIo::try_recv_one mirrors this with normalized_gso_metadata.
• offload.go::GSOSplit validates checksum offset, header bounds, HdrLen >= CsumStart, IP version/type compatibility, and minimum TCP header size before writing segments. Each segment rewrites IPv4 total length, IPv4 ID, and IPv4 header checksum, or IPv6 payload length; TCP segments advance sequence number and clear FIN/PSH until the final segment; UDP segments rewrite UDP length; both protocols recompute transport checksum from the pseudo-header.
• offload_linux.go GRO uses per-batch TCP/UDP flow tables and a write-ordered scatter-gather list. TCP keys include ACK value, source/destination addresses, ports, and IP family. UDP accepts one smaller tail but prevents further append after the tail. NexusNet copies the same behavioral constraints into FallbackOffloadEngine and uses GroFlowScratch to avoid hot-path allocation.
• tailscale/net/batching/conn_linux.go applies sendmmsg/recvmmsg to UDP sockets, not TUN char fds. It coalesces UDP datagrams by scatter-gather with a max segment count and payload limit, and sets UDP GSO control data only when a batch contains multiple compatible datagrams. NexusNet keeps this as a socket-path reference and does not require sendmmsg for /dev/net/tun.

Module Shape

Current prototype files:

Module	State	Role
nexus-agent/src/l3/tun_io/mod.rs	Prototype	新 seam 匯出點，不替換既有 TunDevice。
packet_batch.rs	Prototype	預分配連續 buffer pool，加上 packet metadata/index。
virtio.rs	Prototype	10-byte virtio_net_hdr ABI 和 GsoMetadata。
engine.rs	Prototype	TunIoEngine trait、ProtocolWorker trait、userspace GSO split and TCP/UDP GRO fallback。
linux.rs	Prototype	IFF_VNET_HDR TUN 建立 helper、Linux VNET read/writev adapter、mmsg-shaped socket adapter。
windows.rs	Prototype	Wintun session/API table adapter，批次讀寫 ring packet。
device.rs	Bridge	將 batch-first TunIoEngine 包成既有 TunDevice 介面，供 runtime 漸進切換。
sr.rs	Prototype	自訂 SR header 的 in-place hop advance / strip processor。
protocol.rs	Prototype	L3 raw-IP / CONNECT-IP batch codec，讓 protocol worker 直接處理 PacketBatch。

Existing runtime still uses L3Dataplane<Tun, Transport>, but Linux offload TUN creation now first tries the batch engine through TunIoDevice and falls back to tun-rs if the host rejects the offload path.

Tier 1: OS Adapter

Linux

Linux adapter requirements:

• Create TUN with IFF_VNET_HDR.
• Enable TUN_F_CSUM | TUN_F_TSO4 | TUN_F_TSO6; try TUN_F_USO4 | TUN_F_USO6 but tolerate failure on kernels without UDP USO.
• Prefix received/written packets with the 10-byte legacy virtio_net_hdr.
• Parse gso_type, hdr_len, gso_size, csum_start, csum_offset.
• Prefer vectored/batched I/O. For true TUN char fd, use read/writev-style adapter. For socket-backed paths, use recvmmsg/sendmmsg.

Prototype state:

• LinuxTunFd::create opens /dev/net/tun, sets IFF_VNET_HDR, sets MTU, and configures offload flags.
• LinuxVnetTunIo is the real TUN char-fd adapter. It reads one VNET packet, parses virtio_net_hdr, uses userspace GSO split when needed, and writes batches with writev([virtio_hdr, packet]).
• Its send path now runs userspace GRO with reusable GroFlowScratch before writev, so adjacent TCP/UDP packets can be emitted as one VNET/GSO packet while reporting the original logical packet count to callers. Partial writes map each written coalesced packet back to the number of original logical packets it covered, so retry logic does not duplicate already written segments.
• The receive path can either split a received GSO super-packet into MTU-sized logical packets, or preserve the normalized GSO packet and metadata when the peer adapter can accept VNET/GSO metadata. The preserve path is the current high-throughput Linux relay mode.
• A pending-GSO receive buffer handles the case where the current PacketBatch does not have enough remaining slots for every segment of a kernel-delivered GSO packet. The adapter returns the packets already read and flushes the deferred GSO packet on the next receive call instead of failing with packet batch is full.
• LinuxMmsgTunIo remains available only for socket-like adapters that support recvmmsg/sendmmsg.
• linux_mmsg_adapter_batches_udp_socket_packets validates the recvmmsg / sendmmsg adapter against connected local UDP sockets, including VNET header prefixing/stripping.
• linux_vnet_send_batch_uses_gro_scratch_and_reports_logical_packets validates the Linux VNET send path on an unprivileged socketpair: two UDP packets become one VNET packet with UDP GSO metadata, while send_batch returns two logical packets consumed.
• linux_vnet_gro_spans_map_output_packets_back_to_logical_inputs and linux_vnet_tcp_gro_spans_map_output_packets_back_to_logical_inputs validate UDP and TCP GRO output-to-input accounting for partial-write retry safety.
• L3 runtime now tries LinuxVnetTunIo when TUN offload is enabled, then falls back to tun-rs if the local kernel/device rejects the path.

Windows

Windows adapter target:

• Bind Wintun session.
• Pull packets from Wintun receive ring into PacketBatch metadata slots.
• Write a batch by reserving Wintun send packets and copying from stable packet slices.
• Keep the ring single-owner per queue so worker path stays lock-free.
• Mark vnet_hdr=false; use Tier 2 fallback for GSO/GRO.

Prototype state:

• WintunBatchAdapter can bind an already-created Wintun session and API function table.
• recv_batch drains available Wintun receive-ring packets into PacketBatch.
• send_batch reserves Wintun send-ring packets and copies packet slices into the ring. If a PacketBatch entry carries GSO metadata, the adapter uses the userspace fallback splitter first and writes plain IP segments into Wintun. If ring pressure allows only part of a split GSO packet to be committed, the adapter reports the original logical packet as consumed; Wintun has no cancel API for already committed send packets, and retrying the super-packet would duplicate the committed segments.
• LoadedWintunSession provides a Windows-only wintun.dll loader and owns adapter/session lifetime around the batch adapter.
• Linux-host unit tests compile this adapter under cfg(test) and verify the ring copy path with a synthetic Wintun session.
• Synthetic Wintun tests cover null session rejection, unbound adapter errors, oversized RX packet release, GSO split-before-send, partial send when the TX ring is full, partial split-GSO send consumption, and first send allocation failure.
• cargo check --target x86_64-pc-windows-gnu now passes after gating Linux-only gateway fake TCP, kTLS, and fd socket-tuning paths behind Linux cfgs. This gives the Wintun adapter Windows-target type-check coverage.
• Real Windows Wintun driver/ring validation remains outside this Linux-hosted prototype.

Tier 2: Shared I/O Engine

The external interface is deliberately small:

use std::future::Future;

pub trait TunIoEngine: Send {
    fn name(&self) -> &str;
    fn mtu(&self) -> usize;
    fn capabilities(&self) -> OffloadCapabilities;
    fn recv_batch<'a>(
        &'a mut self,
        batch: &'a mut PacketBatch,
    ) -> impl Future<Output = std::io::Result<usize>> + Send + 'a;
    fn send_batch<'a>(
        &'a mut self,
        batch: &'a PacketBatch,
    ) -> impl Future<Output = std::io::Result<usize>> + Send + 'a;
}

The trait uses native impl Future return types and is consumed through generic static dispatch such as TunIoDevice<E: TunIoEngine>. The batch engine is not used as dyn TunIoEngine, so the hot path does not require async_trait boxed futures while still guaranteeing Send futures to the outer Tokio dataplane.

Important invariants:

• Hot path passes PacketBatch, not Vec<Vec<u8>>.
• PacketBatch owns a stable contiguous pool; packet metadata stores start/len indexes and GSO metadata.
• Protocol workers receive &mut PacketBatch; they should mutate packet bytes and metadata in place.
• Heap allocation is allowed during setup and capacity growth, not inside steady state packet movement.

Userspace Fallback

FallbackOffloadEngine currently implements TCP/UDP GSO split for IPv4 and IPv6, plus conservative TCP/UDP GRO coalescing for same-flow packets with valid checksums. TCP GRO supports in-order append and a Tailscale-style prepend case for one out-of-order segment, including the larger-prepend-to-single-segment case. UDP GRO accepts a final short tail segment and stops appending after that tail. Local regression tests now verify IPv4 TCP GSO segment IDs, TCP sequence advance, FIN/PSH tail-only preservation, IPv4 header checksum regeneration, UDP length rewriting, TCP/UDP transport checksums, GsoKind::None / unsplit NEEDS_CSUM finalization, malformed metadata rejection instead of panic, TCP PSH GRO boundaries, interleaved UDP flow separation, Tailscale-compatible single-segment split behavior when payload length equals GSOSize, bounded output batch failure when a GSO packet needs too many segments, TCP ACK-based flow separation, TCP prepend size constraints, existing GSO metadata preservation for uncoalesced packets, second-pass GRO skip for packets that already carry GSO metadata, and zero-allocation same-flow candidate scanning across multiple pending GRO items. A reusable GroFlowScratch now provides preallocated per-batch TCP/UDP flow indexes for large-batch lookup while keeping the benchmark hot window at zero allocations; the Linux VNET send path uses this scratch before writing virtio packets. Full parity still needs:

• Remaining fuzz-derived GSO edge cases copied from Tailscale's full GSOSplit test corpus.
• Kernel capability probe feeding OffloadCapabilities.

Tier 3: Protocol Workers

Protocol workers should become stateless batch processors:

pub trait ProtocolWorker: Send {
    fn process_batch<'a>(
        &'a mut self,
        batch: &'a mut PacketBatch,
    ) -> impl Future<Output = anyhow::Result<usize>> + Send + 'a;
}

Prototype state:

• L3BatchProtocolWorker encodes TUN PacketBatch entries into raw-IP or CONNECT-IP L3Frame batches.
• It decodes transport datagrams back into a preallocated PacketBatch.
• It centralizes MTU, multicast/broadcast, source-prefix, malformed packet, and unsupported IP-version accounting as L3BatchStats.
• L3Dataplane TUN-to-transport batch pumps now call L3DatagramTransport::send_packet_batch(PacketBatch, L3BatchProtocolWorker) so transports can override a batch-first path without changing the dataplane.
• The default send_packet_batch implementation encodes one packet at a time and no longer builds a whole-batch Vec<L3Frame> before sending.
• QUIC L3 transports override send_frame and send_packet_batch; HTTP/3 CONNECT-IP prefixing now accepts L3Frame directly, so the QUIC path avoids the previous L3Frame -> Bytes -> HTTP/3 Bytes intermediate copy.
• BoringTun userspace WireGuard overrides send_frame, send_packet_batch, and recv_packet_batch; raw-IP WireGuard packets are validated from PacketBatch slots and passed into boringtun without first building transport Bytes.
• L3Dataplane owns preallocated per-direction batch buffers and payload-length scratch vectors. Batch pumps clear and reuse these buffers instead of creating a new PacketBatch for each pump iteration.
• TunDevice now has recv_packet_batch and send_packet_batch compatibility hooks. TunIoDevice forwards caller-owned receive/send PacketBatch values directly to its underlying TunIoEngine, so the runtime TUN-to-transport path no longer needs a Vec<Vec<u8>> staging batch when the batch engine is active.
• FakeTunDevice and FakeDatagramEndpoint implement batch behavior, so local dataplane tests exercise more than one packet per pump.
• fake_dataplane_uses_tun_batch_hooks_without_legacy_vec_path uses a test TUN that fails every legacy single/Vec method; the dataplane passes only if it calls recv_packet_batch and send_packet_batch.
• perf-test tunio uses a counting allocator around the benchmark hot loops and reports allocation calls/bytes. The current userspace batch push, GSO split, and GRO coalesce windows report zero allocations after setup.
• perf-test tunio-real [name] [mtu] [batch_size] [recv_timeout_ms] is the privileged Linux VNET smoke entrypoint. It creates an IFF_VNET_HDR TUN, verifies reported vnet/TCP GSO capabilities, exercises empty send_batch, and waits briefly on recv_batch to prove the async adapter is wired without requiring test traffic.
• perf-test tunio-netns-relay <left_ns> <right_ns> [left_tun] [right_tun] [mtu] [batch_size] [tcp_offload] [udp_offload] is the privileged Linux throughput harness. It enters two existing network namespaces, creates one VNET TUN in each namespace, relays packets through two LinuxVnetTunIo engines, and can preserve GSO metadata when offload is enabled.
• The outer TunDevice / L3DatagramTransport traits still expose Vec / Bytes for compatibility; remaining work is allocation instrumentation and replacing the legacy compatibility calls at the outer edges.

Migration plan:

Step	State	Notes
Add tun_io prototype module and tests.	Complete	Current change.
Add LinuxVnetTunIo read/writev adapter.	Complete	Used for real TUN char fd when offload is enabled.
Add Wintun FFI adapter.	Locally complete	Session/API table adapter, DLL loader shape, and synthetic ring tests exist; real Windows driver validation remains.
Convert QUIC L3 worker to PacketBatch.	Complete + Planned	Runtime batch pumps use reusable batch buffers; QUIC TLS DATAGRAM perf harness validates the CONNECT-IP worker through Quinn TLS without REALITY. Allocation instrumentation outside the synthetic TUN I/O harness remains.
Convert BoringTun worker to PacketBatch.	Complete + Planned	BoringTun overrides raw-IP WireGuard batch send/receive, uses Linux sendmmsg with a preallocated strided slab on the batch send path, and has UDP-loopback batch tests. Allocation instrumentation outside the synthetic TUN I/O harness remains.
Implement GRO coalescing.	Locally complete	IPv4/IPv6 TCP/UDP same-flow coalescing exists; TCP prepend, TCP PSH boundary, interleaved UDP flows, multi-item reverse scan, reusable GroFlowScratch, Linux VNET send-path wiring, UDP short-tail cases, passthrough GSO skip, and logical span mapping are covered by local tests. Production defaults preserve TCP GSO and split UDP GSO unless UDP offload is explicitly enabled.
Implement SR header processor.	Prototype complete	In-place hop advance/strip exists; production auth/counters remain because final SR wire policy is not yet fixed.
Add namespace and protocol performance tests.	Partial	Privileged tunio-netns-relay validates Linux VNET relay throughput with offload on/off. perf-test quic-tls-l3 and perf-test boringtun-l3 validate full Quinn TLS DATAGRAM and BoringTun WireGuard encryption paths on loopback. Real Windows driver validation remains.

Current Risks

Risk	Mitigation
Treating TUN char fd like a UDP socket.	Keep mmsg limited to socket-like adapters; implement Linux TUN read/writev before runtime use.
Hidden allocation in protocol workers.	Add allocation counters or benchmark guard after QUIC and BoringTun migration.
GRO changes packet accounting.	Metrics must count logical packets after split/coalesce, not only syscalls.
GSO metadata from kernel can be misleading.	Parse transport header length from packet when metadata is suspect, matching Tailscale behavior.
Windows/macOS lack VNET header.	Tier 2 fallback remains mandatory, not optional.
Preserving GSO metadata can hide per-segment costs.	Treat iperf endpoint throughput as authoritative; relay packet counters are useful for movement diagnostics, not final capacity claims.
UDP GSO preservation can burst faster than the local queues absorb.	Runtime defaults now preserve TCP GSO only. UDP GSO/USO remains opt-in until socket/TUN queue and multiqueue tuning prove stable under UDP load.

Verification

Current prototype checks:

cd nexus-agent
cargo test tun_io
cargo test fallback_
cargo test dataplane
cargo test fake_dataplane_uses_tun_batch_hooks_without_legacy_vec_path
cargo test linux_mmsg_adapter_batches_udp_socket_packets
cargo test linux_vnet_send_batch_uses_gro_scratch_and_reports_logical_packets
cargo test linux_vnet_tcp_gro_spans_map_output_packets_back_to_logical_inputs
cargo test wintun_adapter
cargo test wintun_adapter_splits_gso_packets_before_send_ring
cargo test quinn_quic_reality_l3_packet_batch_reaches_server
cargo test quinn_quic_reality_l3_datagram_batches_server_replies
cargo test boringtun_packet_batch_round_trips_raw_ip
cargo test tun_io_device_recv_packet_batch_writes_into_caller_pool
cargo test fake_tun_reads_and_writes_packet_batches
cargo test boringtun_transports_exchange_l3_datagrams_over_udp
cargo check
cargo run --bin perf-test -- tunio 10
cargo run --bin perf-test -- tunio-real nntio-smoke 1500 128 100
cargo run --release --bin perf-test -- quic-tls-l3 1000000 1200 128
cargo run --release --bin perf-test -- boringtun-l3 1000000 1200 128
sudo -n env PATH="$PATH" cargo run --bin perf-test -- tunio-netns-relay ns-a ns-b tun-a tun-b 1500 128 true true
CARGO_TARGET_DIR=/tmp/nexus-agent-target-win cargo check --target x86_64-pc-windows-gnu

Example perf-test tunio 10 local smoke output now includes allocation columns:

   batch    push Mpps     gso Gbps     gro Mpps       allocs    alloc bytes
       1         1.90         0.29         0.31            0              0
      32         1.34         0.34         0.09            0              0
     128         1.95         0.33         0.09            0              0
     512         1.10         0.31         0.08            0              0
    1024         1.31         0.35         0.08            0              0

Most recent local run on 2026-06-22:

cargo test fallback_ --manifest-path nexus-agent/Cargo.toml
  34 passed
cargo test tun_io --manifest-path nexus-agent/Cargo.toml
  65 passed, 1 ignored
cargo check --manifest-path nexus-agent/Cargo.toml
  passed
CARGO_TARGET_DIR=/tmp/nexus-agent-target-win cargo check --manifest-path nexus-agent/Cargo.toml --target x86_64-pc-windows-gnu
  passed
cargo run --manifest-path nexus-agent/Cargo.toml --bin perf-test -- tunio 10
  all benchmark hot windows reported allocs=0 and alloc bytes=0
for proto in quic-tls-l3 boringtun-l3; do
  for run in 1 2 3; do
    cargo run --release --manifest-path nexus-agent/Cargo.toml --bin perf-test -- "$proto" 1000000 1200 128
  done
done
  QUIC TLS L3 median: 1.48 Gbps, 0.15 Mpps, 0.00% loss
  BoringTun WG L3 median: 1.70 Gbps, 0.17 Mpps, 0.00% loss
CARGO_TARGET_DIR=/tmp/nexus-agent-target-win cargo check --manifest-path nexus-agent/Cargo.toml --target x86_64-pc-windows-gnu
  passed
git diff --check
  passed

Privileged Linux VNET smoke was also run with sudo on 2026-06-22:

sudo -n env PATH="$PATH" cargo run --manifest-path nexus-agent/Cargo.toml --bin perf-test -- tunio-real nntio-smoke 1500 8 10
  capabilities: vnet_hdr=true, tcp_gso=true, tcp_gro=false, udp_gso=false, udp_gro=false
  empty send_batch: ok
  recv_batch: no packet within 10ms timeout
  Linux VNET TUN smoke PASSED.

This proves the current host can create/configure the IFF_VNET_HDR TUN and exercise the async Linux VNET adapter. It is not a throughput benchmark.

Linux netns throughput

Privileged namespace throughput was run locally on 2026-06-22 with:

• kernel: 6.12.90+deb13.1-cloud-amd64
• path: netns A -> VNET TUN -> perf-test Rust relay -> VNET TUN -> netns B
• MTU: 1500
• batch size: 128
• production crypto/protocol workers: not in path; this isolates TUN movement cost from QUIC and BoringTun encryption cost.

Scenario	TCP P1	TCP P4	UDP	Notes
Kernel veth baseline, no userspace relay	16.85 Gbps	66.16 Gbps	2.17 Gbps at 10 Gbps target, 0.46% loss	Upper bound for this local netns/veth setup, not a NexusNet path.
Debug build, TUN offload disabled	0.26 Gbps	0.25 Gbps	1 Gbps target showed high loss	Invalid as a capacity ceiling; it measured debug code plus no offload.
Release build, TUN offload disabled	4.38 Gbps	Not rerun	1.00 Gbps at 1 Gbps target, 0.00% loss	TCP had many retransmits; plain MTU packets make the relay CPU-bound earlier.
Release build, offload enabled, split/GRO relay	6.86 Gbps	6.35 Gbps	1.99 Gbps at 2 Gbps target, 0.12% loss	Correctly uses VNET offload but still burns CPU splitting and re-coalescing.
Release build, offload enabled, GSO metadata preserved	10.44 Gbps	8.13 Gbps	2.00 Gbps at 2 Gbps target, 5.21% loss	Current best local TUN relay result. TCP retransmits were 0.

Analysis:

• The earlier 0.26 Gbps result was not a valid architecture ceiling. It used a debug binary, offload was disabled, and the temporary relay was doing the most expensive per-MTU-packet path.
• For Linux VNET-to-VNET relay, preserving normalized GSO metadata is important. Splitting a kernel-delivered GSO super-packet and then rebuilding it with GRO is correct, but it spends CPU on work the kernel can already represent.
• The pending-GSO receive buffer is required for correctness. Without it, a GSO super-packet arriving near the end of a batch can require more output packet slots than remain, causing packet batch is full and relay failure.
• The current best number proves the TUN movement layer can reach a 10 Gbps single TCP flow in this local netns harness. It does not prove end-to-end QUIC REALITY or BoringTun throughput, because encryption, congestion control, socket buffers, and protocol framing were intentionally outside the test.
• TCP P4 being lower than TCP P1 points to relay-loop contention or scheduling effects in the single-process harness. Higher multi-flow targets need CPU profiling, queue parallelism, and possibly CPU pinning/multiqueue.
• UDP at 2 Gbps still showed 5.21% loss in the GSO-preserve run, so UDP burst tolerance needed socket/TUN queue tuning before calling that path production ready. The immediate mitigation is to preserve TCP GSO only and split UDP GSO unless UDP offload is explicitly enabled; targeted tests now cover both send and receive behavior.

Protocol-worker throughput

Full protocol-worker throughput was run locally on 2026-06-22 without REALITY. The QUIC harness uses Quinn with a generated self-signed rustls certificate, HTTP/3 DATAGRAM-style L3 framing, CONNECT-IP packet frames, and ChaCha/TLS crypto through Quinn. The WireGuard harness uses BoringTun userspace WireGuard, X25519 keys, preshared key, UDP loopback sockets, and raw-IP PacketBatch send/receive.

Commands:

for proto in quic-tls-l3 boringtun-l3; do
  for run in 1 2 3; do
    cargo run --release --manifest-path nexus-agent/Cargo.toml --bin perf-test -- "$proto" 1000000 1200 128
  done
done

Parameters:

• packets: 1,000,000
• inner IP payload: 1200 bytes
• inner MTU used by harness: 1280
• batch size: 128
• QUIC outer UDP payload: 1536
• Quinn reported max DATAGRAM: 1498

Path	Run 1	Run 2	Run 3	Median	Loss
QUIC TLS L3, Quinn DATAGRAM, CONNECT-IP worker	1.43 Gbps	1.51 Gbps	1.48 Gbps	1.48 Gbps / 0.15 Mpps	0.00%
BoringTun WG L3, raw-IP worker, Linux sendmmsg batch send	1.77 Gbps	1.70 Gbps	1.58 Gbps	1.70 Gbps / 0.17 Mpps	0.00%

Interpretation:

• These are loopback full-protocol numbers, not raw TUN movement numbers. They include TLS/WireGuard encryption, L3 frame encode/decode, socket I/O, and the protocol worker batch path.
• They do not test REALITY. That is intentional for this harness; REALITY handshake and camouflage costs are separate from Quinn TLS DATAGRAM and BoringTun WireGuard dataplane throughput.
• Current throughput is far below the 10.44 Gbps TUN movement ceiling. The remaining bottleneck is the protocol path: per-DATAGRAM crypto, Quinn congestion/pacing, BoringTun's per-packet WireGuard state machine, and UDP socket I/O.
• BoringTun improved after the raw-IP batch path switched to a preallocated strided slab plus Linux sendmmsg, avoiding per-packet Bytes copies and reducing UDP syscalls.

Required before runtime migration:

• Cross-kernel test where UDP USO is unsupported.
• Windows Wintun ring integration test on a real Windows host.
• CPU profiling for multi-flow TCP and UDP loss, including CPU pinning, multiqueue, and larger socket/TUN queue experiments.
• Multi-flow protocol-worker benchmark that avoids single sender/receiver task serialization and reports CPU utilization.