QUIC REALITY Roadmap

Last updated: 2026-05-10

Purpose

Build a true wire-compatible QUIC REALITY mode for NexusNet. The end state is that TCP payloads can traverse a NexusNet chain over QUIC REALITY, while the public UDP-facing protocol remains valid QUIC v1 carrying TLS 1.3 handshake messages in CRYPTO frames. Non-REALITY or rejected clients should be able to fall back to a real camouflage QUIC target such as g.alicdn.com:443.

This file is the durable project memory for the QUIC REALITY work. Update it whenever scope, protocol decisions, milestones, or test results change.

DATAGRAM-specific throughput planning lives in quic-reality-datagram.md. Full L3 point-to-point tunnel planning across frontend, backend, and agent lives in l3ptp-reality.md. Keep this main roadmap focused on the shared QUIC REALITY transport and STREAM/TCP work.

Current Baseline

• TCP REALITY exists in nexus-agent.
• REALITY credentials are part of unified encryption credentials in backend/frontend.
• Existing TCP REALITY test chain has been validated with RAW -> REALITY -> RAW -> iperf3.
• apernet/quic-go was inspected. It is useful as a QUIC transport reference, especially for DATAGRAM, path behavior, GSO/MTU, congestion control, and proxy-oriented tuning. It does not implement REALITY and does not provide the QUIC TLS handshake insertion point we need.
• crates/quic-core has been created with initial QUIC v1 primitives and tests.

Non-Negotiable Goal

Implement real QUIC REALITY, not a custom UDP protocol and not QUIC-over-TCP.

Wire-compatible means:

• The listener speaks legal QUIC packets on UDP.
• TLS 1.3 handshake messages are carried in QUIC CRYPTO frames.
• Initial, Handshake, and 1-RTT packet protection follow RFC 9001.
• REALITY authentication is performed from the ClientHello carried in Initial CRYPTO.
• Accepted connections complete a valid QUIC TLS handshake.
• Rejected or non-REALITY flows can be forwarded to a real QUIC target without decryption.

Initial Scope

Support in the first production-quality target:

• QUIC version 1.
• TLS 1.3.
• X25519 key exchange.
• AES-GCM packet protection.
• Bidirectional QUIC streams for TCP payload forwarding.
• Server-side QUIC REALITY listener.
• Client-side QUIC REALITY upstream connector.
• Fallback UDP forwarding to a configured target.
• NexusNet chain mode for RAW -> QUIC_REALITY -> RAW.

Defer until after the first validated implementation:

• QUIC v2.
• 0-RTT.
• ECH.
• Retry.
• Connection migration.
• Multipath.
• Full browser-grade certificate chain validation for internal NexusNet peer mode.
• QUIC DATAGRAM payload forwarding.

Architecture Decision

Create an internal Rust QUIC implementation for the subset NexusNet needs, rather than trying to force the existing TCP REALITY TLS record logic into quinn or rustls.

Reasoning:

• TCP REALITY works at the TLS record layer.
• QUIC does not use TLS records. TLS handshake messages are fragmented into CRYPTO frames and protected by QUIC packet protection.
• Existing high-level QUIC libraries hide exactly the places where REALITY needs to inspect, authenticate, mutate, and sign the handshake.
• A minimal internal core lets us keep the protocol surface narrow and testable.

Planned crates:

• crates/quic-core: QUIC packet, frame, crypto, and minimal connection state.
• crates/quic-reality: REALITY authentication and QUIC TLS handshake integration.
• src/gateway/quic_reality.rs or equivalent: agent listener/client wiring and chain forwarding.

Current agent module layout:

• src/gateway/quic.rs: public agent-facing types, constants, and include list.
• src/gateway/quic/socket.rs: UDP/TCP socket tuning and send batching.
• src/gateway/quic/server.rs: async UDP listener, connection table, and server-side dispatch.
• src/gateway/quic/client_session.rs: client-side QUIC REALITY session, handshake, ACK, recovery, and stream/DATAGRAM packet building.
• src/gateway/quic/frames.rs: Nexus TCP stream frame and UDP DATAGRAM payload encoding helpers.
• src/gateway/quic/client_pool.rs: TCP relay entry points, pooled client connection, UDP-over-QUIC REALITY relay, and client send flushing.
• src/gateway/quic/probe.rs: QUIC server handshake, REALITY authentication, packet handling, ACK/recovery helpers, and server flight construction.
• src/gateway/quic/upstream.rs: server-side TCP/UDP upstream relay and response packet construction.
• src/gateway/quic/tests.rs: QUIC REALITY unit, integration, and ignored iperf smoke tests.

This is intentionally a first-stage physical split using include!, so all items still live in the original gateway::quic module and private visibility is unchanged. Later cleanup can convert these files into real Rust submodules after the hot paths and ownership boundaries are stable.

Milestone 1: QUIC Primitives

Deliverables:

• QUIC varint encoder/decoder.
• Long header parser/writer.
• Short header parser/writer.
• Packet number encode/decode.
• QUIC v1 Initial secret derivation.
• AEAD open/seal for Initial packets.
• Header protection remove/apply.
• Frame parser/writer for:
  • PADDING
  • PING
  • ACK
  • CRYPTO
  • STREAM
  • CONNECTION_CLOSE
  • RESET_STREAM
  • STOP_SENDING
  • flow-control frames used by early HTTP/3 clients

Validation:

• RFC 9001 Initial secrets test vector passes.
• RFC 9001 header protection mask sample passes.
• Internal Initial packet seal/open round-trip passes.
• QUIC varint boundary tests pass.
• Protected Initial long header shape parsing passes.
• Basic CRYPTO and STREAM frame parsing tests pass.
• Frame writers for ACK, CRYPTO, STREAM, PING, PADDING, and CONNECTION_CLOSE pass round-trip tests.
• CRYPTO stream reassembly helper handles reordering and duplicate data.
• Captured Initial packet from curl --http3 can be parsed.
• Initial CRYPTO data from a real curl HTTP/3 client can be decrypted and extracted.
• Agent QUIC rule path now starts a UDP probe listener that decrypts client Initial packets and sends an Initial CONNECTION_CLOSE response. This is not the final handshake; it validates listener, Initial open, server Initial seal, and response plumbing.
• Local smoke with curl --http3-only -k https://127.0.0.1:4443/ against quic-probe returns QUIC connection has been shut down, while agent logs show decrypted Initial packets and CRYPTO lengths.
• Test coverage now verifies split Initial CRYPTO reassembly and TLS ClientHello parsing through reality-core, including SNI, TLS 1.3 support, and X25519 key_share.
• quic-core now supports protected Handshake long-header packet open/seal with tests.
• quic-probe now keeps per-connection Initial CRYPTO state and can parse a real curl split ClientHello after multiple Initial packets.
• quic-core now supports protected short-header 1-RTT packet open/seal with tests.
• quic-probe now builds the complete server TLS handshake flight for QUIC: ServerHello in Initial CRYPTO, then EncryptedExtensions, Certificate, CertificateVerify, and Finished in Handshake CRYPTO.
• quic-probe now emits QUIC transport parameters extension 57 in EncryptedExtensions.
• quic-probe now handles a coalesced Initial + Handshake server flight sized to 1200 bytes, matching the curl peer's initial max UDP payload behavior.
• Real curl --http3-only -k interop now reaches TLS handshake completion. qlog shows handshake_complete.
• quic-probe verifies the client's Handshake Finished.
• quic-probe derives 1-RTT application keys, decrypts curl 1-RTT short-header packets, and logs HTTP/3 STREAM frames.
• quic-probe sends Handshake and 1-RTT ACK packets using server handshake/application keys; qlog shows curl receiving ACKs for Handshake PN0 and 1-RTT PN0-3.
• quic-probe sends a minimal HTTP/3 control stream SETTINGS frame, QPACK-encoded :status: 200 HEADERS, and a DATA body over 1-RTT short-header packets.
• Real curl --http3-only -k https://127.0.0.1:4443/ against quic-probe now returns HTTP/3 200 with body nexus quic probe.
• quic-probe stores the server Initial/Handshake CRYPTO flight and can re-seal it with fresh packet numbers when duplicate client Initial CRYPTO arrives before handshake completion. This is not full loss recovery, but it covers the first server-flight retransmission path.
• quic-probe tracks peer address, server/client connection IDs, server flight send time, and a small PTO loop. If client Finished does not arrive after the probe PTO, it re-seals and retransmits the stored server flight up to a bounded retry count.
• quic-probe now tracks server Initial and Handshake packet numbers in flight and removes them when client ACK frames acknowledge those packet ranges. The probe PTO only retransmits while the stored server flight still has unacknowledged packets.
• quic-probe now records retransmittable packet metadata for the stored server flight: packet space, packet number, sent time, packet length, bytes-in-flight, RTT samples, smoothed RTT, RTT variance, packet-threshold loss, and exponential PTO backoff. ACK handling now updates recovery state instead of only deleting packet numbers from a set.
• Probe connection state has started moving toward a reusable server connection shape. The connection now has explicit stages: AwaitingClientHello, AwaitingClientFinished, and Established, plus connection methods for peer refresh, Initial packet observation, CRYPTO buffering, ACK observation, and handshake completion.
• QuicRule now has REALITY config fields for mode, local X25519 private key, server names, short IDs, target host, and target port. Agent validation can build a QuicRealityConfig from those fields.
• The QUIC server ClientHello path now supports REALITY classification. When a QUIC rule is in REALITY mode, decrypted Initial CRYPTO is parsed as a TLS ClientHello and authenticated with the configured server names, short IDs, and X25519 private key before building the QUIC server flight.

Status: complete for the current primitive/probe target. Keep extending as new packet/frame types are needed.

Milestone 2: Minimal QUIC Server State Machine

Deliverables:

• UDP async listener.
• Connection table keyed by destination connection ID and client address.
• Per-connection packet number spaces:
  • Initial
  • Handshake
  • Application
• CRYPTO stream reassembly.
• ACK generation.
• Stored server Initial/Handshake CRYPTO flight for basic retransmission.
• Timer-driven probe PTO retransmit for the stored server flight.
• ACK range handling for server Initial/Handshake packets in flight.
• Minimal anti-amplification accounting.
• Handshake packet send/receive path.
• 1-RTT packet send/receive path.
• Simplified PTO/retransmit for handshake CRYPTO data.
• Bytes-in-flight accounting for stored server-flight CRYPTO retransmits.
• RTT sampling and adaptive PTO backoff for probe retransmission.
• Packet-threshold loss detection for Initial/Handshake server packets.
• Idle timeout cleanup.

Validation:

• Toy client and toy server complete a QUIC handshake using internal code.
• One dropped Handshake packet can be retransmitted.
• qlog-like debug output shows Initial -> Handshake -> 1-RTT transition.
• Real curl qlog shows Initial -> Handshake -> 1-RTT transition and handshake_complete.
• Agent logs show client Finished verification and decrypted 1-RTT HTTP/3 STREAM frames.
• Real curl receives an HTTP/3 200 response from the probe server over 1-RTT.

Status: in progress. The probe server handles the happy-path handshake, ACKs, one minimal HTTP/3 response, duplicate-Initial server-flight re-seal, timer-driven probe PTO retransmit, ACK removal for server Initial/Handshake packets in flight, bytes-in-flight accounting, RTT sampling, adaptive PTO backoff, packet-threshold loss for the stored server flight, and an explicit connection-stage model. It still lacks time-threshold loss, congestion control, flow control accounting, full reusable connection-state extraction into quic-core, and stream forwarding.

Milestone 3: TLS 1.3 for QUIC

Deliverables:

• ClientHello parser for:
  • SNI
  • supported_versions
  • supported_groups
  • key_share
  • signature_algorithms
  • ALPN
  • session_id or configured REALITY carrier field
• ServerHello builder.
• EncryptedExtensions builder.
• Certificate builder.
• CertificateVerify builder.
• Finished builder and verifier.
• TLS transcript hash management.
• HKDF schedule for TLS 1.3.
• QUIC TLS secret export:
  • client/server handshake traffic secrets
  • client/server application traffic secrets
  • QUIC packet protection key/iv/hp derivation

Validation:

• Internal toy QUIC client/server derive matching Handshake and 1-RTT keys.
• Finished verification fails on transcript mutation.
• 1-RTT STREAM frames can carry bytes both ways after handshake.
• SSLKEYLOGFILE comparison against curl confirms matching client/server Handshake traffic secrets after fixing TLS 1.3 Derive-Secret empty transcript handling.
• curl --http3-only -k receives a valid HTTP/3 200 response from the probe after client Finished verification.

Status: in progress. Server-side interop with curl is validated through a minimal 1-RTT HTTP/3 response. Client-side QUIC TLS, REALITY authentication, and payload stream forwarding are still pending.

Milestone 4: QUIC REALITY Handshake

Deliverables:

Server:

• Read ClientHello from decrypted Initial CRYPTO stream.
• Extract REALITY authentication material.
• Derive auth key from server private key and client ephemeral X25519 key.
• Match configured short IDs.
• Match configured server names.
• Build REALITY certificate with mutated public key material.
• Sign CertificateVerify with REALITY certificate key.
• Verify client Finished.
• Derive QUIC 1-RTT keys.
• Pull camouflage ServerHello shape from target when needed.

Client:

• Generate X25519 ephemeral key.
• Build QUIC ClientHello with REALITY authentication material.
• Seal authentication material using peer public key.
• Send Initial CRYPTO in QUIC packet form.
• Process ServerHello and encrypted handshake CRYPTO.
• Verify server Finished.
• Send client Finished.
• Enter 1-RTT.

Validation:

• Correct public key + short ID + server name is accepted.
• Wrong public key is rejected.
• Wrong short ID is rejected.
• Wrong server name is rejected.
• Accepted connection logs QUIC REALITY accepted.
• Rejected connection logs QUIC REALITY fallback.

Status: in progress. Server-side ClientHello authentication is wired into the QUIC Initial path and has unit coverage for accepted and rejected ClientHellos. Accepted connections use the configured X25519 private key and REALITY-mutated certificate for the QUIC server flight. Client-side QUIC REALITY now sends authenticated Initial packets, verifies the server flight, sends client Finished, waits for the server Finished ACK, enters 1-RTT, and relays TCP payload over a client-initiated bidirectional stream. Remaining work: target ServerHello shape support, rejected/fallback integration tests, 1-RTT retransmission, flow control, and iperf3 validation.

Milestone 5: Fallback to Real QUIC Target

Deliverables:

• UDP forwarding session map:
  • client address
  • target address
  • last activity
  • byte counters
• First packet classification:
  • valid and accepted REALITY -> internal QUIC handling
  • non-REALITY or rejected -> fallback forwarding
• Transparent packet forwarding to configured target such as g.alicdn.com:443.
• Idle cleanup and target socket lifecycle.

Validation:

• curl --http3 pointed at the NexusNet listener can reach the fallback target behavior.
• REALITY client is accepted and does not leak into fallback.
• Fallback packets are not decrypted or mutated.

Status: pending.

Milestone 6: Stream Forwarding

Deliverables:

• TCP inbound to QUIC bidirectional stream mapping.
• QUIC stream to TCP target mapping.
• EOF and half-close handling.
• RESET_STREAM / STOP_SENDING mapping to connection close behavior.
• Backpressure between TCP socket and QUIC stream flow control.
• Basic connection and stream flow-control windows.

Validation:

• RAW -> QUIC_REALITY -> RAW -> iperf3 works.
• Forward iperf3 run succeeds.
• Reverse iperf3 run succeeds.
• Parallel stream iperf3 run succeeds.
• Logs show REALITY accepted, not fallback.

Status: pending.

Milestone 7: NexusNet Control Plane Integration

Backend:

• Add or confirm Mode::QUIC_REALITY.
• Extend protobuf config fields for QUIC REALITY inbound and upstream.
• Extend encryption credentials metadata for QUIC REALITY:
  • target host
  • target port
  • server names
  • short IDs
  • X25519 key pair
  • optional ALPN
  • optional congestion profile
• Update chain builder for:
  • RAW -> QUIC_REALITY -> RAW
  • QUIC_REALITY -> RAW
  • future multi-hop QUIC REALITY

Agent:

• Apply config and spawn UDP QUIC REALITY listeners.
• Build upstream QUIC REALITY clients from pushed config.
• Report metrics:
  • accepted handshakes
  • rejected handshakes
  • fallback sessions
  • active QUIC connections
  • active streams
  • throughput
  • retransmits

Frontend:

• Add QUIC REALITY mode in L4 chains.
• Add credential form fields for QUIC REALITY.
• Show public key, target, server names, and short IDs.
• Surface agent status and handshake counters where existing UI allows.

Validation:

• Docker compose build succeeds for backend, agent, and frontend.
• Config push from backend to both test agents includes QUIC REALITY fields.
• Chain replay after agent reconnect restores listeners.

Status: pending.

Milestone 8: Test Matrix

Unit tests:

• varint boundaries.
• packet header parse/write.
• packet number decode.
• Initial secret derivation.
• header protection.
• AEAD open/seal.
• CRYPTO frame reassembly.
• TLS transcript hash.
• TLS Finished verify.
• REALITY auth success and failure.
• certificate signature mutation.

Integration tests:

• internal toy client/server.
• accepted QUIC REALITY handshake.
• rejected QUIC REALITY handshake.
• fallback to g.alicdn.com:443.
• RAW -> QUIC_REALITY -> RAW -> iperf3.
• dropped handshake packet recovery.
• reordered Initial/Handshake packets.
• multiple parallel TCP streams.
• Docker compose chain replay.

Interop tests:

• curl --http3.
• upstream quic-go client.
• quiche client if available.
• Wireshark decode sanity check.
• qlog trace review.

Status: pending.

Milestone 9: Performance Work

Deliverables:

• UDP batch receive/send where platform supports it.
• GSO support where platform supports it.
• Buffer pool for packet buffers.
• Tunable flow-control windows.
• Optional CUBIC/BBR-like congestion controller abstraction.
• Pacing.
• Profiling scripts and flamegraph notes.

Validation:

• iperf3 forward/reverse/parallel results recorded.
• CPU profile captured under sustained load.
• No unbounded memory growth under many short connections.

Status: pending.

Immediate Next Steps

1. Add 1-RTT retransmission and ACK tracking for TCP stream data; current 1-RTT STREAM frames are ACK-gated but not loss-recovered.
2. Tune throughput after correctness: add fresh-packet retransmission of STREAM data, batch ACKs, implement real packet number decoding, and benchmark forward/reverse/parallel iperf3.
3. Replace the QUIC REALITY server flight's synthetic ServerHello shape with a camouflage ServerHello shape from the configured target where feasible.
4. Add fallback UDP forwarding to g.alicdn.com:443 for rejected/non-REALITY clients.
5. Harden TCP-over-QUIC stream relay with: EOF/half-close mapping, RESET_STREAM/STOP_SENDING handling, stream receive buffering limits, and connection/stream flow-control accounting.
6. Continue extracting probe-only QUIC state into a reusable server connection state machine: move packet number spaces, recovery, ACK generation, and CRYPTO buffering behind a stable connection API.

Decision Log

2026-05-10: Use internal Rust QUIC core

Use an internal core because REALITY needs direct access to ClientHello, TLS transcript, certificate signing, and QUIC packet protection transitions. Existing high-level QUIC libraries are valuable references but hide too much of this flow.

2026-05-10: Reference apernet/quic-go but do not vendor it

apernet/quic-go is a proxy-optimized fork of quic-go. It improves DATAGRAM, congestion control injection, path behavior, GSO/MTU behavior, and high-throughput defaults. It does not implement REALITY. We will reference it for transport engineering decisions, not copy it as the core implementation.

2026-05-10: Create quic-core

Created crates/quic-core and added it to the agent workspace. Initial implementation includes QUIC varint, protected long header parsing, packet number encode/decode, QUIC v1 Initial secret derivation, packet protection key derivation, AES-GCM packet open/seal, AES header protection, and parsers for core frames. cargo test -p quic-core passes.

2026-05-10: Validate real curl Initial decryption

Captured a real curl --http3-only Initial packet sent to a local UDP socket and added it as crates/quic-core/tests/curl_initial.rs. quic-core decrypts the packet with QUIC v1 Initial keys and extracts the TLS ClientHello from a CRYPTO frame.

2026-05-10: Add QUIC probe listener

Replaced the old no-op QUIC forwarder stub with a UDP listener that can decrypt a client Initial packet and reply with a QUIC Initial packet containing ACK and CONNECTION_CLOSE frames. This is intentionally a probe server, not the final QUIC REALITY handshake, but it proves the agent can bind UDP QUIC rules and send protected Initial responses through quic-core.

2026-05-10: Probe listener smoke test

Added src/bin/quic-probe.rs for local smoke testing. Running it on 127.0.0.1:4443 and connecting with curl --http3-only -k produced a QUIC shutdown error on curl, which confirms curl accepted enough of the protected Initial response to process the CONNECTION_CLOSE. Logs showed successful Initial decryption and CRYPTO extraction.

2026-05-10: Validate CRYPTO reassembly into TLS parser

Added a split Initial CRYPTO test that seals two client Initial packets, receives them out of order, reassembles the CRYPTO stream, and parses the resulting ClientHello with reality-core. This validates the handoff point from QUIC packet handling to REALITY/TLS handshake classification.

2026-05-10: Add Handshake long-header packet support

Added generic long-header open/seal helpers and protected Handshake packet support to quic-core. The initial tests now include Handshake packet round-trip coverage.

2026-05-10: Emit first ServerHello Handshake packet

Updated quic-probe to keep per-connection Initial CRYPTO reassembly state. It parses a real curl split ClientHello, derives an X25519 shared secret, builds a TLS 1.3 ServerHello, derives QUIC server handshake packet protection keys from the TLS handshake traffic secret, and sends a protected QUIC Handshake packet containing a ServerHello CRYPTO frame. This is not yet a complete handshake because EncryptedExtensions, Certificate, CertificateVerify, Finished, client Finished verification, and 1-RTT keys are still pending.

2026-05-10: Reach curl QUIC TLS handshake completion

Added QUIC transport parameters encoding, TLS EncryptedExtensions extension 57 support, full QUIC server handshake flight generation, Handshake packet receive, short-header 1-RTT packet receive, and ACK writeback. Fixed two protocol bugs found with real curl interop:

• TLS 1.3 Derive-Secret with an empty transcript must use Hash("") as context, not an empty byte string.
• TLS Certificate message encoding must not add a nested certificate-list length inside the certificate_list vector.

Validation:

• cargo test -p quic-core -p reality-core passes.
• cargo check passes.
• curl --http3-only -k against quic-probe reaches qlog handshake_complete.
• SSLKEYLOGFILE shows curl's Handshake traffic secrets matching the probe's derived secrets.
• Probe logs verify client Finished and decrypt curl 1-RTT HTTP/3 STREAM packets.
• Probe sends Handshake and 1-RTT ACKs; curl qlog records those ACKs.

Remaining reason curl still times out: quic-probe is not an HTTP/3 responder yet. The QUIC/TLS layer has reached 1-RTT; application response and stream forwarding are the next layer.

2026-05-10: Add minimal HTTP/3 response over 1-RTT

Updated quic-probe to answer curl's first client-initiated bidirectional request stream. The server now sends:

• HTTP/3 server control stream ID 3 with an empty SETTINGS frame.
• Response stream HEADERS using a zero-dynamic-table QPACK block for :status: 200.
• Response DATA body nexus quic probe\n.

Validation:

• cargo test -p quic-core -p reality-core passes.
• cargo test -p nexus-agent quic::tests --lib passes.
• cargo check passes.
• curl --http3-only -k -sS -D - https://127.0.0.1:4443/ returns HTTP/3 200 and body nexus quic probe.

Remaining work: this is still a probe responder, not production HTTP/3 or NexusNet stream forwarding. The next implementation step is a reusable server connection state machine with retransmission/loss recovery and REALITY authentication hooks.

2026-05-10: Store server flight for first retransmission path

Updated quic-probe to retain the server Initial CRYPTO (ServerHello) and Handshake CRYPTO (EncryptedExtensions, Certificate, CertificateVerify, Finished) after building the first server flight. If duplicate client Initial CRYPTO arrives before client Finished, the probe now re-seals the same CRYPTO data with fresh Initial and Handshake packet numbers and sends the coalesced flight again.

Validation:

• cargo test -p quic-core passes.
• cargo check passes.
• curl --http3-only -k -sS -D - https://127.0.0.1:4443/ still returns HTTP/3 200 and body nexus quic probe.

Remaining work: this is a first retransmission primitive only. Full QUIC loss recovery still needs ACK tracking per packet number space, bytes-in-flight accounting, PTO scheduling, and CRYPTO retransmit queues.

2026-05-10: Add timer-driven probe PTO retransmit

Updated the probe listener's async loop so receive timeouts also run connection maintenance. Each probe connection now records the peer address, server/client connection IDs, server-flight send time, and retransmit count. If client Finished has not arrived after the probe PTO, the listener re-seals the stored Initial/Handshake CRYPTO flight with fresh packet numbers and sends it to the saved peer, bounded by a small retry limit.

Validation:

• cargo test -p quic-core -p reality-core passes.
• cargo check passes.
• curl --http3-only -k -sS -D - https://127.0.0.1:4443/ still returns HTTP/3 200 and body nexus quic probe.

Remaining work: PTO is still fixed-duration and probe-only. Production QUIC needs ACK tracking per packet number space, RTT estimation, exponential PTO/backoff, loss detection, bytes-in-flight accounting, and CRYPTO frame queues rather than re-sending a whole stored flight blindly.

2026-05-10: Add server-flight ACK tracking

Added in-flight tracking for server Initial and Handshake packet numbers. Client ACK frames in Initial or Handshake packet spaces now remove acknowledged server packet numbers using QUIC ACK range semantics. The probe PTO loop checks this in-flight state before retransmitting, so it stops once the stored server flight has been acknowledged or client Finished completes the handshake.

Validation:

• cargo test -p nexus-agent quic::tests --lib passes with ACK range unit coverage.
• cargo test -p quic-core -p reality-core passes.
• cargo check passes.
• curl --http3-only -k -sS -D - https://127.0.0.1:4443/ still returns HTTP/3 200 and body nexus quic probe.

Remaining work: this is ACK removal only. Production recovery still needs RTT sampling, ACK delay handling, packet metadata, byte accounting, loss timers, and selective CRYPTO retransmission.

2026-05-10: Add probe loss recovery metadata and adaptive PTO

Replaced the probe's fixed server-flight timer state with ProbeLossRecovery. The probe now records sent packet metadata for Initial and Handshake server-flight packets, tracks bytes in flight, samples RTT from newly acknowledged packets, maintains smoothed RTT and RTT variance, applies packet-threshold loss detection, and computes PTO with exponential backoff. Retransmits still re-seal the stored server CRYPTO flight, but the resend decision is now driven by recovery state instead of a fixed 300ms timer.

Validation:

• cargo test -p nexus-agent quic::tests --lib passes, including recovery tests for ACK byte accounting, RTT sampling, PTO backoff, and packet-threshold loss.
• cargo test -p quic-core -p reality-core passes.
• cargo check passes without warnings.
• curl --http3-only -k -sS -D - https://127.0.0.1:4443/ against quic-probe still returns HTTP/3 200 and body nexus quic probe.

Remaining work: recovery is still attached to the probe listener and retransmits the whole stored flight. Next step is to extract the per-connection state into a reusable QUIC server connection object before adding REALITY authentication.

2026-05-10: Start extracting server connection state

Renamed the probe connection state into QuicServerConnection and added an explicit stage model: AwaitingClientHello, AwaitingClientFinished, and Established. The packet handlers now go through connection methods for peer refresh, Initial packet-number observation, Initial/Handshake CRYPTO buffering, ACK observation, and handshake completion. This keeps behavior unchanged while creating the boundary where QUIC REALITY authentication will plug in.

Validation:

• cargo test -p nexus-agent quic::tests --lib passes.
• cargo test -p quic-core -p reality-core passes.
• cargo check passes.
• docker compose build passes and builds both nexus-agent-nexus-agent-hk:latest and nexus-agent-nexus-agent-sg:latest.
• curl --http3-only -k -sS -D - https://127.0.0.1:4443/ against quic-probe still returns HTTP/3 200 and body nexus quic probe.

Remaining work: the connection object still lives in src/gateway/quic.rs and still has probe-specific HTTP/3 response code. Continue extracting packet-number spaces, ACK generation, recovery, and CRYPTO buffering behind a stable API before moving it to quic-core.

2026-05-10: Wire server-side QUIC REALITY authentication

Added REALITY fields to QuicRule and wired agent validation so a QUIC REALITY listener receives server names, short IDs, target address, and the local X25519 private key from control-plane config. The QUIC Initial path now authenticates the decrypted TLS ClientHello with reality-core::authenticate_client_hello before building the server flight. Accepted QUIC REALITY ClientHellos use the configured X25519 private key and a REALITY-mutated certificate signed by the configured Ed25519 key.

Validation:

• cargo test -p nexus-agent quic::tests --lib passes, including QUIC REALITY ClientHello accept/reject tests.
• cargo test -p quic-core -p reality-core passes.
• cargo check passes.
• Probe mode without REALITY still returns HTTP/3 200 and body nexus quic probe to curl --http3-only -k.

Remaining work: there is not yet a QUIC REALITY client connector, so this is server-side auth plumbing and unit coverage rather than an end-to-end QUIC REALITY connection. Next steps are target ServerHello shape support, client-side QUIC REALITY ClientHello generation, and fallback UDP forwarding for rejected/non-REALITY packets.

2026-05-10: Add client-side QUIC REALITY Initial builder

Added a QUIC client Initial builder in quic-core and a QUIC REALITY ClientHello builder in the agent. The client-side builder generates an X25519 ephemeral key, builds a TLS 1.3 ClientHello with ALPN h3, seals the REALITY session_id with the peer public key, wraps the ClientHello in a QUIC CRYPTO frame, and emits a padded QUIC v1 Initial datagram. A server-path unit test decrypts that datagram, extracts the ClientHello, and authenticates it with the QUIC REALITY server config.

Validation:

• cargo test -p nexus-agent quic::tests --lib passes, including the client-built Initial -> server auth test.
• cargo test -p quic-core -p reality-core passes; quic-core now has direct coverage for padded client Initial generation.
• cargo check passes.
• Probe mode still returns HTTP/3 200 and body nexus quic probe to curl --http3-only -k.

Remaining work: this is a packet builder, not yet a socket connector. Next step is to send this datagram over UDP, parse the server Initial/Handshake flight, derive Handshake keys, verify server Finished, send client Finished, and reach 1-RTT.

2026-05-10: Add TCP-over-QUIC REALITY stream relay skeleton

Added a Nexus TCP-over-QUIC stream framing layer on top of 1-RTT bidirectional streams. Accepted QUIC REALITY server connections now recognize the NEXUS-TCP/1 stream preface, acknowledge stream open, connect one upstream TCP socket per stream, forward stream payload to the upstream socket, and seal upstream replies back into 1-RTT STREAM frames. Upstream read/write runs in per-stream async tasks and reports data back to the QUIC UDP loop through channels.

Validation:

• cargo test -p nexus-agent quic::tests --lib passes with stream open, split preface, rejection, and server response offset coverage.
• cargo check passes.

Remaining work: this is still a minimal relay. It does not yet implement production QUIC flow control, stream-level retransmission of 1-RTT data, RESET_STREAM/STOP_SENDING mapping, or backpressure.

2026-05-10: Make TCP over QUIC REALITY deployable from control plane

Changed TCP over QUIC REALITY from planned to supported in the transport profile model and frontend preset. Backend chain deployment now emits QuicRule for nodes whose inbound profile is TCP over QUIC REALITY. For previous hops whose next hop is QUIC REALITY, backend marks the TCP rule with upstream_quic_reality and includes the next-hop REALITY public key, server names, and short IDs.

Agent proto and validation now carry upstream_quic_reality. The TCP forwarder detects this flag and uses the QUIC REALITY upstream path instead of opening a TCP socket to the next hop.

Validation:

• cargo check passes in nexus-backend.
• npm run lint && npm run build passes in nexus-frontend.
• cargo test -p nexus-agent quic::tests --lib and cargo check pass in nexus-agent.

2026-05-10: Add client-side QUIC REALITY handshake session

Added a client-side QUIC REALITY session state machine in the agent. It sends the authenticated Initial datagram, processes coalesced server Initial/Handshake packets, derives Handshake keys, verifies the server Finished, builds raw QUIC client Finished CRYPTO data, derives 1-RTT keys, and can seal client-initiated TCP stream packets.

reality-core now exposes ClientHandshakeState::build_client_finished_message for QUIC, where Finished is carried as CRYPTO plaintext rather than a TLS record.

Validation:

• quic_reality_client_session_completes_handshake_with_server_path runs client and server packet handlers in memory: client Initial -> server flight -> client Finished -> server Established.
• cargo test -p reality-core passes.
• cargo test -p nexus-agent quic::tests --lib passes.
• cargo check passes.

Remaining work: socket-level integration tests, 1-RTT data retransmission, flow control, and iperf3 validation are still pending.

2026-05-10: Validate socket-level TCP over QUIC REALITY echo and iperf3

Added socket-level Tokio tests for the full local data path: TCP client -> local TCP ingress -> QUIC REALITY upstream client -> UDP QUIC REALITY listener -> upstream TCP server -> back to the TCP client. The default test uses an echo server and exercises the actual async TCP/UDP tasks rather than only in-memory packet handlers. An ignored smoke test runs the same path into local iperf3.

The test exposed and fixed two protocol bugs:

• The client previously entered 1-RTT as soon as it verified the server Finished. It now waits for the server Handshake ACK confirming the client Finished before sending TCP stream data.
• The server previously derived application traffic keys before verifying and transcript-updating the client Finished. It now re-derives 1-RTT application packet-protection keys after verify_client_finished, matching the client's transcript and allowing short-header STREAM packets to decrypt.

The server listener loop was also changed from timeout polling to tokio::select! over UDP receives and upstream TCP relay events, removing the artificial 250ms delay before upstream responses. Its receive buffer was raised to 64KB. TCP-over-QUIC FIN handling now maps QUIC stream FIN to upstream TCP write shutdown while keeping the upstream reader alive, so echoed data and remote FIN can still return after a client half-close.

The iperf3 smoke exposed that unbounded 1-RTT sends can overrun the UDP socket and create unrecoverable stream gaps. The client now tracks 1-RTT ACK frames, gates application sends with a small packet-count window, uses 4-byte 1-RTT packet numbers, and sends conservative 1KB STREAM chunks. This is intentionally correctness-first and is not the final throughput design.

Validation:

• cargo test quic::tests --lib passes in nexus-agent with 17 QUIC tests, including socket-level TCP echo, FIN-after-payload mapping, and an ignored iperf3 smoke.
• cargo test quic_reality_tcp_socket_iperf3_smoke --lib -- --ignored --nocapture passes on this machine with local /usr/bin/iperf3.
• Current local forward single-stream iperf3 smoke result with receiver-nonzero assertion: sender 1.00 MBytes in 1.00s, 8.38 Mbits/sec; receiver 256 KBytes in 1.03s, 2.03 Mbits/sec.
• cargo check passes in nexus-agent.

Remaining work: 1-RTT data is ACK-gated but still not retransmitted or flow-controlled like production QUIC. The current iperf3 number is a smoke result, not a performance target. Reverse/parallel iperf3, rejected/non-REALITY fallback forwarding, and production stream send queues are still pending.

2026-05-10: Add BBR-inspired client congestion controller

Replaced the fixed packet-count send window with a small BBR-inspired controller on the QUIC REALITY client application path. It tracks bytes in flight, RTT samples from 1-RTT ACKs, min RTT, a max-bandwidth estimate, a BDP-derived congestion window, and a simple pacing rate. Client TCP reads now feed a bounded pending queue, while STREAM sends are gated by pacing and bytes-in-flight. The server upstream TCP read buffer was raised to 16KB. Client STREAM chunks were tested at 8KB but were unstable because missing STREAM packets still create unrecoverable gaps without fresh-packet retransmission, so the stable chunk size remains 4KB.

This is BBR-inspired, not a full BBRv1/v2 implementation. Full BBR still needs round tracking, ProbeBW/ProbeRTT phases, loss response, proper packet-number recovery, byte-accurate delivered-rate sampling, and fresh-packet retransmission of STREAM frames.

Validation:

• cargo test quic::tests --lib passes in nexus-agent.
• cargo check passes in nexus-agent.
• cargo test quic_reality_tcp_socket_iperf3_smoke --lib -- --ignored --nocapture passed 3 consecutive runs with local /usr/bin/iperf3.
• Stable local forward single-stream iperf3 smoke after BBR-inspired changes: sender 1.75 MBytes in 1.00s, 14.7 Mbits/sec; receiver 1.00-1.12 MBytes in about 1.04s, 8.08-9.10 Mbits/sec.

Remaining work: retransmission still resends the old packet bytes with the old packet number, which is only a smoke-level fallback. The next throughput step is to retain STREAM frame metadata and retransmit lost/unacked stream ranges in new 1-RTT packets with fresh packet numbers, then add ACK batching and real packet-number decoding.

2026-05-10: Add STREAM-metadata retransmission, ACK ranges, and PN recovery primitive

Compared the current transport gaps against quiche, quic-go, and quinn. The common production pattern is:

• Store retransmittable frame metadata, not sealed packet bytes.
• Re-send lost STREAM ranges in a new packet number.
• Maintain received packet-number ranges for ACK generation.
• Recover truncated packet numbers from the largest received packet number.
• Keep UDP payloads near path MTU and rely on scheduling/batching instead of oversized UDP datagrams.

Implemented the first production-facing slice of that model:

• Client 1-RTT application sends now store STREAM frame metadata: stream ID, offset, data, and FIN.
• Application retransmission no longer re-sends old short-packet bytes. Timed-out packets are marked lost, removed from bytes-in-flight, their STREAM frames are queued, and retransmission seals the same stream offset/data into a fresh short-header packet with a new packet number.
• Client application chunk size now targets a 1200-byte UDP payload budget with conservative short-header/frame/encryption headroom, reducing the risk of IP fragmentation on real paths.
• Server 1-RTT receive state now tracks all received application packet numbers and emits ACK frames with gap ranges instead of ACKing only the current packet.
• Server 1-RTT ACKs are now batched. It flushes ACKs after two ACK-eliciting application packets or after a 10ms maximum ACK delay, and the listener maintenance timer now wakes at the earliest ACK/PTO deadline instead of a fixed 250ms tick.
• quic-core now has decode_truncated_packet_number and an open_short_packet_with_keys_and_largest helper. The agent still sends 4-byte packet numbers for now, but the receive path is ready for shorter packet-number encodings.
• The BBR-inspired controller now has a loss hook that reduces bytes-in-flight and backs off the model before fresh STREAM retransmission.

Validation:

• cargo test -p quic-core passes with packet-number recovery coverage.
• cargo test --lib quic::tests passes in nexus-agent with 21 passed and 1 ignored, including fresh-packet STREAM retransmission, ACK range, ACK batching, and ACK maintenance-deadline tests.
• cargo check passes in nexus-agent.
• cargo test quic_reality_tcp_socket_iperf3_smoke --lib -- --ignored --nocapture passes. Current conservative-MTU local result: sender 1.00 MBytes in 1.00s, 8.38 Mbits/sec; receiver 512 KBytes in about 1.04s, 4.05 Mbits/sec.
• cargo fmt passes.

Remaining work:

• Client loss detection is timeout-only; add packet-threshold and time-threshold loss for application packets.
• STREAM retransmission is frame-level but not yet byte-range-coalesced across partial ACK/loss. Next step is a real per-stream send buffer like quinn's send_buffer and quic-go's sendStream.
• BBR is still a simplified controller. Add delivered-rate samples, app-limited detection, ProbeBW/ProbeRTT phases, and better loss response.
• Packetization is now MTU-conservative, but high throughput needs packet batching, sendmmsg, GSO, and a scheduler that can fill each datagram with multiple frames where safe.
• The current deployment still creates one QUIC connection per TCP ingress connection. The long-term model is one QUIC REALITY connection per peer pair, with many client TCP connections mapped to bidirectional streams.

2026-05-10: Reach 3Gbps local TCP-over-QUIC REALITY throughput

Investigated why the local RAW -> QUIC REALITY -> RAW -> iperf3 smoke was only delivering about 4-8Mbit/sec in debug and about 2Gbit/sec receiver-side in release. The main causes were not crypto alone:

• The smoke test used Tokio's default single-thread #[tokio::test], forcing client connector, UDP listener, upstream relay, and TCP tasks onto one executor thread.
• The client send loop emitted one UDP packet per select wake.
• Loopback used WAN-sized 1200-byte packetization even though the local test needs a LAN throughput profile.
• The initial cwnd/pacing model was far below the requested throughput target.
• Server STREAM parsing cloned large STREAM payloads before forwarding them upstream.
• The service did not tune QUIC-side TCP/UDP socket buffers.

Implemented the local high-throughput path:

• Added automatic loopback LAN packetization with 56KB stream chunks, while keeping WAN MTU mode available through NEXUS_QUIC_REALITY_MTU_MODE=wan.
• Increased client application cwnd/pacing ceilings for the LAN path and changed client sends to burst-drain queued STREAM packets per wake.
• Tuned UDP sockets and QUIC-side TCP sockets with larger send/receive buffers.
• Added a borrowed application-frame parser for the server 1-RTT hot path so STREAM payloads are not cloned by parse_frames before forwarding.
• Added a sequential STREAM fast path on the server that bypasses the reassembly buffer when offsets arrive in order, while keeping the assembler synchronized for gaps/retransmits.
• Added application packet-threshold loss detection so ACK gaps can trigger fresh STREAM retransmission before the timeout path.
• Changed the iperf3 smoke to #[tokio::test(flavor = "multi_thread", worker_threads = 4)], matching the production multi-thread runtime more closely.

Validation:

• cargo test --lib quic::tests passes in nexus-agent.
• cargo test -p quic-core passes.
• cargo check passes in nexus-agent.
• cargo test --release quic_reality_tcp_socket_iperf3_smoke --lib -- --ignored --nocapture passed twice after the multi-thread/runtime and data-path changes:
  • Run 1: sender 496 MBytes in 1.00s, 4.15Gbit/sec; receiver 381 MBytes in 1.05s, 3.06Gbit/sec.
  • Run 2: sender 504 MBytes in 1.00s, 4.23Gbit/sec; receiver 375 MBytes in 1.02s, 3.09Gbit/sec.

Remaining work:

• This reaches the local 3Gbps target in the LAN profile, not the public-internet MTU profile.
• WAN mode still needs packet batching, sendmmsg, UDP GSO, and a scheduler to reach high throughput without relying on 56KB loopback datagrams.
• Multi-TCP over one QUIC REALITY connection is still required. The failed iperf3 -P 4 experiment showed that the current "one TCP connection = one QUIC connection" model overloads the single UDP listener and harms delivered receiver throughput.
• The next architectural step is a peer-level QUIC REALITY connection manager with multiple bidirectional streams per connection.

2026-05-10: Add peer-level QUIC REALITY client pool and multi-stream TCP reuse

Implemented the next architectural slice after the 3Gbps local single-stream work:

• Added QuicRealityClientPool, a long-lived client-side QUIC REALITY actor with one UDP socket, one QUIC/TLS/REALITY handshake, and many client-initiated bidirectional TCP streams.
• Client application events now carry stream_id, so server responses can be demultiplexed back to the correct local TCP connection.
• The TCP forwarder now creates the QUIC REALITY pool once per forwarder when upstream_quic_reality is enabled, then maps each accepted TCP connection to a new stream ID (0, 4, 8, ...) instead of creating a fresh QUIC connection per TCP accept.
• apply_config now starts QUIC listeners before TCP forwarders, so same-config TCP rules can connect their upstream QUIC REALITY pools to freshly started local QUIC rules.
• Added a pooled echo test that opens two local TCP connections concurrently and verifies both are relayed over the same QUIC REALITY pool.
• Added an ignored release smoke for iperf3 -P 4 over the pooled QUIC REALITY path.
• Replaced pool send-side HashMap scanning with a ready-stream round-robin queue so parallel TCP streams share the pooled QUIC sender more evenly.

Validation:

• cargo fmt passes.
• cargo check passes in nexus-agent.
• cargo test -p quic-core passes.
• cargo test --lib quic::tests passes in nexus-agent with 22 passed and 2 ignored.
• cargo test --release quic_reality_pooled_tcp_socket_parallel_iperf3_smoke --lib -- --ignored --nocapture passes. Initial local loopback result with iperf3 -P 4: sender aggregate 1.03 GBytes in 1.00s, 8.83Gbit/sec; receiver aggregate 470 MBytes in 1.07s, 3.67Gbit/sec.
• After round-robin pool scheduling, the same pooled iperf3 -P 4 smoke passes with sender aggregate 1.16 GBytes in 1.00s, 9.93Gbit/sec; receiver aggregate 543 MBytes in 1.14s, 3.99Gbit/sec. Per-stream receiver rates were also much more even, around 974Mbit/sec to 1.02Gbit/sec each.
• cargo test --release quic_reality_tcp_socket_iperf3_smoke --lib -- --ignored --nocapture still passes. This run delivered sender 505 MBytes in 1.00s, 4.23Gbit/sec; receiver 365 MBytes in 1.02s, 2.99Gbit/sec, showing single-stream local throughput is still near the 3Gbit/sec target but has small run-to-run variance.

Remaining work:

• The client pool currently keeps a simple stream pending queue and scans streams for pending data. Replace this with a proper fair packet scheduler that fills datagrams from multiple streams.
• Pooled stream close is still graceful FIN-oriented. Add RESET_STREAM / STOP_SENDING mapping and stream-level cleanup timers.
• Server-to-client 1-RTT STREAM responses are not yet retransmission-tracked with metadata like the client send path. Add server application send buffers and ACK processing before calling this production-grade.
• WAN mode still needs sendmmsg, UDP GSO, realistic MTU packetization, and byte-range send buffers to sustain high throughput without 56KB loopback datagrams.
• The same pool abstraction should later host QUIC DATAGRAM streams for L3 tunnel/IP-packet transport alongside TCP bidirectional streams.

2026-05-10: Add server-to-client STREAM retransmission and client ACKs

Closed the next reliability gap in the TCP-over-QUIC REALITY path:

• Client 1-RTT receive now tracks server application packet numbers and sends ACK frames back to the server, with the same packet-count/max-delay batching model used by the server.
• Server 1-RTT short-packet handling now parses client ACK frames from the borrowed application-frame parser instead of treating them as opaque Other frames.
• Server TCP response packets now store retransmittable STREAM frame metadata: stream ID, stream offset, payload, and FIN.
• Server application retransmission no longer depends on old sealed packet bytes. Timed-out or packet-threshold-lost response packets queue their STREAM metadata and are re-sealed with fresh short-header packet numbers.
• Listener maintenance now wakes for the earliest pending server application retransmission in addition to ACK/PTO deadlines.

Validation:

• cargo fmt passes.
• cargo check passes in nexus-agent.
• cargo test -p quic-core passes.
• cargo test --lib quic::tests passes in nexus-agent with 24 passed and 2 ignored.
• Added tests for server STREAM fresh-packet retransmission and client ACK clearing of server application retransmit state.
• cargo test --release quic_reality_pooled_tcp_socket_parallel_iperf3_smoke --lib -- --ignored --nocapture passes after this change. This run delivered sender aggregate 1.14 GBytes in 1.00s, 9.79Gbit/sec; receiver aggregate 457 MBytes in 1.06s, 3.61Gbit/sec.

Remaining work:

• Server retransmission is still packet/frame-level, not a full byte-range send buffer with partial ACK coalescing.
• Pooled TCP stream teardown still needs cleanup timers.
• The fair scheduler is round-robin at stream granularity; it still does not fill one datagram with frames from multiple streams or use sendmmsg/GSO.

2026-05-10: Add RESET_STREAM / STOP_SENDING stream teardown

Implemented the first reset/stop teardown slice:

• Borrowed 1-RTT frame parsing now decodes RESET_STREAM and STOP_SENDING instead of treating them as opaque Other frames.
• Client receive path maps server RESET_STREAM / STOP_SENDING into local stream reset events, removes client stream state, and shuts down the mapped local TCP side.
• Client pool sends RESET_STREAM when local TCP read/write fails, then removes queued stream state.
• Server receive path maps client RESET_STREAM / STOP_SENDING to upstream relay reset, removes per-stream server receive state, and drops retransmission metadata for that stream.
• Upstream TCP relay command channel now has an explicit reset command that shuts down the upstream write half and stops the relay task.

Validation:

• cargo fmt passes.
• cargo check passes in nexus-agent.
• cargo test -p quic-core passes.
• cargo test --lib quic::tests passes in nexus-agent with 26 passed and 2 ignored.
• Added tests for RESET/STOP parser coverage and server cleanup after a client reset.
• cargo test --release quic_reality_pooled_tcp_socket_parallel_iperf3_smoke --lib -- --ignored --nocapture still passes after RESET/STOP integration. This run delivered sender aggregate 1.20 GBytes in 1.00s, 10.3Gbit/sec; receiver aggregate 420 MBytes in 1.06s, 3.31Gbit/sec. The receiver distribution was roughly 741-869Mbit/sec per stream.

Remaining work:

• Reset packets are currently sent as control frames without retransmission metadata. Add control-frame retransmission tracking if reset delivery must be guaranteed under loss.
• Add stream cleanup timers for streams that are idle, reset only on one side, or waiting on FIN/ACK indefinitely.
• Add STOP_SENDING generation from server back to client when upstream write/read fails.

2026-05-10: Add Linux UDP sendmmsg batching for QUIC REALITY packets

Implemented the first syscall-batching slice for the QUIC REALITY hot path:

• Added safe async wrappers for connected UDP batch sends and peer-addressed UDP batch sends.
• On Linux, those wrappers use sendmmsg through a small audited FFI helper. Non-Linux and partial/error paths fall back to Tokio send / send_to.
• Client retransmission bursts and client/pool STREAM send bursts now collect sealed packets and flush them through the connected batch sender.
• QUIC server response lists, ACK flushes, handshake retransmits, application retransmits, and upstream response events now go through the peer-addressed batch sender.
• Added Linux sockaddr encoding coverage for IPv4/IPv6, with SAFETY comments on the unsafe FFI boundaries.

Validation:

• cargo fmt passes.
• cargo check passes in nexus-agent.
• cargo test -p quic-core passes.
• cargo test --lib quic::tests passes in nexus-agent with 27 passed and 2 ignored.
• cargo test --release quic_reality_pooled_tcp_socket_parallel_iperf3_smoke --lib -- --ignored --nocapture passes with sendmmsg batching. This run delivered sender aggregate 1.18 GBytes in 1.00s, 10.2Gbit/sec; receiver aggregate 428 MBytes in 1.10s, 3.28Gbit/sec.

Remaining work:

• This is syscall batching, not UDP GSO. WAN MTU throughput still needs segmentation-aware packet scheduling and GSO where available.
• send_to batching currently batches contiguous packets for the same peer. If many peers interleave, add per-peer outbound queues instead of sorting hot-path packets.
• Control frames such as RESET_STREAM are still sent individually.

2026-05-10: Group peer-addressed UDP batches by QUIC peer

Improved the server-side sendmmsg batching layer:

• send_to_datagrams now groups peer-addressed packets by SocketAddr before calling sendmmsg.
• Packet order is preserved within each peer group, and peer groups retain first-seen order.
• This prevents interleaved upstream events, ACK flushes, and retransmission output from reducing server sends back to many tiny per-packet syscalls.
• Added a Linux-only unit test for peer grouping behavior.

Validation:

• cargo fmt passes.
• cargo check passes in nexus-agent.
• cargo test -p quic-core passes.
• cargo test --lib quic::tests passes in nexus-agent with 28 passed and 2 ignored.
• cargo test --release quic_reality_pooled_tcp_socket_parallel_iperf3_smoke --lib -- --ignored --nocapture passes. This single-peer local smoke delivered sender aggregate 1.22 GBytes in 1.00s, 10.4Gbit/sec; receiver aggregate 419 MBytes in 1.09s, 3.22Gbit/sec.

Remaining work:

• Peer grouping helps server syscall batching when output is interleaved across peers; it does not solve the single-peer LAN smoke bottleneck by itself.
• The next throughput step is still real packet scheduling/GSO: aggregate multiple STREAM frames into MTU/GSO-sized output while preserving per-stream fairness and loss metadata.

2026-05-10: Batch upstream response events and reduce 1-RTT hot-path logging

Improved the server-side high-throughput response path:

• The upstream TCP event drain now batches the first received event together with any immediately queued events and sends the resulting QUIC packets with one send_to_datagrams call.
• This removes one async UDP send await per upstream read event on the hot response path.
• Per-STREAM 1-RTT packet logging was moved from info! to trace!; high-throughput tests should not spend cycles formatting/log-checking every STREAM frame.

Validation:

• cargo fmt passes.
• cargo check passes.
• cargo test --lib quic::tests passes with 28 passed and 2 ignored.
• cargo test -p quic-core passes with 21 unit tests and 2 integration tests.
• cargo test --release quic_reality_pooled_tcp_socket_parallel_iperf3_smoke --lib -- --ignored --nocapture passes. This run delivered sender aggregate 1.19 GBytes in 1.00s, 10.2Gbit/sec; receiver aggregate 435 MBytes in 1.08s, 3.38Gbit/sec.
• cargo test --release quic_reality_tcp_socket_iperf3_smoke --lib -- --ignored --nocapture passes. This run delivered sender 475 MBytes in 1.00s, 3.98Gbit/sec; receiver 391 MBytes in 1.09s, 3.01Gbit/sec.

Status:

• The local release smoke now reaches the 3Gbit/sec receiver-side target on both pooled parallel TCP and single TCP paths.
• This is still loopback/LAN mode throughput using large 56KB stream chunks. WAN-grade QUIC REALITY still needs MTU-sized packet scheduling, UDP GSO where available, and a per-stream byte-range send buffer.

2026-05-10: Coalesce small pooled STREAM frames into one 1-RTT packet

Implemented the first real packet-scheduler slice on the client pool send path:

• Split client TCP STREAM frame construction from packet sealing. build_tcp_stream_packet now builds a frame through build_tcp_stream_frame and then seals it as before.
• The pooled TCP sender now drains the ready-stream round-robin queue into a coalesced frame list before sealing a 1-RTT short packet.
• Multiple small STREAM frames can now share one QUIC packet while preserving per-packet retransmission metadata as a Vec<QuicRealityClientStreamFrame>.
• Large LAN frames that exceed the WAN payload budget are still sent alone, preserving the current high-throughput loopback behavior.
• Added coverage proving two small pooled streams are sealed into one application packet with two STREAM frame metadata entries.

Validation:

• cargo fmt passes.
• cargo check passes.
• cargo test --lib quic::tests passes with 29 passed and 2 ignored.
• cargo test -p quic-core passes with 21 unit tests and 2 integration tests.
• cargo test --release quic_reality_pooled_tcp_socket_parallel_iperf3_smoke --lib -- --ignored --nocapture passes. This run delivered sender aggregate 1.17 GBytes in 1.00s, 10.0Gbit/sec; receiver aggregate 437 MBytes in 1.09s, 3.36Gbit/sec.

Status:

• The pooled sender now has the structural ability to fill packets with multiple STREAM frames.
• The current coalescing budget is conservative and mainly helps small-frame / WAN-mode / interactive multiplexing cases.
• Next packetization work should move from whole-frame coalescing to a per-stream byte-range send buffer that can split large writes into MTU-sized STREAM ranges, then add UDP GSO for LAN/high-BDP paths.

2026-05-10: Add pooled per-stream send slicing for MTU-shaped STREAM ranges

Implemented the next packetization slice on the pooled client sender:

• QuicRealityPoolStream now tracks an offset into the front pending TCP chunk, so one large TCP read can be emitted as multiple STREAM frame byte ranges instead of one indivisible frame.
• The pool runner passes a max STREAM frame data budget into the scheduler. WAN mode can use the MTU-sized budget, while LAN/loopback still uses the larger chunk budget for throughput.
• The first frame on a new QUIC stream is capped to the WAN payload budget minus the NEXUS-TCP/1 open prefix even in LAN mode. This gets the open prefix and early payload to the server in an MTU-sized packet before larger data ranges, improving multi-stream startup robustness.
• Added tests for splitting a large pending chunk into contiguous STREAM offsets and for limiting the first open frame to the WAN-sized budget.

Validation:

• cargo fmt passes.
• cargo check passes.
• cargo test --lib quic::tests passes with 31 passed and 2 ignored.
• cargo test -p quic-core passes with 21 unit tests and 2 integration tests.
• A first release pooled smoke after raw slicing had aggregate receiver 3.50Gbit/sec but exposed one parallel iperf receiver stream at 0 Bytes. After capping new-stream open frames to the WAN-sized budget, cargo test --release quic_reality_pooled_tcp_socket_parallel_iperf3_smoke --lib -- --ignored --nocapture passes with all four receiver streams active: sender aggregate 1.22 GBytes in 1.00s, 10.5Gbit/sec; receiver aggregate 461 MBytes in 1.09s, 3.55Gbit/sec.

Status:

• The pooled sender now has per-stream byte-range slicing, which is the necessary foundation for real MTU packetization.
• Remaining gap: the server response path still emits large LAN chunks as whole frames. It needs the same byte-range slicing before WAN mode can be considered packetization-complete.
• UDP GSO is still not implemented; the current work controls STREAM frame sizes, not kernel segmentation.

2026-05-10: Add server response slicing and coalesced response packets

Implemented the matching server-side STREAM response packetization slice:

• Server TCP upstream responses are now chunked through server_stream_response_data_limit.
• WAN mode uses the MTU-sized QUIC_REALITY_WAN_STREAM_CHUNK_SIZE for server STREAM response ranges; default LAN/loopback mode keeps the 56KB chunk size for local throughput.
• Server response packets now use build_quic_tcp_response_packets, which can coalesce multiple small STREAM responses into one 1-RTT packet while preserving retransmission metadata.
• Added a server STREAM frame encoded-length helper so response coalescing uses actual QUIC frame size accounting rather than an ad hoc payload estimate.
• Added tests covering small server response coalescing and large response split offsets/FIN placement.

Validation:

• cargo fmt passes.
• cargo check passes.
• cargo test --lib quic::tests passes with 33 passed and 2 ignored.
• cargo test -p quic-core passes with 21 unit tests and 2 integration tests.
• cargo test --release quic_reality_pooled_tcp_socket_parallel_iperf3_smoke --lib -- --ignored --nocapture passes. This run delivered sender aggregate 1.22 GBytes in 1.00s, 10.4Gbit/sec; receiver aggregate 447 MBytes in 1.11s, 3.37Gbit/sec.

Status:

• Client pooled sends and server upstream responses both now have byte-range slicing and small-frame coalescing.
• The remaining packetization gap is full MTU-aware packet assembly around encrypted short-packet overhead and UDP GSO. Current frame budgets are conservative and do not yet use kernel segmentation.
• Pooled iperf aggregate remains above 3Gbit/sec, but per-stream fairness still has run-to-run variance. A later scheduler pass should add explicit per-stream byte quotas/deficit round-robin instead of only ready-queue round-robin.

2026-05-10: Add deficit byte scheduling for pooled client streams

Implemented the first byte-fair scheduler pass for the pooled client sender:

• Each QuicRealityPoolStream now tracks a send_deficit byte credit.
• The ready-stream scheduler now uses deficit round-robin semantics: a stream receives scheduler credit before it can emit a STREAM byte range, then the emitted byte count is deducted.
• Added a dynamic scheduler quantum. WAN mode keeps MTU-sized credit; LAN/loopback mode grows the quantum to the active frame budget so 56KB local chunks do not cause many empty scheduler rotations.
• Added a fairness test showing two large streams emit frames in alternating stream order.

Validation:

• cargo fmt passes.
• cargo check passes.
• cargo test --lib quic::tests passes with 34 passed and 2 ignored.
• cargo test -p quic-core passes with 21 unit tests and 2 integration tests.
• A first release pooled smoke with a fixed WAN-sized quantum still passed but exposed a throughput/fairness regression: receiver aggregate 424 MBytes in 1.10s, 3.23Gbit/sec, with one stream receiving only 128KB. A second run dropped aggregate receiver to 397 MBytes in 1.15s, 2.90Gbit/sec. The cause was that LAN 56KB frames required dozens of empty quantum rotations.
• After switching to a dynamic quantum, cargo test --release quic_reality_pooled_tcp_socket_parallel_iperf3_smoke --lib -- --ignored --nocapture passes with sender aggregate 1.23 GBytes in 1.00s, 10.6Gbit/sec; receiver aggregate 452 MBytes in 1.10s, 3.46Gbit/sec. Per-stream receiver rates were much tighter at roughly 831-901Mbit/sec.

Status:

• Pooled client send fairness is materially better in the local release smoke.
• This is still send-side fairness only. Server response scheduling now slices/coalesces frames, but it does not yet have a per-stream deficit scheduler of its own.
• Next options: add server response deficit scheduling, or move down one layer to UDP GSO / UDP_SEGMENT.

2026-05-10: Stabilize pooled throughput after fairness regression

Fast follow-up after a throughput drop:

• Upstream response events are now reordered round-robin by (connection_id, stream_id) within each drained batch before packet sealing. This prevents one upstream reader from monopolizing a whole server response drain cycle.
• Added a unit test for upstream event round-robin ordering.
• Reduced the temporary application STREAM retransmission timeout from 250ms to 50ms. In the 1-second local iperf smoke, a single early missing STREAM range can otherwise stall a stream for a large part of the test duration. This is a tactical setting until RTT/PTO-driven application recovery replaces the fixed timer.

Validation:

• cargo fmt passes.
• cargo check passes.
• cargo test --lib quic::tests passes with 35 passed and 2 ignored.
• cargo test -p quic-core passes with 21 unit tests and 2 integration tests.
• Three consecutive cargo test --release quic_reality_pooled_tcp_socket_parallel_iperf3_smoke --lib -- --ignored --nocapture runs after the fix:
  • receiver aggregate 452 MBytes in 1.07s, 3.54Gbit/sec
  • receiver aggregate 406 MBytes in 1.08s, 3.14Gbit/sec
  • receiver aggregate 404 MBytes in 1.07s, 3.16Gbit/sec

Status:

• The pooled release smoke is back above the 3Gbit/sec receiver-side target for repeated runs.
• Per-stream receiver rates are no longer showing 0 Bytes or near-zero starvation in these runs.
• The fixed 50ms application retransmission timeout is not the final QUIC recovery design; replace it with RTT/PTO-based loss recovery.

2026-05-10: Replace fixed application retransmit timeout with bounded RTT/PTO recovery

Implemented the next recovery and packetization stabilization pass:

• Replaced the temporary fixed 50ms application STREAM retransmission timer with ApplicationRecoveryTimer.
• Client and server application packet ACK handling now samples RTT from the original sent packet timestamp and resets the PTO loss backoff after ACK progress.
• Client and server application loss paths now apply bounded PTO backoff instead of retrying with a flat timeout.
• The application PTO is intentionally clamped between 20ms and 250ms for the current local/agent transport profile. This avoids the earlier 250ms stall in 1-second iperf runs, while still avoiding an unbounded retransmit storm after repeated loss.
• Short-packet STREAM packing now reserves explicit short header and AEAD tag budget. The helper quic_reality_short_payload_budget separates encrypted packet payload budget from the more conservative WAN STREAM chunk budget.
• Added unit tests covering RTT/PTO bounds and short-packet header/AEAD budget accounting.

Validation:

• cargo fmt --check passes.
• cargo check passes.
• cargo test --lib quic::tests passes with 37 passed and 2 ignored.
• cargo test -p quic-core passes with 21 unit tests and 2 integration tests.
• cargo test --release quic_reality_pooled_tcp_socket_parallel_iperf3_smoke --lib -- --ignored --nocapture passes. This run delivered sender aggregate 1.21 GBytes in 1.00s, 10.4Gbit/sec; receiver aggregate 443 MBytes in 1.09s, 3.39Gbit/sec. The four receiver streams were balanced at roughly 822-881Mbit/sec.
• docker compose config --quiet passes.
• docker compose build passes and builds both nexus-agent-nexus-agent-hk:latest and nexus-agent-nexus-agent-sg:latest.

Status:

• The pooled QUIC REALITY smoke is again above the 3Gbit/sec receiver-side target after replacing the fixed retransmit timer.
• The direct cause of the previous local throughput drop was early stream-range loss interacting with a coarse fixed application retransmit delay and batch-drain unfairness. The current code addresses both: upstream responses are batch-reordered by stream, and application retransmission now follows bounded RTT/PTO state.
• Remaining recovery gap: this is still an application-level STREAM recovery shim over the custom QUIC core. The next protocol-quality stage should move closer to RFC 9002 recovery by integrating PTO/loss state with ACK delay, sent-packet metadata, and congestion delivery-rate samples in one packet-space recovery module.
• Remaining throughput gap: no UDP GSO / UDP_SEGMENT yet. Loopback/LAN mode reaches the 3Gbit/sec receiver target with large chunks and batching; WAN-grade MTU packetization still needs segmentation-aware sends to keep syscall cost low.

Next engineering stages:

1. Do not enable server response deficit scheduling yet. A prototype caused local pooled iperf variance and was reverted; keep the current direct response packetization plus upstream-event round-robin until recovery and UDP send scheduling are stronger.
2. Move application recovery into a shared recovery module for 1-RTT packets, with ACK delay handling, loss-time calculation, and PTO probes modeled after RFC 9002.
3. Add Linux UDP GSO support for MTU-sized short packets, with runtime fallback to sendmmsg.
4. Add DATAGRAM frame plumbing for future L3 tunnel mode. TCP-over-QUIC should stay STREAM based; IP packet forwarding should use QUIC DATAGRAM once the transport is stable.

2026-05-10: Revert server response scheduler prototype to restore the stable 3Gbit/sec point

Outcome:

• Reverted the server response deficit scheduler prototype from the runtime path.
• Kept the previously stable pieces: upstream event round-robin, pooled client deficit scheduling, bounded RTT/PTO application recovery, and short-packet budget accounting.
• Restored the application recovery timer initialization to the prior 50ms baseline.

Validation after revert:

• cargo fmt passes.
• cargo check passes.
• cargo test --lib quic::tests passes with 37 passed and 2 ignored.
• cargo test --release quic_reality_pooled_tcp_socket_parallel_iperf3_smoke --lib -- --ignored --nocapture passes with sender aggregate 1.26 GBytes in 1.00s, 10.8Gbit/sec; receiver aggregate 452 MBytes in 1.11s, 3.41Gbit/sec. Per-stream receiver rates were roughly 905, 861, 860, and 782Mbit/sec.

Current stable point:

• Keep this version as the local performance baseline.
• The next scheduler work should be behind a flag or a separate experiment, not merged into the default runtime path until it beats this baseline repeatedly.