Oxy-Style Flow Core

The agent dataplane now has a pure flow core under nexus-agent/src/flow_core.rs. It models the same split used by Cloudflare Oxy/Bumblebee: packet classification stays separate from the platform runtime that turns an IP flow into a socket.

Current Boundary

• parse_packet reads IPv4/IPv6, common IPv6 extension headers, TCP, UDP, and ICMP.
• FlowKey normalizes a 5-tuple and can derive the reverse flow.
• FlowCore keeps short-lived flow state, first-byte prefaces, L7 sniffing, and counters. The shared DPI layer now distinguishes HTTP/1, HTTP/2 cleartext preface, and gRPC metadata when those first bytes are visible, so policy can route gRPC differently from generic HTTP/TLS flows.
• FlowPolicy is a pure hook layer for packet, flow, L4, and L7 routing decisions.
• FlowDispatcher maps core decisions to a runtime trait.
• FlowRuntime is the only layer allowed to forward raw packets, promote TCP to a kernel-owned stream, promote UDP, or hand traffic to an L7 handler.

The core does not own sockets, TUN file descriptors, namespaces, NAT state, or HTTP sessions. That is deliberate. Packet parsing must be deterministic and unit-testable, while socket ownership remains a Linux/runtime concern.

Source Layout

The flow core must stay split by responsibility. nexus-agent/src/flow_core.rs is the module entrance and orchestration layer, not a dumping ground for every parser, policy, handler, and test.

• flow_core/packet.rs: IP/TCP/UDP/ICMP parsing, FlowKey, and packet metadata.
• flow_core/dpi.rs: pure first-packet L7 sniffing for TCP and UDP protocols.
• flow_core/l7.rs: L7 protocol and DPI classification metadata.
• flow_core/policy.rs: pure policy/matcher data types for packet and flow hooks.
• flow_core/stream.rs: stream disposition, preread, replay, and L4/L7 dispatch.
• flow_core/datagram.rs: UDP/L7 datagram handler and response egress interfaces.
• flow_core/l4.rs: TCP splice interface and L4 upstream connection contract.
• flow_core/http.rs: HTTP/1, HTTP/2, gRPC, upgrade intent, and HTTP proxy policy.
• flow_core/upstream.rs: static upstream pools, matching, health, and connectors.
• flow_core/runtime.rs: dispatcher/runtime traits, L3 packet egress, and system runtime adapter.
• flow_core/linux.rs: Linux netns/TUN/NAT/listener promotion runtime entrance.
• flow_core/linux/: Linux promotion model, plan, command, executor, stream, and system runner modules.
• flow_core/tests.rs: shared flow-core test harness.
• flow_core/tests/: classifier, policy, runtime, HTTP, and upstream test modules.

New domain-heavy code should get a focused module before it grows into the entrance file. Keep public exports narrow and re-export stable API from flow_core.rs.

Oxy Research Notes

Cloudflare's public Oxy write-ups define the model this flow core is following:

• Oxy can on-ramp raw L3 IP packets, track tenant/application context, and keep those packets as IP packets when no higher-layer handling is required.
• For TCP, Oxy upgrades an IP flow to a stream by using Linux TCP instead of a userspace TCP stack: packets are NATed into a TUN interface inside an isolated network namespace, a listener accepts the kernel-owned TCP connection, and the accepted socket returns to the proxy runtime.
• Once a TCP stream exists, Oxy can run a stream-start hook and choose whether to keep the connection at L4, splice it, or pass it through an HTTP stack for L7 handling. That decision belongs to application policy, not to the packet parser.
• UDP is simpler: Oxy can strip/add IP+UDP headers in userspace and treat the payload as a datagram without creating a kernel stream.
• The reliability split is also important: Cloudflare describes Bumblebee as the IP-flow-to-TCP-socket upgrader, Splicer as the socket relay component, and Fish as the L3 egress component. We mirror those as narrow Rust traits rather than one monolithic dataplane object.

Primary sources:

• From IP packets to HTTP: the many faces of our Oxy framework
• Oxy: Fish/Bumblebee/Splicer subsystems to improve reliability
• How Oxy uses hooks for maximum extensibility
• Oxy is Cloudflare's Rust-based next generation proxy framework

Decision Model

L3 packets produce one of these actions:

• ForwardPacket: keep packet forwarding at L3 and pass the packet to the runtime PacketEgress.
• ForwardToPromotedStream: feed a packet from an already-promoted flow into the stream/TUN runtime boundary. FIN/RST packets set close_after_forward so the runtime can release the off-ramp after the final packet is delivered.
• PromoteToStream: move the flow into an L4 stream boundary.
• HandleL7: handle datagram-style L7 directly, currently used for UDP DNS/QUIC-style decisions.
• Drop: terminal failure path.

TCP promotion always targets StreamTarget::KernelTcp. Even when the first payload is HTTP, the action remains TCP promotion with StreamDisposition::L7Proxy. This preserves the Oxy rule that TCP stream ownership belongs to the kernel, not to a userspace TCP reassembler.

Default TCP promotion only starts at a kernel-recoverable boundary. Today that means a SYN-bearing first packet. Mid-flow ACK/payload packets stay at the L3 forwarding boundary even when policy asks for TCP promotion, because userspace cannot recreate kernel TCP state from payload bytes alone.

Once a SYN has promoted a TCP flow, the flow table keeps the selected StreamTarget. Later packets in either direction for that canonical flow return ForwardToPromotedStream, not ForwardPacket. The runtime must route those packets into the same promotion off-ramp, normally by injecting client-side packets into the promotion TUN and relaying kernel-emitted packets back to the original L3 path. This matches Oxy/Bumblebee's model: TCP ownership moves to the kernel, while the packet path becomes a pump around that kernel stream.

Promotion state is committed in two phases. FlowCore selects the off-ramp, but FlowDispatcher rolls that state back if the runtime cannot create the kernel promotion. This prevents later packets from being sent to a non-existent promotion namespace. When an already-promoted flow sees FIN or RST, the final packet is still delivered to the promoted stream boundary, then the runtime gets PromotedStreamClosed and the core removes the flow state.

Idle expiry is also explicit. FlowCore::expire_idle removes stale flow state and returns ExpiredPromotedFlow entries for promoted TCP off-ramps. The dispatcher converts those into FlowRuntime::close_promoted_stream calls before processing the next packet, producing PromotedStreamExpired without delivering a synthetic final packet. This keeps timeout cleanup distinct from FIN/RST shutdown and lets the Linux runtime release retained TUN/listener state.

When the packet path cannot identify L7 yet, TCP promotion uses StreamDisposition::DetectL7. The runtime must keep the accepted kernel stream at a preread boundary, collect the first application bytes, then continue to an L4 relay or L7 proxy.

preread_stream is the shared stream-side classifier for that boundary. It combines packet-path preface bytes with bytes read from the accepted stream, stops when L7 is complete enough for routing or when the configured byte limit is hit, and returns the complete preface so the runtime can replay those bytes into the selected L4/L7 handler.

SystemFlowRuntime now applies this preread step when a Linux promotion returns an actual promoted stream and the disposition is DetectL7, then hands the same stream plus final preface/disposition to PromotedTcpStreamHandler. If the stream is already known to be L4 or L7, it is handed directly to the handler without extra preread. Planning-only promotion still leaves DetectL7 visible in the event, which keeps the missing platform stream boundary explicit.

UDP promotion targets StreamTarget::UserspaceUdp. DNS and QUIC can be handed to L7 directly because they are datagram protocols.

Policy Hook Model

FlowPolicy runs after packet parsing and first-byte classification, before the default TCP/UDP behavior. Rules are pure data:

• match fields: IP version, direction, transport, CIDR, source/destination port, L7 protocol, L7 name, and payload presence;
• actions: forward at L3, drop, promote to stream, or handle L7;
• higher priority wins when more than one rule matches.

The hook preserves the Oxy stream boundary. A TCP rule that asks for L7 handling is converted into PromoteToStream with StreamDisposition::L7Proxy when L7 is already known, or StreamDisposition::DetectL7 when it is not. It does not hand TCP payloads to an L7 handler directly. UDP rules can hand datagrams to L7 directly.

SD-WAN Integration

The SD-WAN userspace dataplane now observes ingress and egress packets through SdwanFlowBridge, which owns a FlowDispatcher<SystemFlowRuntime<_>>:

• packets read from attachment TUNs are dispatched before NNSR encapsulation;
• packets popped at egress are dispatched before TUN/NAT/service handling;
• the bridge disables TCP kernel promotion so the current PTP forwarding path stays at the L3 boundary;
• DNS-style UDP still reaches the L7/datagram runtime path for policy and metrics.

Tests assert DNS traffic increments L7 handoff counters on both ingress and egress, while SD-WAN TCP kernel promotions and generic UDP promotions remain zero. This gives the SD-WAN dataplane one shared classifier/runtime seam for L3/L4/L7 policy work without changing the PTP carrying path yet.

Linux Runtime Contract

nexus-agent/src/flow_core/linux.rs defines the next platform boundary:

• KernelTcpPromotionRequest
• KernelTcpPromotionPlan
• KernelTcpPromotionExecution
• KernelTcpPromotionStepResult
• KernelTcpPromotionOutcome
• KernelTcpPromotionExecutor
• KernelTcpPromotionStepRunner
• LinuxPromotionCommand
• LinuxPromotionCommandRunner
• PromotedTcpStreamHandler
• KernelTcpPromoter
• SystemKernelTcpPromoter

The current Linux promoter builds a deterministic plan and returns both the plan and an execution report. The default executor is planning-only: every step is reported as planned, no Linux state is changed, and no stream is returned. A real Linux executor must mark applied steps and return a promoted TcpStream once the namespace/TUN/listener handoff is implemented.

StepRunnerKernelTcpExecutor now owns the executable step state machine. It validates the plan, invokes a KernelTcpPromotionStepRunner in order, records each step result, stops on the first failure, and requires the final accept step to produce a stream.

LinuxKernelTcpStepRunner is command-backed. It converts each promotion step into typed platform commands for ip, nft, TUN creation, initial L3 packet injection, TCP listener bind, and TCP accept. LinuxPromotionCommandRunner is the only layer that will execute those commands and return a PromotedTcpStream from the accept boundary. Command construction, step ordering, failure propagation, and stream return are test-covered.

SystemLinuxPromotionCommandRunner now has the first real Linux execution boundary. It can run ip and nft, create a VNET TUN inside a named netns using the existing tun_io path, retain the TUN fd for the flow lifetime, bind the promoted TCP listener from that namespace, inject the captured first L3 packet, then accept the resulting stream. Bind must happen before injection so the kernel has a listening socket ready when the SYN reaches the TUN path. The request carries both the full first packet and the TCP preface: the full packet is for TUN injection, while the preface is replayed into stream preread/L4/L7 handling.

The same runner also implements PromotedStreamPacketSink and PromotedStreamPacketSource. SystemFlowRuntime can be constructed with that I/O object so ForwardToPromotedStream packets are injected into the retained promotion TUN instead of being counted as generic L3 forwards, and so kernel-emitted packets can be read back from that TUN for outer L3 egress. The default runtime still uses a no-op sink for planning-only tests; the command-backed runtime wires the promoter and packet I/O to the same runner so both the initial SYN and follow-up packets hit the same netns/TUN state.

Plain L3 forwarding is also an explicit runtime seam. SystemFlowRuntime forwards ForwardPacket actions to PacketEgress, and the resulting PacketForwarded event records the packet length. The default NoPacketEgress keeps planning tests no-op, while SD-WAN or raw tunnel code can inject an adapter that sends unpromoted packets to the selected outer dataplane path.

pump_promoted_stream_egress is the current one-shot pump boundary. It reads a bounded batch from PromotedStreamPacketSource and passes each packet to PromotedStreamPacketEgress, which is where SD-WAN, raw tunnel, or future L4/L7 runtime code will undo any outer context/NAT and send the packet back toward the original peer. close_promoted_stream is the paired cleanup boundary; the system runner drops retained TUN/listener state for the flow after the final packet has been injected.

PromotedTcpStreamHandler is the stream-side handoff boundary. Once the Linux executor returns a TcpStream, SystemFlowRuntime keeps ownership through any required preread, then transfers the stream, preface, and final StreamDisposition to that handler. This is the place where the next layer will implement L4 splice, TLS handling, HTTP proxying, and future L7 hooks. The no-op handler is only for planning/tests; it must not be treated as the final data plane.

PrefaceReplayStream wraps a promoted stream and replays the captured preface before reading from the underlying socket. This is required because preread and packet-path classification consume bytes that L4 splice or an L7 HTTP handler must still see. DispatchingPromotedTcpStreamHandler is the first concrete splitter on top of that replay stream: L4Proxy and unresolved DetectL7 go to an L4StreamHandler, while L7Proxy goes to an L7StreamHandler with the detected protocol metadata.

L4SpliceHandler is the first concrete L4 handler. It asks an L4UpstreamConnector for the upstream stream with an L4UpstreamContext that contains the flow, endpoints, captured preface, and any detected stream metadata such as TLS SNI. It keeps the promoted side wrapped in PrefaceReplayStream and uses Tokio bidirectional copy so bytes flow in both directions. L4SpliceObserver receives the completed splice counters (client_to_upstream and upstream_to_client) for metrics and debugging.

HttpL7StreamHandler is the first concrete L7 handler boundary. It reads an HTTP/1 request head from the replay stream, parses method, target, version, headers, Host, and HTTP upgrade intent, then calls HttpRequestHandler with the parsed head and the same replay-capable stream for body/response handling. HttpPolicyHandler adds the first request-head policy layer: rules match method, Host, path, and structured upgrade intent. They can return a local HTTP/1 response immediately, rewrite the request method/target/headers before continuing to the fallback handler, or let unmatched requests fall through unchanged. HttpRequestRewrite regenerates the raw HTTP/1 head while preserving any body bytes already read with the header chunk, so HttpProxyHandler can forward a rewritten request without dropping prefetched body data. HTTP/2 and gRPC use a separate Http2L7StreamHandler boundary. It reads the connection preface far enough to parse HTTP/2 frame headers, records SETTINGS and HEADERS presence in Http2PrefaceInspection, and validates protocol shape for the initial frame window: first frame must be SETTINGS, SETTINGS/PING/GOAWAY must use stream 0, stream frames must not use stream 0, fixed-length frames must have the RFC-defined payload lengths, and frames cannot exceed the configured max frame payload length. By default protocol violations are rejected before the stream reaches the application handler. This is intentionally still a frame boundary, not a full HPACK/gRPC implementation, but it gives the dataplane a correct place to attach that logic without misparsing cleartext H2 as HTTP/1. DispatchingL7StreamHandler is the protocol splitter for HTTP/1 versus HTTP/2/gRPC handlers, and PrefaceReplayStream preserves the bytes inspected at the boundary.

HttpProxyHandler is the first concrete HTTP relay handler. It asks an HttpUpstreamConnector for an upstream stream, replays the captured HttpRequestHead.raw bytes back into the upstream side, then uses bidirectional copy for request body and response bytes. This preserves body bytes that may have been read together with the header chunk. By default upstream connect failures are returned as runtime errors; when with_failure_response is set, the proxy writes that local HTTP/1 response to the promoted stream and closes it cleanly. HttpProxyObserver receives completed relay counters for request-head bytes and both stream directions.

UDP/datagram handling is now a separate runtime boundary instead of a counter event. SystemFlowRuntime passes both PromoteToStream(UserspaceUdp) and datagram HandleL7 actions to a DatagramHandler with a DatagramContext containing the flow, local/remote socket tuple, current payload, detected L7 metadata, and disposition. DatagramRelayHandler is the first concrete handler: it asks a DatagramUpstreamConnector to relay the payload and returns request and response byte counts plus an optional response buffer. When a response is present, SystemFlowRuntime forwards it to DatagramResponseEgress, the paired boundary for reinjecting UDP/L7 responses toward the outer dataplane or client path. The default egress is a no-op for planning/tests.

StaticUpstreamPolicy and StaticTcpUpstreamConnector are the first upstream selection layer. Rules reuse FlowMatcher for L3/L4 fields and L7 metadata: L4 splice can route on preread classification such as TLS/SNI, while HTTP relay can add method, Host, and path matchers before selecting an UpstreamPool. Pools expose ordered candidates to connectors. The default load-balancer is round-robin; UpstreamLoadBalancing::FlowHash instead derives a stable index from the canonical 5-tuple plus L7 metadata, which keeps QUIC/UDP and sticky HTTP flows on the same upstream while the pool is healthy. The TCP connector implements both L4UpstreamConnector and HttpUpstreamConnector, so the same policy shape drives raw L4 splice and HTTP relay; it retries the next candidate when a TCP connect attempt fails. StaticUpstreamPolicy also tracks passive outlier state per endpoint. Failed TCP connect attempts eject an endpoint for a configurable cooldown; candidate generation first applies the load-balancer ordering, then skips ejected endpoints while at least one healthy candidate remains, and falls back to the full pool only when every endpoint is cooling down. StaticUdpUpstreamConnector uses the same policy for datagrams, including DNS/QUIC metadata supplied by DPI, and performs a single UDP request/optional response relay against the selected candidate. This is intentionally static data for now; DB-backed policy, active health probing, and service discovery should replace the source of rules without changing the L4/L7/datagram handler contracts.

The default promoter still uses the planning executor. Opting into the command runner requires constructing SystemKernelTcpPromoter with StepRunnerKernelTcpExecutor<LinuxKernelTcpStepRunner>, because the real path requires CAP_NET_ADMIN, a Linux host, and /var/run/netns access.

The next runtime slice must validate the real path under a privileged smoke:

1. Create or reuse an anonymous netns keyed by flow tuple.
2. Create a TUN in that namespace.
3. Install route and NAT state for the flow.
4. Bind the promoted TCP listener in the namespace.
5. Inject L3 packets into the TUN so Linux owns TCP sequencing and reassembly.
6. Accept the resulting TcpStream.
7. Pass the stream to L4 relay or L7 proxy based on StreamDisposition.

cargo run --bin flow_core_netns_smoke -- <service-netns> <client-netns> is the privileged Linux smoke for this boundary. It does not touch the live agent. It creates a service/client veth pair in caller-provided namespaces, drives the command-backed promoter through netns/TUN/route/NAT/listener bind/initial packet injection, reads the kernel's SYN-ACK from the promotion TUN, then injects ACK and HTTP preface packets back into the same TUN. The smoke passes only when the command-backed promoter returns a promoted TCP stream.

The next step after that smoke is replacing the smoke-only packet script with a long-running runtime TUN pump for the promotion namespace. Both pump halves now exist as interfaces: ingress injects client follow-up packets into the promotion TUN, and egress reads kernel-emitted packets from the TUN into an outer egress trait. The remaining integration work is binding those traits to the concrete SD-WAN/raw tunnel sender and promotion lifecycle.

Promotion trigger correctness is enforced in the core: full TCP handoff must happen at a kernel-recoverable boundary such as SYN/early flow setup; a mid-flow payload-only promotion cannot recreate kernel TCP state by itself.

Do not add userspace TCP reassembly to FlowCore. If TCP promotion cannot use the kernel path, the runtime should keep the flow at packet forwarding or drop it with an explicit reason.