Runtime Config Hot Reload

Status: Current

Last reviewed: 2026-05-16

This page documents the runtime configuration model used by nexus-backend and nexus-agent. It is the design reference for agent hot reload behavior, config versioning, apply reporting, and port conflict handling.

Summary

Runtime configuration is delivered as a DB-backed RuntimeConfigSnapshot. The snapshot is the aggregate root for one node's runtime target:

• PingoraConfig: L4 gateway listeners, routes, upstreams, TLS material, and listener policies.
• L3TunnelConfig: L3 tunnel rules and listener policies.
• NodeConfig: agent-owned node connectivity settings, currently multiplex enablement and port.

The production secure-control path sends one snapshot per config generation. The agent applies the snapshot as one target and reports the result with ConfigApplyReport.

Legacy split ConfigUpdate payloads still exist in protobuf for compatibility, but production backend publish/replay paths use RuntimeConfigSnapshot.

Problems Fixed

The previous hot reload behavior had several failure modes:

• Config generations were not meaningful enough for reconnect and replay. Some paths sent fixed versions, and some manual push paths could invent versions outside the DB-backed state.
• L4, L3, and node connectivity were streamed as independent partial updates. A lagged or failed stream could leave an agent with a mixed runtime target.
• Broadcast receiver lag was handled by continuing the stream, which could skip the newest authoritative state.
• Agent apply could partially change listeners before a later validation or L3 step failed.
• L3 tunnel updates and gateway L3 QUIC ingress updates could diverge.
• Node connectivity fields were not part of replay, so reconnect could miss multiplex changes.
• The backend had no explicit apply ack or nack from the agent.
• Port conflicts outside the manager were only logged and skipped, which made a failed desired listener look like a successful config apply.

Snapshot Contract

RuntimeConfigSnapshot is authoritative. If a domain has no desired runtime objects, the snapshot still carries an empty config for that domain. Absence of rules means "remove those runtime objects", not "leave previous objects alone".

The snapshot must include:

• config_version
• node_id
• pingora
• l3_tunnels
• node_config

The backend wraps the snapshot in ConfigUpdate.runtime_snapshot; the outer ConfigUpdate.config_version mirrors the snapshot version for compatibility.

Version Semantics

config_version is derived from persisted configuration state. The version is a microsecond timestamp generation built from the maximum relevant updated_at values for the target node:

• nodes.config_updated_at
• tunnel chains that reference the node by ID or name.
• Listener policies for the node.
• L3 networks containing the node.
• L3 endpoints in peer networks containing the node.
• L3 routes in peer networks containing the node.

nodes.updated_at is intentionally not the source of config generation. Heartbeat, hello, metrics, runtime listener reports, and ConfigApplyReport can update node runtime metadata without creating a new configuration generation.

Only backend paths that intentionally publish runtime config call AppState::publish_runtime_config(node_id). That method bumps nodes.config_updated_at and sends the current authoritative snapshot.

Publish Paths

The following backend paths publish snapshots:

• port policy upsert.
• L3 network create/update/deploy/delete flows.
• L4 chain create/update/deploy/delete flows.
• Node multiplex changes.
• POST /api/nodes/{id}/config refresh.

Tunnel chain update publishes to the union of previous transit nodes and new transit nodes. This is required when an update removes a node from a chain; the removed node must receive a new empty or reduced snapshot so stale listeners are removed.

Tunnel chain delete resolves affected transit nodes before deleting the row, then publishes snapshots after the delete.

POST /api/nodes/{id}/config is now a refresh endpoint. It accepts an empty body or {} and sends the current DB-backed snapshot. Inline tcp_rules or upstreams payloads are rejected because they bypass the persisted config generation model.

Replay And Lag Handling

On secure-control ConfigRequest, the agent sends its last applied config_version. The backend rebuilds the current snapshot for that node and sends it only if the current version is newer.

If the backend broadcast receiver lags, it does not trust intermediate missed messages. It rebuilds and sends the current authoritative snapshot.

The legacy AgentService.StreamConfig path follows the same snapshot replay shape for compatibility, but SecureAgentService.Control is the production agent control path.

Agent Apply Flow

The agent applies snapshots in this order:

1. Reject snapshots for a different node_id.
2. Ignore snapshots with config_version <= last_config_version.
3. Validate the snapshot has PingoraConfig and L3TunnelConfig.
4. Prepare L3 tunnel state without mutating the L3 runtime.
5. Apply the merged gateway config, including L4 and L3 QUIC ingress.
6. Commit prepared L3 tunnel state.
7. Apply NodeConfig multiplex settings.
8. Mark the version applied and update the version sent in future ConfigRequest messages.
9. Send ConfigApplyReport.

The agent keeps legacy split update handling for compatibility, but snapshot handling is the production behavior.

Rollback Rules

Snapshot apply must not leave the agent in a half-new state.

The agent snapshots previous gateway, L3, and multiplex runtime state before applying a new snapshot. If gateway validation or listener reconciliation fails, gateway state is restored internally. If L3 commit fails after gateway apply, the previous gateway state is restored. If NodeConfig apply fails after gateway and L3 changes, previous gateway, L3, and multiplex state are restored.

The multiplex listener binds the replacement port before shutting down the old listener. A failed port change therefore does not tear down the current multiplex listener.

Rollback is best-effort for operating system resources. If restoration itself fails, the agent logs an error and reports the original apply as rejected.

Port Conflict Semantics

Port checks distinguish two cases:

• The address is already owned by this ForwarderManager. This is a normal hot reload case. The agent updates the existing listener's route/upstream config in place where possible.
• The address is bound outside this manager. This is an apply error. The agent rejects the snapshot instead of skipping the listener.

The same rule applies to TCP listeners, UDP listeners, QUIC listeners, and L3 QUIC ingress listeners.

This means a config that asks for :443 does not silently succeed while the agent fails to run :443. Operators should see a rejected ConfigApplyReport with a message containing already bound outside this manager.

Apply Reports

After every non-stale snapshot apply attempt, the agent sends ConfigApplyReport:

• node_id
• config_version
• accepted
• message
• runtime_listeners

The backend stores this under node metadata:

• agent_config_apply
• node_port_runtime_listeners
• node_port_runtime_reported_at

The report update does not bump config_updated_at, so a report does not create a new config generation.

Operational Checks

To verify a node applied the expected snapshot:

1. Check node metadata agent_config_apply.version.
2. Confirm agent_config_apply.accepted is true.
3. Compare agent_config_apply.message; successful applies report applied.
4. Inspect /api/nodes/ports/runtime or node metadata node_port_runtime_listeners.
5. If a listener is missing, check for a rejected report and a port conflict message.

To force a refresh without changing stored config:

curl -X POST "$BACKEND/api/nodes/$NODE_ID/config" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}'

Do not use this endpoint to send inline L4 rules. Persist the desired L4/L3 resources first.

Compatibility

The protobuf still contains split update variants:

• ConfigUpdate.pingora
• ConfigUpdate.l3_tunnels
• ConfigUpdate.node_config

The agent still accepts them so older control-plane tools and tests do not break immediately. New production code should send runtime_snapshot.