Observability Matrix

Status: Roadmap

Last reviewed: 2026-05-15

This roadmap defines the future monitoring and audit telemetry plane for NexusNet. It is a design target, not current behavior.

The goal is to make every agent produce structured observations that can be used by:

• user-facing system status checks.
• node health monitoring.
• network performance monitoring.
• SD-WAN path selection and failover.
• security audit and compromise detection.
• long-term TimeDB / TimescaleDB analysis.

Target Architecture

agent runtime
  -> observability collector modules
  -> local bounded queue
  -> secure control stream MetricsReport frame
  -> backend telemetry ingest module
  -> hot status cache + audit event stream + TimeDB hypertables
  -> dashboard, health checks, SD-WAN decision engine, alerting

Module Boundaries

Component	Planned module	Responsibility
Agent collector root	nexus-agent/src/observability/mod.rs	Owns the collection loop, local queue, rate limits, and report batching.
Agent system collector	nexus-agent/src/observability/system.rs	CPU, memory, disk, process, boot identity, binary measurement, and runtime health.
Agent network collector	nexus-agent/src/observability/network.rs	Interfaces, IP addresses, DNS view, public address view, route table, socket/listener status.
Agent performance collector	nexus-agent/src/observability/perf.rs	RTT, jitter, packet loss, throughput, TCP_INFO, QUIC DATAGRAM, Noise, and L3 metrics.
Agent security collector	nexus-agent/src/observability/security.rs	Config hash, credential epoch, pin validation, config replay/apply state, suspicious events, and integrity signals.
Agent event packer	nexus-agent/src/observability/report.rs	Normalizes events into protobuf envelopes and sends them over the secure control stream.
Backend ingest root	nexus-backend/src/telemetry/mod.rs	Verifies report identity, deduplicates events, classifies severity, and dispatches to storage.
Backend status engine	nexus-backend/src/telemetry/status.rs	Maintains current node/link/system state for dashboard and health APIs.
Backend TimeDB writer	nexus-backend/src/telemetry/timedb.rs	Writes time-series samples into the TimescaleDB instance.
Backend audit bridge	nexus-backend/src/telemetry/audit.rs	Converts security and control-plane observations into durable audit entries.
SD-WAN signal builder	nexus-backend/src/telemetry/sdwan.rs	Produces path score inputs for route selection and failover.

gRPC Envelope

The existing secure control stream already has SECURE_FRAME_KIND_METRICS_REPORT. Future work should extend MetricsReport into a versioned observability envelope rather than creating another unauthenticated channel.

Planned shape:

message ObservabilityReport {
  string node_id = 1;
  uint64 sequence = 2;
  google.protobuf.Timestamp collected_at = 3;
  string schema_version = 4;
  string agent_version = 5;
  string config_hash = 6;
  uint64 config_version = 7;
  string credential_epoch = 8;
  repeated Observation observations = 9;
}

message Observation {
  string id = 1;
  string category = 2;       // system, network, performance, security, audit
  string subject_type = 3;   // node, link, listener, route, upstream, credential
  string subject_id = 4;
  string severity = 5;       // debug, info, warn, critical
  google.protobuf.Timestamp observed_at = 6;
  map<string, string> labels = 7;
  bytes payload_json = 8;
}

Rules:

• Reports are sent only inside the encrypted secure control stream.
• node_id, sequence, config_hash, and credential_epoch are mandatory.
• Backend must reject stale or replayed sequences per active control session.
• Payloads are typed by category and subject_type; backend persists raw JSON and extracts indexed fields into typed tables.
• Reports must be bounded. Large detail objects are sampled, truncated, or sent as diagnostic artifacts in a later design.

Data Matrix

Security And Audit Signals

Signal	Labels	Fields	Use
Config applied	config_version, config_hash, signature_id	apply result, listener hash, route hash, duration	Detect stale config, failed rollout, rollback.
Config replay state	config_version, config_hash, replay_source	requested version, applied hash, duplicate skipped, replay latency, backend session id hash	Restart recovery, stale config detection, and rollout audit.
Credential epoch	credential_epoch, credential_ids_hash	active key ids hash, revoked key ids hash, next rotation time	Revocation and stale-key detection.
SPKI pin validation	upstream_id, pin_id, transport	expected pins, observed SPKI SHA-256, result, peer address	Detect MITM between agent and upstream.
REALITY validation	credential_id, server_name, short_id, listener	accept/reject reason, target host, protocol, peer address	REALITY abuse detection and route audit.
Noise handshake	credential_id, listener, peer_node_id	result, handshake duration, rekey count, error kind	Detect Noise failure spikes or stale keys.
Control session	session_id_hash, psk_fingerprint	open/close reason, sequence window, rekey age	Detect duplicate sessions, replay, abnormal disconnects.
Node identity drift	node_id	boot id hash, machine id hash, binary hash, public IP/ASN, region	Detect cloned nodes or unexpected relocation.
Integrity signal	binary_hash, config_hash	process start time, executable path hash, container image digest	Detect replaced agent binary or unexpected runtime.
Operator action echo	audit_id, resource_type	applied version, node ack set, failed node set	Correlate user action with distributed apply state.

System Status Signals

Signal	Labels	Fields	Use
Node resource snapshot	node_id	CPU, memory, load average, uptime, process RSS, fd count	Node health and capacity planning.
Disk state	mount	free bytes, inode pressure, read-only flag	Detect host storage pressure and agent instability.
Process state	pid, agent_version	restart count, panic count, last error class, runtime thread count	Agent health status.
Clock state	node_id	wall clock offset, monotonic drift estimate, NTP sync state	TLS/REALITY validity and log ordering.
Capability report	feature	supported, enabled, reason, kernel version	DCO, GSO/GRO, TUN, QUIC DATAGRAM readiness.
Listener state	transport, listen_addr, listen_port	bound, owner, route count, policy hash, bind error	Port conflict and routing status.

Network State Signals

Signal	Labels	Fields	Use
Interface snapshot	ifname	addresses, MTU, carrier, rx/tx counters, drops, errors	Link health and SD-WAN availability.
Route table snapshot	table, ifname	default route, route count, managed route hash	Detect route drift and L3 apply failures.
Public address view	source	observed IPv4/IPv6, ASN, region, changed flag	Node reachability and clone detection.
DNS view	resolver	answer set hash, latency, failure rate	Detect poisoned DNS or regional divergence.
Peer reachability	peer_node_id, transport	reachable, last success, failure reason, NAT observation	SD-WAN peer graph.
Upstream reachability	upstream_id, transport	connect success, error class, observed peer cert pin	Origin health and MITM detection.

Network Performance Signals

Signal	Labels	Fields	Use
Peer link metric	peer_node_id, transport	RTT, jitter, packet loss, throughput estimate	SD-WAN path score.
TCP_INFO snapshot	listener, upstream	srtt, rttvar, cwnd, retrans, pmtu	TCP path quality and congestion insight.
QUIC connection metric	route_id, upstream	handshake time, RTT, loss, cwnd, datagram sent/received/dropped	QUIC REALITY path score.
Noise session metric	peer_node_id	handshake duration, packets, bytes, decrypt failures	UDP ENC health.
L3 tunnel metric	network_id, endpoint_id, peer_endpoint_id	packets, bytes, drops, MTU errors, route installs	L3 health and routing decisions.
Listener route metric	chain_id, listener	active connections, total connections, errors, bytes in/out	Capacity and policy tuning.
Health probe metric	upstream_id	TCP connect time, TLS pin result, REALITY result	Origin selection and alerting.

User-Facing Status Signals

Status	Inputs	Output
Node status	control session, resource snapshot, config ack, listener state	online, degraded, offline, maintenance, compromised-suspected
Link status	peer reachability, RTT, loss, jitter, QUIC/Noise errors	healthy, degraded, unstable, down
Upstream status	reachability, pin validation, route errors, health probes	healthy, degraded, pin-mismatch, unreachable
Security status	pin mismatch, identity drift, stale credential epoch, config replay mismatch	normal, suspicious, compromised-suspected
SD-WAN score	latency, loss, jitter, throughput, stability, security flags	numeric route score and reason codes

TimeDB Storage Plan

The backend compose file already includes a postgres-timescale service for future metrics. The ingest module should separate current-state tables from time-series hypertables.

Planned hypertables:

Hypertable	Time column	Primary dimensions	Examples
node_metric_ts	time	node_id, metric	CPU, memory, disk, process counters.
link_metric_ts	time	node_id, peer_node_id, transport	RTT, jitter, loss, throughput.
listener_metric_ts	time	node_id, transport, listen_addr, listen_port	route counts, active connections, errors.
route_metric_ts	time	node_id, route_id, chain_id, network_id	bytes, packets, connection count.
security_event_ts	time	node_id, event_type, severity	pin mismatch, identity drift, stale config.
probe_result_ts	time	node_id, target_id, probe_type	TCP connect, TLS pin, REALITY, DNS.
sdwan_score_ts	time	src_node_id, dst_node_id, transport	score, selected path, reason codes.

Retention:

• high-frequency raw samples: 7-14 days.
• downsampled 1-minute rollups: 30-90 days.
• security events and audit-derived records: retain according to audit policy.

Backend Processing Pipeline

Planned ingest flow:

1. Decrypt SecureFrame.
2. Decode ObservabilityReport.
3. Verify node identity, active session, sequence, config hash, and schema version.
4. Deduplicate observations by (node_id, sequence, observation_id).
5. Update hot status cache.
6. Write selected samples to TimeDB.
7. Convert security observations to audit log entries.
8. Emit SD-WAN decision inputs.

Backend must not trust node-reported compromise signals alone. Security status should combine self-reports with cross-node probes and backend-side facts.

Cross-Node Verification

To detect MITM or compromised routes, backend should schedule independent probes:

• Ask multiple nodes to observe the same upstream SPKI pin.
• Compare DNS answers and public address views across regions.
• Compare REALITY handshake result and observed target behavior.
• Mark a node as suspicious when only that node reports divergent pins or route behavior.

Cross-node verification is a detection layer; it is not a replacement for TPM or remote attestation.

Security Requirements

• Never send plaintext private keys or PSKs in telemetry.
• Hash identifiers when exact values are not needed.
• Redact IPs only when the feature can still function; SD-WAN and health logic often need exact endpoint addresses.
• Treat agent telemetry as untrusted input until verified against control-plane state and cross-node observations.
• Security observations must include enough context for audit review but must not include secret material.

Implementation Phases

Phase	State	Work
1. Schema and modules	Planned	Add observability agent module, telemetry backend module, and versioned report protobuf.
2. Config replay audit	Planned	Report config hash, signature id, replay/apply state, listener hash, and apply result.
3. Security pins	Planned	Report SPKI pin validation results and pin mismatch events.
4. System and listener status	Planned	Report resource snapshots, listener bind state, route counts, and capability reports.
5. Network performance	Planned	Report peer link, TCP_INFO, QUIC, Noise, L3, and health probe metrics.
6. TimeDB ingest	Planned	Add typed TimescaleDB hypertables, retention, and rollup policy.
7. SD-WAN signals	Planned	Feed route score inputs from TimeDB/current status into SD-WAN policy.
8. Attestation-ready signals	Planned	Add boot id, binary hash, image digest, TPM measurement placeholders, and trust-level scoring.