Preflight is observing a live local agent · run the attack

BEHAVIORAL CONTRACTS FOR AI AGENTS

Your agent ran.Was it allowed?

Preflight learns which agent actions are safe, tests the rules against real SigNoz history, and blocks an irreversible action before it reaches production. Detection is automatic; blocking always needs a human.

01Without Preflightunprotected

Poisoned incident

→

drop_database runs with no snapshot

→1,200 rows gone. No backup. Unrecoverable.
02With Preflightrecoverable

Same poisoned incident

→

Unsnapshotted drop is refused

→Snapshot created. Backed-up drop allowed. Recoverable.
ENGINEWaiting
LOCAL MODEL--
EVALUATOR--
TRAJECTORYILLUSTRATIVE EXAMPLE
Preview only

Illustrative sequence. Not measured until you run an episode.

STEPS4TOKENS--LATENCY--BLOCKED--
  1. 01get_incidentreadexample
  2. 02snapshot_databasesnapshotexample
  3. 03read_logsreadexample
  4. 04drop_databasedestructiveexample
AWAITING RUNRun an episode to begin

Preflight reads the ordered tool calls and their real outcomes, then decides.

Observe→Evaluate→Decide
THE OBSERVABILITY GAP

Every request returned 200. That says nothing about whether the agent followed policy.

RETURNED 200 OK100%● looked healthy
CONTRACT-PASSING100%● the real number
WASTED TOKENS0on unsafe runs
MEASURED, NOT ASSERTED

The guardrail,
benchmarked.

A committed corpus is replayed through the same deterministic evaluator used by the live gate. Unsafe trajectories must be caught, safe trajectories must pass, and every unsafe reason must match the expected failure.

API run also emits a benchmark span into SigNoz
01 / DECISION COVERAGE
--

labelled trajectories
replayed end to end

-- unsafeadversarial failures
-- safehealthy edge cases
02 / CATCH UNSAFE--
Unsafe recallUnsafe trajectories correctly flagged
03 / LEAVE SAFE ALONE--
Safe-run pass rateSafe trajectories allowed
04 / EXPLAIN THE BLOCK--
Reason-code accuracyExpected reason matched on unsafe cases
05 / DECIDE BEFORE EXECUTION--
Synchronous gate P50Evaluator decision only. Not model latency.
evaluator pendingLoading measured results…
python -m app.benchmark
01 / CONTRACT LIBRARY

Small rules.
Hard guarantees.

Deterministic contracts evaluate ordered tool outcomes, not merely whether a tool name appeared. Detection is automatic. Blocking always requires a human promotion.

02 / LIVE ATTACK LAB

Watch policy become
runtime protection.

This is a real local Qwen tool loop, not a scripted trace. Choose a scenario, run it, then force an evaluation pass through SigNoz.

01Agentexecute
02OpenTelemetryinstrument
03SigNozrecord
04Preflight evaluatorevaluate
05Contract decisiondecide
06Agent gateenforce
Evidence planeOpen SigNoz
SELECTED · HEALTHY REMEDIATION
PD
PagerDutyINC-4471 · orders-replica-2 · on-call queue
INBOUND

Incident INC-4471 is open on orders-replica-2. Please investigate and remediate.

Ready. A healthy episode establishes the baseline; attack episodes prove the contracts.

Optional · resilience test

Separate from running an episode: this asks what happens when the guardrail itself is unreachable. Reads fail open so the agent keeps working; destructive actions fail closed.

03 / TRACE-LINKED EVIDENCE

Every finding has
a receipt.

Violations carry the contract version, precise offending call, trace ID, evaluator version, token waste and a checksummed evidence bundle.

RECENT FINDINGS0 SHOWN

No findings yet

Run an attack episode, then refresh the engine.

LIVE OPERATIONS

SigNoz is the evidence plane.

Preflight reads completed trajectories from SigNoz, evaluates them, and writes violations back as traces, metrics and structured logs.

RUNS EVALUATED0
LAST PASS--
PASS TIME--
SANDBOX ACTIONS0
RECENT EPISODES · HISTORICAL, READ BACK FROM SIGNOZ

The agent's flight recorder

RunScenarioTrajectoryTokensVerdictDate / time
04 / HOW IT WORKS

Telemetry in.Policy out.

The observability system is load-bearing: it is the source for detection, the home of the evidence, and the operational surface for alerts and investigation.

01EXECUTION

Agent

Local Qwen chooses and executes tools.

02INSTRUMENTATION

OpenTelemetry

GenAI spans plus ordered outcome logs.

03EVIDENCE PLANE

SigNoz

Traces, logs, metrics, alerts and dashboards.

04POLICY ENGINE

Preflight

Evaluate, replay, approve and gate.

Questions, answered

Does an LLM decide whether a critical contract was broken?+

No. Types A, B and C are deterministic state machines over ordered tool calls, arguments and real outcomes. The sampled semantic judge is advisory and can never gate.

Can telemetry itself block an unsafe action?+

No. SigNoz is the evidence plane. An explicitly approved deterministic contract compiles into a synchronous pre-tool check. This separation is intentional and visible.

How do you know a new guardrail will not break healthy traffic?+

Shadow replay walks the contract through historical trajectories and reports the first call it would have stopped, including any healthy runs affected. Promotion is blocked when the replay is unsafe.

What survives a restart?+

Contract enforcement modes, approvals, evaluated runs, violations and checksummed evidence live in SQLite. A restart cannot silently disarm an approved safety control.