Core Concepts
faultgate is an offline, local-first evaluation for AI agents. You hand it the traces your agent recorded. It checks every run for known failure modes, measures your run-to-run noise, and exits non-zero only when a candidate is a real regression. The default checks need no model.
Input format
faultgate reads an OTLP/JSON export that uses the OpenTelemetry GenAI semantic conventions. One trace is one agent run:
| Span | Attributes read |
|---|---|
root invoke_agent |
gen_ai.operation.name = invoke_agent, gen_ai.agent.name, gen_ai.input.messages (the user prompt), gen_ai.output.messages (the final answer), optional faultgate.termination (why the run stopped, e.g. step cap) |
child execute_tool |
gen_ai.operation.name = execute_tool, gen_ai.tool.name, gen_ai.tool.call.arguments, gen_ai.tool.call.result |
Set gen_ai.operation.name. A span without it is still read as a tool call when its name is execute_tool or execute_tool <tool> (the semconv span name), and precheck warns that it was recognised by name only.
Legacy gen_ai.prompt and gen_ai.completion are accepted as a fallback. If your exporter records structure but not content, faultgate warns you. examples/traces.json is a complete reference file.
Checks
Every check has one of three kinds:
- Rule: plain code over the trace structure. Instant, deterministic, no model.
- Golden: a rule that also needs the expected answer (
--golden golden.json, keyed by scenario or by prompt text). - Judge: a yes/no question a model answers from the rendered trace. It only runs when you pass
--judge.
All three kinds are calibrated the same way. The full list is on the Checks Reference, and the measured accuracy is on Trust & Calibration.
Regression or noise? Tolerance bands
Agents are not deterministic. FAULTLINE P11 ran the same model on the same 30 inputs five times and got pass rates from 0.60 to 0.87. A fixed threshold turns that spread into a coin-flip gate. faultgate measures your spread instead: you record three or more baseline runs, and it builds a band from them.
Scenarios are matched across runs by faultgate.scenario, else gen_ai.agent.id, else a hash of the prompt.
faultgate also lists regressions: scenarios that were clean in every baseline run and are not clean now. Read those first. The Gate & Statistics page shows the verdicts and the sample sizes.