Picture the same model running the same thirty inputs five times. In FAULTLINE P11, I recorded pass rates from 0.60 to 0.87, a twenty-seven point swing. Same model, same inputs.
A hard threshold turns that variance into a coin flip. If your release gate demands 0.80 clean runs, one run passes at 0.87 and another blocks at 0.60. You end up chasing ghosts. I built faultgate so the gate measures the noise first.
regression or noise
The tool is a local-first release gate for AI agents. Hand it recorded traces, and it parses the structure, checks every run for known failure modes, measures run-to-run noise, and exits non-zero only when a candidate run shows a real regression. The core engine runs offline. It requires Python 3.11 or newer and has one dependency, pydantic. Default checks run in plain code with no model required.
Here is the raw unedited output on five real runs of openai/gpt-5.6-luna on a multi-hop retrieval task, recorded by FAULTLINE.
$ faultgate check examples/traces.json
trace search_loop abstention policy repeated_call unrecovered_tool_error
------------------------------------------------------------------------------------------------------------
fc2cdab304bd FAIL 1.00 ok 1.00 ok 1.00 ok 1.00 ok 1.00
6d1f9d67a2d4 ok 1.00 FAIL 1.00 ok 1.00 ok 1.00 ok 1.00
8bf3879339eb ok 1.00 ok 1.00 ok 1.00 ok 1.00 ok 1.00
3a9e507a0f6d ok 1.00 ok 1.00 ok 1.00 ok 1.00 ok 1.00
2ab3b6c1e891 ok 1.00 ok 1.00 ok 1.00 ok 1.00 ok 1.00
2/5 traces failed · search_loop=1, abstention=1, policy=0, repeated_call=0, unrecovered_tool_error=0 · judge none · 0.0s
$ echo $?
1Trace 1 looped for 12 steps and never answered. Trace 2 said it could not find the information. Traces 3 through 5 answered cleanly. The terminal exit code was 1 because two runs failed structural checks.
the noise baseline
To separate genuine regressions from normal serving drift, faultgate measures your spread across baseline runs. You supply two or more baseline trace files, generate a tolerance band, and grade candidates against that spread.
$ faultgate baseline examples/gate/baseline_glm_run1.json examples/gate/baseline_glm_run2.json --out band.json
examples/gate/baseline_glm_run1.json: 30 traces · clean rate 0.93 · 0.0s
examples/gate/baseline_glm_run2.json: 30 traces · clean rate 0.93 · 0.0s
band [0.93, 0.93] over 30 scenarios × 2 runs · 26 clean in every run · written to band.json
$ faultgate check examples/gate/candidate_gpt.json --band band.json
18/30 traces failed · search_loop=18, abstention=0 · judge none · 0.0s
gate: FAIL · clean rate 0.40 vs band [0.93, 0.93] from 2 baseline runs · fail below 0.83
regressions (clean in every baseline, not now): r-0208, r-0214, r-0216, r-0224, r-0226, r-0232, r-0258, ...
$ faultgate check examples/gate/candidate_qwen.json --band band.json
gate: FAIL · clean rate 0.70 vs band [0.93, 0.93] from 2 baseline runs · fail below 0.83
$ faultgate check examples/gate/baseline_glm_run2.json --band band.json
gate: PASS · clean rate 0.93 vs band [0.93, 0.93] from 2 baseline runs · fail below 0.83The verdict rules calculate tolerances from the number of scenarios in the band, written as n. A candidate earns a PASS with exit code 0 when clean rate meets or exceeds the minimum observed baseline minus 1/n. A candidate earns a WARN with exit code 0 when clean rate falls between minus 1/n and minus 3/n, meaning an engineer should inspect the trace artifacts. A candidate earns a FAIL with exit code 1 when clean rate drops below the minimum observed baseline minus 3/n.
When a candidate fails, faultgate lists regressions by scenario identifier. These are scenarios that were clean in every baseline run and broke in the candidate. You read those first. In the live drill for FAULTLINE P11, running candidate models openai/gpt-5.6-luna and qwen/qwen3.7-flash against a z-ai/glm-5.3-flash baseline through an API judge cost $1.84. In faultgate, evaluating the identical thirty scenarios took 0.0 seconds and zero model spend.
trusting code over models
I calibrated the original checks against human failure-mode labels on 200 real agent runs from FAULTLINE P4, reporting on all 200 and on the P5 frozen test split of 60 runs. That calibration covers search_loop, abstention, wrong_answer, truncation, and wrong_direction, while policy is synthetic against attack shapes. The two new checks in v0.2.0, repeated_call and unrecovered_tool_error, were measured separately in FAULTLINE P18 against a blind second rater on public benchmark traces rather than production traffic. On those 97 real runs repeated_call reached a kappa of 0.95, while unrecovered_tool_error caught 13 of 23 unrecovered errors confirmed by the blind second rater for a recall of 0.57. It misses plain-text errors and run-ending API failures, so flagged traces still need a manual read.
The search_loop rule scored a Cohen kappa of 0.98 on 200 runs and 1.00 on the test split. When I put that exact same question to model judges, Laya scored a kappa of 0.09 and GLM-5.3 scored minus 0.03. Where a failure is structural, plain code beats every model I measured. Code wins on structure.
The wrong_direction check asks whether an agent walked a multi-hop dependency chain backwards. Our only local judge for that question is Laya, which fires on 39% of clean runs and lands at a kappa of 0.02. That judge is noise. I shipped it anyway because an explicit verification contract is the product. The day a local model scores kappa 0.70 or higher on it, it is a one-line adapter, and the calibration harness proves it.
The mechanical trade-off is stark. Deterministic rules are fast and free, but they cannot parse semantic intent. Model judges can read semantics, but on a question a rule answers at kappa 0.98, the judges I measured scored 0.09 and minus 0.03.
what is missing
I keep boundaries explicit. The current release does not catch premature stop errors where an agent answered with an intermediate entity along the chain, because that needs the full traversal chain beyond the final answer. It does not check semantic prompt injections where an agent followed malicious text inside an external document, because that requires a calibrated judge that scores reliably. It does not run your agent. You run your agent in your own environment and pipe the resulting traces into faultgate.
You can install the package now with pip install faultgate. A GitHub Action runs it in CI. The job fails on gate FAIL, WARN and PASS pass, and the report lands in faultgate-report.json for upload as an artifact. Explore the documentation on faultgate and inspect the source code on GitHub. The toolkit is built on FAULTLINE, and every number in the faultgate README links back to the run that produced it.