Gate & Statistics

Build a band and gate a candidate

Agents are not deterministic, so faultgate measures your spread before it judges a candidate. Record three or more baseline runs of the same agent, build a band from them, then gate each candidate against that band:

$ faultgate baseline examples/gate/baseline_glm_run1.json examples/gate/baseline_glm_run2.json --out band.json
note: 2 baseline runs measure noise poorly; use 3 or more
examples/gate/baseline_glm_run1.json: 30 traces · clean rate 0.93 · 0.0s
examples/gate/baseline_glm_run2.json: 30 traces · clean rate 0.93 · 0.0s
band [0.93, 0.93] over 30 scenarios × 2 runs · 26 clean in every run · written to band.json

$ faultgate check examples/gate/candidate_gpt.json --band band.json
18/30 traces failed · search_loop=18, abstention=0, policy=0, repeated_call=0, unrecovered_tool_error=0, context_overflow=0 · judge none · 0.0s
gate: FAIL · clean rate 0.40 vs band [0.93, 0.93] from 2 baseline runs · fail below 0.83 · p=0.000
regressions (clean in every baseline, not now): r-0208, r-0214, r-0216, r-0224, r-0226, r-0232, r-0258, ...

$ faultgate check examples/gate/candidate_qwen.json --band band.json
gate: FAIL · clean rate 0.70 vs band [0.93, 0.93] from 2 baseline runs · fail below 0.83 · p=0.005

$ faultgate check examples/gate/baseline_glm_run2.json --band band.json
gate: PASS · clean rate 0.93 vs band [0.93, 0.93] from 2 baseline runs · fail below 0.83 · p=0.500

These are real runs (table rows and most stderr notes omitted): two z-ai/glm-5.3-flash baselines, and openai/gpt-5.6-luna and qwen/qwen3.7-flash candidates, on the same 30 multi-hop scenarios. The verdicts match FAULTLINE P11’s live drill (R4 FAIL, R1 FAIL, R2 PASS). That drill cost $1.84 and an API judge. This one took 0.0 s and no model.

Verdicts

Between the PASS and FAIL thresholds, the gate tests whether the candidate’s clean rate is below the baseline mean. The variance is the binomial noise of both sides plus the run-to-run variance the baseline measured, so a noisy baseline needs a bigger drop to fail. When the gate cannot tell, it says so and sizes the fix.

Verdict	Rule (n = scenarios in the band)	Exit
PASS	clean rate ≥ min observed − 1/n	0
FAIL	clean rate < min observed − 3/n, or below PASS and the drop is significant (one-sided, α = 0.05)	1
INCONCLUSIVE	in between and not significant: the gate says how many scenarios it would need (power 0.80)	0

An inconclusive result looks like this:

gate: INCONCLUSIVE · clean rate 0.93 vs band [1.00, 1.00] from 2 baseline runs · fail below 0.90 · p=0.072
inconclusive: a drop of 0.07 needs 89 scenarios to tell from noise (alpha 0.05, power 0.80) · run 59 more

That example used two failures injected into a run of a baseline that was clean on all 30 scenarios (tests/test_band.py). required_n is the standard two-proportion sample size for the observed drop. The gate result in the JSON report carries the same numbers under stats.

Build the band from 3 or more baseline runs. With 2, the run-to-run variance is a guess, and the gate warns.

Grade correctness

With expected answers, the gate grades correctness too:

$ faultgate baseline examples/gate/baseline_glm_run*.json --golden examples/gate/golden.json --out band.json
examples/gate/baseline_glm_run1.json: 30 traces · clean rate 0.80 · 0.0s      # P11: 24/30 grounded
examples/gate/baseline_glm_run2.json: 30 traces · clean rate 0.87 · 0.0s      # P11: 26/30 grounded
band [0.80, 0.87] over 30 scenarios × 2 runs · 21 clean in every run

$ faultgate check examples/gate/candidate_gpt.json --golden examples/gate/golden.json --band band.json
21/30 traces failed · search_loop=18, abstention=0, policy=0, repeated_call=0, unrecovered_tool_error=0, context_overflow=0, wrong_answer=3, truncation=3 · judge none · 0.0s
gate: FAIL · clean rate 0.30 vs band [0.80, 0.87] from 2 baseline runs · fail below 0.70 · p=0.000

$ faultgate check examples/gate/candidate_qwen.json --golden examples/gate/golden.json --band band.json
18/30 traces failed · search_loop=9, abstention=0, policy=0, repeated_call=1, unrecovered_tool_error=0, context_overflow=0, wrong_answer=9, truncation=8 · judge none · 0.0s
gate: FAIL · clean rate 0.40 vs band [0.80, 0.87] from 2 baseline runs · fail below 0.70 · p=0.000

Qwen returns the head of the answer token without its suffix in 8 of 30 runs. Nobody had noticed.

For the flags, see the CLI Reference. For the policy hard block that applies on top of the band, see Business Rules.