INSTALL

pip install faultgate
pip install 'faultgate[laya]'   # local judge: downloads Laya once (~800 MB, Apache-2.0, pinned revision), offline after
pip install 'faultgate[api]'    # any API model through litellm
pip install 'faultgate[pii]'    # Microsoft Presidio for --redact presidio (names, locations; MIT)

Requires Python 3.11 or newer. One dependency (pydantic). The built-in checks are code — nothing to download.

PyPI Repository Issues

0.09 → 0.98

SEARCH_LOOP KAPPA, LOCAL JUDGE → RULE (N = 200)

CHECKS (6 RULE · 2 GOLDEN · 1 JUDGE)

DEPENDENCY (PYDANTIC)

MIT

LICENSE

01 / WHAT'S NEW IN 0.3.0

Business rules, hard blocks, significance testing, vendor trace import, and redaction.

Business rules

Define numeric limits, authenticated identity, and tool prerequisite orders in policy.json.

{
  "limits":   [{"tool": "issue_refund", "arg": "amount", "min": 0, "max": 500}],
  "identity": {"arg": "customer_id", "session_attr": "enduser.id", "tools": ["issue_refund", "modify_billing_record"]},
  "requires": [{"tool": "modify_billing_record", "after": "verify_customer_identity", "result_contains": "VERIFIED"}]
}

Hard block

With --band, any policy violation fails the gate regardless of clean rate, naming the offending step and values.

step 4: modify_billing_record.customer_id 'C-999' != session enduser.id 'C-123'

INCONCLUSIVE + sample size

Between PASS and FAIL thresholds, runs a one-sided significance test and sizes the needed scenarios at power 0.80.

gate: INCONCLUSIVE · clean rate 0.93 vs band [1.00, 1.00] from 2 baseline runs · fail below 0.90 · p=0.072
inconclusive: a drop of 0.07 needs 89 scenarios to tell from noise (alpha 0.05, power 0.80) · run 59 more

Two failures injected into a clean baseline run; tests/test_band.py.

Reads your existing traces

Automatically maps vendor exports from Langfuse, LangSmith, Braintrust, and Phoenix onto the GenAI span model.

--format otlp: OTLP/JSON, GenAI semantic conventions
--format openinference: Arize Phoenix spans
--format langfuse: trace objects with observations
--format langsmith: run list with tool errors
--format braintrust: logged spans with type: tool

precheck

Validates export integrity, span IDs, parent linkages, and timestamps before gating.

precheck: OK · format langfuse · checked 1 of 1 traces · 0 error(s), 0 warning(s)

Redaction

Masks PII and credentials in API judge prompts, terminal reasons, CI comments, and JSON reports using stdlib regex.

step 2: update_profile.email '<EMAIL>' != session enduser.id '<EMAIL>'

81 emails, 39 phone numbers, 69 JWTs found in 30 real benchmark traces.

PR comment + audit pack

Upserts one in-place PR comment and packages a verifiable audit artifact without copying trace payloads.

--markdown PATH: verdict, gate numbers, fired checks, failing traces, repro command
--audit DIR: verdict.json (with input SHA-256s), failing_traces.json, repro.sh

context_overflow

Detects when a run hit the model context window or provider output token limit (0 FP on 125 real traces).

gen_ai.response.finish_reasons: length / max_tokens or provider context limit error

02 / HOW THE GATE DECIDES

Significance testing against your measured baseline band instead of arbitrary coin-flip thresholds.

Agents are not deterministic. Between the PASS and FAIL thresholds, faultgate tests whether the drop is statistically significant against the baseline mean. The variance accounts for binomial sampling noise and measured run-to-run spread:

Verdict	Rule (n = scenarios in the band)	Exit
PASS	clean rate ≥ min observed − 1/n	`0`
FAIL	clean rate < min observed − 3/n, or below PASS and the drop is significant (one-sided, α = 0.05)	`1`
INCONCLUSIVE	in between and not significant: the gate says how many scenarios it would need (power 0.80) · run N more	`0`

A policy violation is a hard block. With --band, a run whose clean rate sits inside the measured band still FAILs if the policy check fired on any trace (exit 1); the gate reports policy violation in N trace(s) (hard block). Noise explains a flaky answer, not an unauthorized tool call or parameter breach.

$ faultgate check examples/gate/candidate_gpt.json --band band.json
18/30 traces failed · search_loop=18, abstention=0, policy=0, repeated_call=0, unrecovered_tool_error=0, context_overflow=0 · judge none · 0.0s
gate: FAIL · clean rate 0.40 vs band [0.93, 0.93] from 2 baseline runs · fail below 0.83 · p=0.000
regressions (clean in every baseline, not now): r-0208, r-0214, r-0216, r-0224, r-0226, r-0232, r-0258, ...

$ faultgate check examples/gate/baseline_glm_run2.json --band band.json
gate: PASS · clean rate 0.93 vs band [0.93, 0.93] from 2 baseline runs · fail below 0.83 · p=0.500

Real runs: two z-ai/glm-5.3-flash baselines and openai/gpt-5.6-luna on 30 multi-hop scenarios. Regressions clean in every baseline and broken now are named explicitly.

03 / WHAT IT CHECKS

Nine checks across structural rules, golden answers, and calibrated judges.

Check	Kind	Fires when
`search_loop`	rule	The run ended with no answer at its step budget and most tool calls returned nothing.
`abstention`	rule	The final answer explicitly says the information could not be found.
`policy`	rule	A tool call broke the policy: unlisted tool, denied argument pattern, oversized argument, call budget, argument limit, identity mismatch, or missing prerequisite.
`repeated_call`	rule	The same tool was called with identical arguments and returned identical results >= 3 times.
`unrecovered_tool_error`	rule	A tool call produced a structural error with no later successful call of the same tool, or a span errored (error.type or status code 2) and no later span made a successful tool call.
`context_overflow`	rule	The run hit the model's context window or output token limit: a span's gen_ai.response.finish_reasons says length / max_tokens, or a provider error naming the context limit (context_length_exceeded, "maximum context length", "prompt is too long") in a span status, error.type, tool result or faultgate.termination.
`wrong_answer`	golden	Answered, and the expected answer is not in the final answer.
`truncation`	golden	The answer has the head of the expected token but not the whole token (Onyx for Onyx-4413).
`wrong_direction`	judge	On a multi-hop task, the agent walked the dependency chain backwards.

A rule is plain code over trace structure: instant, deterministic, no model. A golden check is a rule requiring expected answers (--golden golden.json). A judge check is an opt-in question scored by a local model or API judge.

04 / BUSINESS RULES

Domain invariants on tool arguments, authentication, and prerequisite call sequences.

Pass --policy policy.json to enforce business constraints on what the agent did, not just which tool names it touched:

{
  "limits":   [{"tool": "issue_refund", "arg": "amount", "min": 0, "max": 500}],
  "identity": {"arg": "customer_id", "session_attr": "enduser.id", "tools": ["issue_refund", "modify_billing_record"]},
  "requires": [{"tool": "modify_billing_record", "after": "verify_customer_identity", "result_contains": "VERIFIED"}]
}

limits: Numeric floor or ceiling on a tool argument (dotted paths like refund.amount). Strings like "$1,200.50" are parsed as numbers. Non-numeric or non-object values violate; absent arguments pass.
identity: Argument must match the authenticated principal from root span attribute session_attr (default enduser.id). Unverifiable calls without this attribute fail closed.
requires: Tool may only execute after an earlier call to after, optionally requiring its returned payload to contain specified text (result_contains).

See a full customer-support policy file in examples/policy_support.json →.

05 / TRACE FORMATS

Native OTLP/JSON and zero-config vendor export normalization.

faultgate reads exports from common agent-observability tools and maps them onto the same span attributes so every check, policy, and judge runs without alterations. Format is auto-detected or specified via --format.

--format	Export	Steps come from
`otlp`	OTLP/JSON, GenAI semantic conventions	execute_tool spans and tool calls inside chat messages
`openinference`	OTLP with openinference.span.kind (Arize Phoenix)	TOOL spans: tool.name, tool.parameters or input.value, output.value
`langfuse`	trace objects with observations (GET /api/public/traces/{id})	TOOL observations; level: ERROR + statusMessage become span errors
`langsmith`	a list of runs	run_type: tool runs; error becomes a tool error
`braintrust`	a list of logged spans	span_attributes.type: tool spans

06 / DATA STAYS LOCAL: REDACTION

In-memory trace evaluation with built-in regex redaction for all outgoing text.

faultgate reads traces in memory and never writes payloads to disk or remote servers. When sending trace text to API judges or rendering failure reasons in CI comments and JSON reports, credentials and PII are masked by default:

--redact	Masks	Needs
`builtin (default)`	emails, credit cards (Luhn-checked), US SSNs, phone numbers, IPv4/IPv6, API keys, bearer tokens, JWTs	nothing (stdlib regex)
`presidio`	the above plus Presidio's recognisers: person names, locations, more	pip install 'faultgate[pii]' and a spaCy model
`off`	nothing	none

Rules always see raw values to evaluate business logic accurately; only exported summaries are masked: step 2: update_profile.email '<EMAIL>' != session enduser.id '<EMAIL>'.

07 / IN CI

GitHub Action release gating with PR comments and reproducible audit packs.

- uses: Faultline-lab/faultgate@main
  with:
    traces: traces/candidate.json      # exported by your agent's OTel pipeline
    band: traces/band.json             # from `faultgate baseline`, committed
    golden: traces/golden.json         # optional
    policy: traces/policy.json         # optional
    format: auto                       # auto, otlp, langfuse, langsmith, braintrust, openinference
    redact: builtin                    # builtin, presidio, off
    comment: "true"                    # upsert PR comment (needs pull-requests: write)

Job summary: Writes full Markdown report with p-values, broken rules, and latency/cost deltas to the GitHub Actions job summary.
PR comment: Upserts one PR comment in place across pushes (tracked via ). Requires permissions: pull-requests: write.
Audit artifact: Uploads faultgate-audit artifact containing verdict.json (with SHA-256 hashes of inputs), failing_traces.json, and repro.sh. No raw prompts or payloads are copied.
Step output: Exposes final outcome as steps.gate.outputs.verdict (PASS, FAIL, or INCONCLUSIVE).

08 / HOW MUCH TO TRUST A VERDICT

Empirical scoring against human failure-mode labels.

Every check and judge is scored against human failure-mode labels on 200 real agent runs (FAULTLINE P4), reported on all 200 and on P5's frozen test split (n = 60). Regenerate with scripts/calibrate.py.

Check	Kind	Positives	TPR	TNR	κ (n=200)	κ (P5 test, n=60)
`search_loop`	rule	36	1.00	0.99	0.98	1.00
`abstention`	rule	1	1.00	1.00	1.00	— (0 positives)
`policy`	rule	synthetic	4/4 P9 categories	0 FP on 290 real traces	—	—
`context_overflow`	rule	synthetic	finish-reason and provider-error forms in tests/test_checks.py	0 FP on the 125 shipped real traces	—	—
`wrong_answer`	golden	4	1.00	1.00	1.00	1.00
`truncation`	golden	1	1.00	1.00	1.00	— (0 positives)
`wrong_direction`	Laya judge	1	1.00	0.61	0.02	0.00

For comparison, the same search_loop question put to judges instead of a rule: Laya κ 0.09, GLM-5.3 (API) κ −0.03. Where a failure is structural, code beats every model we measured, which is why the default checks are rules.

READING THE CALIBRATION HONESTLY

search_loop is solid on this task family. The human coder saw the same structural signals, so κ measures agreement with that coding, not ground truth from nowhere.
abstention and truncation have one positive each to learn from. abstention is a phrase list; a paraphrase it doesn't know is a miss.
wrong_answer agrees with the human verdict on all 21 answered runs. It is substring containment, case-insensitive: right for short factual answers, wrong for free-text ones — say what "expected" means for your task before you trust it.
policy has no human-labelled positives: FAULTLINE's agent rejects hostile calls before they become steps, so its traces never contain an executed attack. Recall is by construction on the P9 attack shapes; the number that matters is zero false positives on real runs.
context_overflow tests whether the run aborted due to context length exhaustion. It fires on finish reasons and provider error payloads with 0 false positives across the 125 benchmark traces.
wrong_direction is the only check that needs reading, and the only local judge we have is noise on it (fires on 39% of clean runs). It ships because the contract is the product: the day a local model scores κ ≥ 0.7 on it, that's a one-line adapter, and this harness is what proves it.
Laya's context is 512 tokens. faultgate compacts the trace to fit (answer and termination first, middle steps elided) and counts every call that still overflows; the count is printed.

MEASURED IN FAULTLINE P18

Check	Kind	Measured trust	Source
`repeated_call`	rule	κ 0.95 vs a blind second rater (Claude Opus 5.5) on 97 real runs	FAULTLINE P18
`unrecovered_tool_error`	rule	Caught 13 of 23 unrecovered errors confirmed by the blind second rater (Claude Opus 5.5) (recall 0.57, κ 0.54); misses plain-text errors and run-ending API failures (faultgate has no check for those yet; read flagged traces manually)	FAULTLINE P18

Span error = error.type or OTLP status code 2; P18 measured error.type only. Benchmark traces from Exgentic/agent-llm-traces-v2.

09 / WHAT IT DOESN'T DO YET

Explicit design non-goals and deferred capabilities.

Premature stop: answered with an intermediate entity of the chain. Needs the traversal chain, not just the final answer.
Semantic injection checks: the agent obeyed text it read. P9 has the taxonomy; needs a judge that scores.
Running your agent: faultgate reads traces; it never executes anything.

10 / BUILT ON FAULTLINE

Research provenance and empirical derivation.

FAULTLINE is the research workbench behind this tool: 30 days of simulator, 16 measured projects, two papers. faultgate is the part you install. Every number in the faultgate README links back to the run that produced it.

faultgate repository Read the research →

Ship the ones you can measure.

Install faultgate for local release gating in CI, or book a Readiness Review to build your domain policy pack.

pip install faultgate Book a Readiness Review →See pricing