Quickstart

Install

pip install faultgate

Requires Python 3.11 or newer. One dependency (pydantic). The built-in checks are code, so there is nothing to download. Judges are optional extras:

pip install 'faultgate[laya]'   # local judge: downloads Laya once (~800 MB, Apache-2.0, pinned revision), offline after
pip install 'faultgate[api]'    # any API model through litellm
pip install 'faultgate[pii]'    # Microsoft Presidio for --redact presidio (names, locations; MIT)

Read How much to trust a verdict before you enable a judge.

Run the built-in checks

The output below is real and unedited. It comes from five agent runs: openai/gpt-5.6-luna on a multi-hop retrieval task, recorded by FAULTLINE. Trace 1 looped for 12 steps and never answered. Trace 2 said it could not find the answer. Traces 3 to 5 answered.

$ faultgate check examples/traces.json
trace         search_loop       abstention        policy            repeated_call     unrecovered_tool_error  context_overflow
------------------------------------------------------------------------------------------------------------------------------
fc2cdab304bd  FAIL 1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00
6d1f9d67a2d4  ok   1.00         FAIL 1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00
8bf3879339eb  ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00
3a9e507a0f6d  ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00
2ab3b6c1e891  ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00         ok   1.00

2/5 traces failed · search_loop=1, abstention=1, policy=0, repeated_call=0, unrecovered_tool_error=0, context_overflow=0 · judge none · 0.0s
$ echo $?
1

Usage

faultgate check traces.json                          # built-in rule checks
faultgate check traces.json --json report.json       # full per-trace verdicts
faultgate check traces.json --golden golden.json     # expected answers: correctness + truncation
faultgate check traces.json --policy policy.json     # what the agent was allowed to do
faultgate check traces.json --judge laya             # also run judge checks, locally
faultgate check traces.json --judge litellm:openai/gpt-5.6-luna   # any API model
faultgate check traces.json --markdown pr.md --audit audit/   # PR comment + audit files
faultgate precheck traces.json                       # validate an export before gating
faultgate checks                                     # list checks and their kind
faultgate judges                                     # list judge specs

faultgate baseline run1.json run2.json run3.json --out band.json   # measure your noise (3+ runs)
faultgate check candidate.json --band band.json          # regression, or just noise?

Exit codes: 0 clean · 1 at least one check fired (or, with --band, gate FAIL) · 2 bad input, or an output file (--json, --markdown, --audit, --out) that cannot be written. Missing parent directories of output files are created.

Next, Core Concepts explains the trace model and the three kinds of checks. The CLI Reference lists every command and flag.