Secure by default, because it runs in your CI.
faultgate is a CLI and a GitHub Action. There is no FAULTLINE server in the path of your traces.
01 / WHAT STAYS, WHAT CAN LEAVE
Stays in your CI runner
- Traces read in memory: faultgate reads your traces in memory and writes no trace payloads anywhere.
- Rules, policy & bands: Deterministic checks, policy rules, and baseline noise calculations run entirely locally.
- Local Laya judge: Runs offline on your runner; no trace content leaves the machine.
- Zero trace payloads written anywhere: The audit pack copies no prompts, answers or tool payloads.
Can leave your CI runner
- API judge prompts: When you explicitly pass
--judge litellm:..., the rendered trace goes to your configured model provider. Masked by default. - Failure reasons: Human-readable reasons printed in your terminal, written to
--json, emitted into CI logs, and posted to PR comments. Masked by default. - Masked by default: Both outbound streams are masked for the patterns listed below (builtin), or Presidio's recognisers if you enable them.
02 / REDACTION
faultgate reads your traces in memory and writes no trace payloads anywhere. Rules, policy, bands and the local Laya judge never send anything off the machine. Two things can travel: the trace an API judge (litellm:) reads, and the reasons that end up in your terminal, --json report, CI logs and PR comments. By default both are masked:
--redact | Masks | Needs |
|---|---|---|
builtin (default) | emails, credit cards (Luhn-checked), US SSNs, phone numbers, IPv4/IPv6, API keys, bearer tokens, JWTs | nothing (stdlib regex) |
presidio | the above plus Presidio's recognisers: person names, locations, more | pip install 'faultgate[pii]' and a spaCy model |
off | nothing | — |
MASKED REASON EXAMPLE
step 2: update_profile.email '<EMAIL>' != session enduser.id '<EMAIL>'Rules always see the raw values; only what leaves is masked. An identity policy compares the real customer id in memory, while terminal outputs, JSON reports, and PR comments redact secrets and identifiers. On the 30 real benchmark traces from make demo-real the built-in masker finds 81 emails, 39 phone numbers and 69 JWT access tokens, all real-looking customer data or credentials; it changes none of the text in the shipped FAULTLINE traces.
03 / FIXED IN 0.3.1
Redaction digit-run boundary fix
A card number written next to another number (4111 1111 1111 1111 123-45-6789) joined its digit run, failed the Luhn check as a whole and was written unmasked to the JSON report, the Markdown summary and failing_traces.json. Cards are now found as Luhn-valid windows of whole digit groups, SSNs are masked first, and a card at the end of a sentence (...1111.) is masked too.
Found in the first outside test (tau2-bench retail, 180 real runs).
pip install -U faultgate04 / AUDIT TRAIL
--audit writes verdict.json, which records the SHA-256 of every input file, plus failing_traces.json and repro.sh. No prompts, answers or tool payloads are copied.
verdict.jsonVerdict, exit code, faultgate version, UTC timestamp, every input file with its SHA-256, and gate numbers.
failing_traces.jsonEach failing trace and every reason that fired, with PII and credentials masked by redact.
repro.shExact shell command to reproduce the evaluation and verdict locally against the same inputs.
--audit writes verdict.json (inputs with SHA-256), failing_traces.json, repro.sh; no prompts, answers or tool payloads copied.
05 / SUPPLY CHAIN
Permissive open-source license. Read, modify, and audit every line of code.
Core rules run on Python standard library with no external model weights or network downloads.
Heavyweight judges and recognizers are opt-in extras (faultgate[laya], faultgate[api], faultgate[pii]).
Laya judge downloads once at a pinned revision (~800 MB, Apache-2.0), running completely offline after.
Published on PyPI.
06 / COMPLIANCE
No SOC 2 report yet. faultgate runs in your environment, so your own controls and audits apply.
There is no FAULTLINE cloud backend, hosted data sink, or multitenant database ingesting your agent runs. Because the evaluation gate runs entirely within your existing CI runners or private infrastructure, your organization’s current access controls, network boundaries, and compliance programs remain the governing authority.
07 / REPORT A VULNERABILITY
We welcome responsible security research. If you find a security vulnerability or suspect a boundary violation in faultgate or FAULTLINE tools, please review our security policy and notify us directly: