faultgate 0.3.0: a refund over the limit fails, even inside the noise

Picture a support agent that issues a $1,200 refund against a $500 limit, and on another ticket edits billing for customer C-999 while the session belongs to C-123. Nothing loops and nothing abstains, so the clean rate stays inside your noise band.

In 0.2.0 that candidate passed, because no rule read tool arguments. That verdict was wrong. Noise explains a flaky answer, not a refund over the limit. Version 0.3.0 makes business rules a hard block. When an agent violates a business rule, aggregate statistics become irrelevant. The build must fail immediately.

business rules on tool arguments

Earlier releases checked unlisted tools or infinite loops. Version 0.3.0 inspects tool arguments. You declare business constraints inside policy.json using limits, identity, and requires.

{
 "limits":   [{"tool": "issue_refund", "arg": "amount", "min": 0, "max": 500}],
 "identity": {"arg": "customer_id", "session_attr": "enduser.id", "tools": ["issue_refund", "modify_billing_record"]},
 "requires": [{"tool": "modify_billing_record", "after": "verify_customer_identity", "result_contains": "VERIFIED"}]
}

The limits key sets numeric floors and ceilings on arguments. Dotted paths like refund.amount work, and strings like "$1,200.50" parse into numbers. Present non-numeric arguments fail. The identity key binds arguments to root span attribute enduser.id. If an agent edits records for another customer, faultgate flags the step. The requires key enforces call order. Billing updates cannot run until verification succeeds.

When checks fail, faultgate prints the offending step:

step 4: modify_billing_record.customer_id 'C-999' != session enduser.id 'C-123'

Policy violations are a hard block. With --band, a run inside the tolerance band still fails if any trace triggers a policy rule. The gate prints policy violation in N trace(s) (hard block) and records them under policy_violations in the report.

Rules do not guess. That determinism carries a cost. Checks fail closed on missing data. If your exporter drops root span attribute enduser.id, faultgate cannot verify identity and marks checked calls as violations. You get false failures until your exporter preserves required session attributes. That is the price of verification.

an honest inconclusive

In version 0.2.0, runs between pass and fail thresholds earned WARN. That verdict was evasive. It told teams to inspect artifacts without stating whether a drop was real.

Version 0.3.0 replaces WARN with INCONCLUSIVE. Between PASS and FAIL thresholds, faultgate runs a one-sided test against the baseline mean. Variance combines binomial sampling noise from both sides with measured run-to-run baseline spread. A noisy baseline demands a larger drop to fail. Significant drops at alpha 0.05 trigger FAIL with exit code 1. Unresolved drops return INCONCLUSIVE with exit code 0.

gate: INCONCLUSIVE · clean rate 0.93 vs band [1.00, 1.00] from 2 baseline runs · fail below 0.90 · p=0.072
inconclusive: a drop of 0.07 needs 89 scenarios to tell from noise (alpha 0.05, power 0.80) · run 59 more

This example comes from tests/test_band.py, where two failures were injected into a baseline clean on thirty scenarios. The gate calculates the two-proportion sample size needed to separate a 0.07 drop from noise at alpha 0.05 and power 0.80. It tells you to run 59 more scenarios. Sizing is explicit.

Bands built from fewer than three runs measure noise poorly, so faultgate warns to use three or more runs. Existing band.json files continue working, and reports expose calculations under stats. If your scripts parsed verdict == "WARN", update them to parse "INCONCLUSIVE".

vendor formats and redaction

Release gates must read existing traces. Version 0.3.0 adds a --format flag supporting otlp, openinference for Arize Phoenix, langfuse, langsmith, and braintrust. Without the flag, faultgate detects the format automatically.

The normalizer maps vendor exports onto OpenTelemetry GenAI semantic conventions, extracting tool calls, errors, finish reasons, token counts, and user identifiers. Validate files before gating:

precheck: OK · format langfuse · checked 1 of 1 traces · 0 error(s), 0 warning(s)

Precheck flags duplicate span identifiers, inverted timestamps, and unnamed tool calls, exiting two on errors.

Everything runs locally. Rules, policy checks, baseline bands, and the local Laya judge execute offline with zero cloud telemetry. When calling API judges or posting PR comments, --redact builtin masks sensitive values using standard library regular expressions. Rules inspect raw values so customer identifiers match. Text leaving the engine receives masks:

step 2: update_profile.email '<EMAIL>' != session enduser.id '<EMAIL>'

On thirty real benchmark traces sampled in make demo-real, the built-in regular expressions found 81 emails, 39 phone numbers, and 69 JWT access tokens. On the shipped FAULTLINE traces it changes nothing.

ci and audit packs

In CI, the GitHub Action writes to the job summary and updates one pull request comment in place. The comment details the verdict, gate statistics, p-value, triggered checks, hard blocks, failing traces with steps, and reproduction commands.

Each run uploads an audit artifact named faultgate-audit. It packages verdict.json with SHA-256 hashes of all inputs, failing_traces.json, and repro.sh. Trace payloads are never copied, and reasons remain masked. The Action exposes the verdict output, and fails closed if the gate never ran.

Version 0.3.0 also adds context_overflow as a default rule. It catches runs hitting context windows or token limits through gen_ai.response.finish_reasons or provider error messages, and it fired on none of the 125 real traces shipped with faultgate.

I keep boundaries explicit. The current release omits premature stop errors where an agent answered with an intermediate entity, because that requires graph traversal beyond the final answer. It omits semantic prompt injections where an agent obeyed text inside a document, because that demands a calibrated judge. It executes no agent code. You run your agent in your own infrastructure and pipe traces into faultgate.

Install or upgrade the package:

pip install -U faultgate

Read the product overview on faultgate and review source code on GitHub. Check release history in the CHANGELOG. If your team is establishing release gates for production pipelines, review pricing for a Readiness Review.