faultgate 0.3.1 · fixes from the first outside test, 180 real tau2-bench runsRead the changelog →
THE PLATFORM

One loop: evaluate, gate, guard.

Three open-source tools that make every decision about a non-deterministic agent repeatable.

01 / CONTINUOUS LOOP

From lab simulation to CI gating to runtime interception.
ONE CONTINUOUS LOOP · EVALUATE → GATE → GUARD
LOOP
Evaluate01 / WORKBENCH

Measure the agent over repeated runs with a deterministic oracle

1,350 graded runs · deterministic oracle, no LLM judge
Gate02 / RELEASE GATE

Turn test runs into rule-checked release decisions in CI

PASS / FAIL / INCONCLUSIVE in CI
Guard03 / RUNTIME FIREWALL

Check tool calls at runtime against pinned tool definitions

Flags poisoned tool descriptions · blocks unknown tools and rug-pulls in enforce mode
CLOSED LOOP: EVALUATE IN LAB → GATE IN CI → GUARD AT RUNTIME → FEED BACK

02 / THREE PILLARS

Empirical facts from our research and open-source tooling.
01 / EVALUATE

FAULTLINE Evals

The open-source research workbench for multi-trial reliability and fault-injection evaluations.

Evaluate autonomous agents under repeated execution before shipping. Turn run-to-run variance into empirical noise bands with a deterministic oracle.

Repeated trials

1,350 graded runs across 150 held-out hard Tier-3 scenarios at pinned temperature 0.0.

Deterministic dual-condition oracle

Evaluates normalized string answers and verified citation hashes against a frozen corpus. Zero LLM judges in the evaluation loop.

Seeded fault injection

Reproducible catalog (F1–F6) covering schema violations, synthetic latency, payload corruption, and infinite tool loops.

Pre-registered hypotheses

Master Experiment Contract (MEC.md) and all primary hypotheses committed to git prior to execution.

Paired McNemar statistics

Rigorous 2×2 contingency table significance testing (McNemar exact p = 5.61 × 10⁻⁹³ across 1,800 paired runs).

pass^k joint reliability

Empirical test of naive compounding shows joint reliability holds within 2.5 percentage points at k=3.

Explore FAULTLINE Evals
02 / GATE

faultgate

A local-first release gate for AI agents in CI.

Reads your agent’s recorded traces, measures run-to-run noise, checks for structural failure modes, and exits non-zero only when a candidate is a real regression.

Empirical noise bands

Measures candidate runs against an empirical spread from 3+ baseline runs instead of a coin-flip fixed threshold.

Significance test + INCONCLUSIVE

Runs a one-sided significance test (α = 0.05) vs baseline mean. When underpowered, outputs exact required sample size.

Business rules as hard block

Enforces limits (numeric floor/ceiling), identity (enduser.id verification), requires, same_arg, and max_calls_per_arg.

Vendor trace import

Auto-detects and normalizes traces from OTLP/JSON GenAI semconv, Langfuse, LangSmith, Braintrust, and Arize Phoenix.

PR comment + audit pack

In-place GitHub PR comments, job summaries, and audit artifacts (verdict.json with input SHA-256 sums, repro.sh).

Explore faultgate
03 / GUARD

mcp-guard

A local firewall for Model Context Protocol (MCP) tool calls.

Sits between your AI client and MCP servers: logs every call, flags poisoned tool descriptions, and blocks rule violations in enforce mode.

Pins tool definitions on first run

Records tool schema and description hashes to detect silent modifications or rug-pull attacks by rogue servers.

Flags poisoned descriptions

Scans tool manifests for prompt injection heuristics (hidden instructions, exfiltration targets, sensitive paths).

Blocks rule violations in enforce mode

Returns MCP error responses on unknown_tool, extra_argument, and tool_changed without touching tool backends.

Logs argument names never values

Audit log (~/.mcp-guard/audit.jsonl) records call structure and rule outcomes with zero customer payload leaks.

Audit mode vs enforce mode

Runs in non-disruptive audit mode by default; switch to enforce mode with a single CLI command.

Explore mcp-guard

03 / HOW TO ENGAGE

Use the open-source tools yourself, or run engagements with us.

Open source or with us.

The platform is open source under the MIT licence. You can adopt and run every tool yourself, or bring us in to design your evaluation suite, red-team your tools, or maintain your gates.

DimensionOpen source (free)Services
Core toolsAll 3 tools MIT licensed (FAULTLINE, faultgate, mcp-integrity). Free forever.Installed, configured, and integrated directly into your environment.
Eval suite designSelf-serve using the FAULTLINE workbench schemas and sample scenarios.Custom scenario build from your agent’s real tasks, with deterministic golden oracles.
Red-team testingRun the public MCPTox benchmarks and document-text injection suites locally.Full attack battery modelled on Pub 02 and P9, with paired confidence interval reports.
Ongoing re-baseliningRun faultgate baseline on candidate runs whenever you change models or prompts.Reliability retainer coverage: model-swap migration checks and rule-pack maintenance.
Engineering supportPublic GitHub issues, documentation, and community discussions.Priority support channel.

Models will stay non-deterministic. Our job is to make every decision about them deterministic.

Samir Sawarkar, Founder
START REPRODUCIBLE EVALS

Make your agent’s next release a deterministic decision.

Explore our services for custom evals, red-teams, and reviews, or install the CLI for free today.

Talk to the founderAll services →