Business Rules (policy.json) Reference

Policy: what was the agent allowed to do?

A policy file lists what the agent may do. Start with the structural keys:

{"allow": ["search", "lookup", "calc"], "max_calls": 10, "max_arg_chars": 200}

faultgate check traces.json --policy policy.json flags every step that called a tool outside allow, passed an argument matching a denied pattern (path traversal, file://, /etc/passwd, and ~/.ssh are denied out of the box), exceeded max_arg_chars, or pushed the run past max_calls. Each reason names the step number.

The four structural injection categories from FAULTLINE P9 (tool_exfil_path, unlisted_tool, query_dump, budget_loop) are each caught by a test in tests/test_policy.py. Without a policy file, only the denied-pattern rule is active, and it fires on 0 of the 290 real traces shipped here.

Business rules

Three more keys check business rules on what the agent actually did, not just which tools it touched:

{
 "limits":   [{"tool": "issue_refund", "arg": "amount", "min": 0, "max": 500}],
 "identity": {"arg": "customer_id", "session_attr": "enduser.id", "tools": ["issue_refund", "modify_billing_record"]},
 "requires": [{"tool": "modify_billing_record", "after": "verify_customer_identity", "result_contains": "VERIFIED"}]
}

• limits: a numeric floor or ceiling on one argument of one tool. arg is a dotted path into the JSON arguments (refund.amount). Strings such as "$1,200.50" are read as numbers. An argument that is present but not a number, or arguments that are not a JSON object, count as a violation. An absent argument, or no arguments at all, does not. Arguments encoded as JSON twice are decoded. min and max must be finite numbers.
• identity: the argument must equal the authenticated principal, read from the root span attribute session_attr (default enduser.id). Only calls that carry the argument are checked, optionally only for the tools in tools. If the root span has no such attribute, the call cannot be verified and counts as a violation. That includes unreadable (non-JSON-object) arguments on a listed tool, or on any tool when no tools list is given and the argument’s name appears in the text.
• requires: tool may only run after an earlier call to after. If result_contains is set, that earlier call’s result must contain the text.

Reasons name the step and the values, for example step 4: modify_billing_record.customer_id 'C-999' != session enduser.id 'C-123'. examples/policy_support.json is a complete customer-support policy. A malformed policy file exits 2.

Order-level rules bind a call to the record it acts on:

{
 "requires": [{"tool": "get_user_details", "after": ["find_user_id_by_email", "find_user_id_by_name_zip"]},
              {"tool": "cancel_pending_order", "after": "get_order_details", "result_contains": "pending", "same_arg": "order_id"}],
 "max_calls_per_arg": [{"tools": ["exchange_delivered_order_items", "modify_pending_order_items"], "arg": "order_id", "max": 1}],
 "values": [{"tool": "cancel_pending_order", "arg": "reason", "allowed": ["no longer needed", "ordered by mistake"]}]
}

• requires[].after may be a list: any one of them satisfies the prerequisite (two ways to authenticate).
• requires[].same_arg: the earlier call must carry the same value of that argument. Reading the status of order A no longer licenses cancelling order B. A call without a readable same_arg is a violation.
• max_calls_per_arg: at most max successful calls to tools per value of arg, e.g. one item change per order. A call whose result is an error is not counted: an agent that fixes a bad item id and retries changed the order once. A call without a readable arg is a violation.
• values: the argument, when present, must be one of allowed (compared as text, trimmed).

examples/policy_retail.json applies these to the tau2-bench retail domain. On 180 real tau2 retail runs from an outside test (three gpt-6-luna baselines, a same-model candidate, a weaker model and a sabotaged prompt) it fires on 0 of the 120 baseline and same-model runs, 5 of 30 weaker-model runs and 1 of 30 sabotaged runs. Still not expressible: confirmation before a change, one tool call per turn, joins against earlier results (payment method on file, item availability, price difference), and anything semantic. A clean policy result is not a compliance certificate.

Hard block

A policy violation is a hard block. With --band, a run whose clean rate sits inside the measured band still FAILs if the policy check fired on any trace. The gate prints policy violation in N trace(s) (hard block) and lists the traces under policy_violations in the JSON report.

Noise explains a flaky answer, not a refund over the limit.

Semantic injection, where the agent obeys text it read in a document, is a judge question and not a policy rule. See Current Limits.