AgentJudge

AgentJudge interposes a second LLM — the judge — between the agent and every tool call. Before a tool runs, the judge reads the agent’s intent, the proposed call, and a rubric you define, then returns allow, deny, or ask (escalate to you). The agent stays autonomous; the judge is the guardrail.

Activate it with /policy guard and a judge_model:

/policy guard judge_model=anthropic/claude-haiku-4-5

The context bar switches to guard · anthropic/claude-haiku-4-5 and every tool call now routes through the judge before execution.

Tip

If you omit judge_model, it defaults to the session’s current model. Since the judge runs on every tool call, this is usually more expensive and slower than you want — specify a fast, cheap model like anthropic/claude-haiku-4-5 explicitly.

How the judge decides

On each tool call, the judge receives:

1. The rubric — your policy rules (what’s in scope, what’s off limits).
2. A transcript slice — a filtered view of the agent’s conversation, controlled by the transcript strategy.
3. The proposed tool call — tool name and arguments.

The judge responds with one of:

Decision	What happens
Allow	Tool call executes normally.
Deny	Tool call is blocked. By default the agent sees a [POLICY DENIED] message with the judge’s reason and replans (configurable via on_deny — see Deny behavior).
Ask	The rubric marks this capability as ASK — a permission prompt appears for the operator to approve or reject. (The judge denies when uncertain, not asks.)

Caution

By default (ask_behavior=prompt), ASK decisions wait indefinitely for an operator response — including in background and headless sessions. If no one is watching, the run stalls. For unattended sessions, set ask_behavior=deny so ASK decisions fail closed automatically:

/policy guard judge_model=anthropic/claude-haiku-4-5 preset=standard_pentest ask_behavior=deny

Defining scope

You tell the judge what’s allowed through a rubric — the rules it applies to every tool call. There are three ways to set one up, from simplest to most granular.

Presets

Named baselines that cover common pentesting postures:

/policy guard judge_model=anthropic/claude-haiku-4-5 preset=recon_only
/policy guard judge_model=anthropic/claude-haiku-4-5 preset=standard_pentest
/policy guard judge_model=anthropic/claude-haiku-4-5 preset=red_team

Preset	What it allows
recon_only	Reconnaissance and basic operations (file reads, process management, network connections). Everything else is denied — exploitation, credential access, privilege escalation, lateral movement, persistence, defense evasion, collection, exfiltration, and impact.
standard_pentest	Recon + exploitation (except custom exploits) + credential access (except spraying/brute-force/keylogging/phishing) + privilege escalation (domain escalation requires approval via ASK) + lateral movement + defense evasion (except log tampering/timestomping/indicator removal) + collection (except clipboard/email/audio). No persistence, no exfiltration beyond data staging, no impact, no file deletion.
red_team	Everything except impact (data destruction, service disruption, ransomware, defacement, resource hijacking).

Presets are rendered into natural-language rubric text that the judge reads — they aren’t magic tokens. You can preview the rendered rubric via the /policy scope modal (see below).

Freeform rubric

A plain-text string appended to the default safety-floor rules:

/policy guard judge_model=anthropic/claude-haiku-4-5 rubric=no-credential-brute-forcing,target:api.example.com

Note

The /policy command splits arguments on whitespace — rubric values containing spaces are not supported via the TUI. Use short comma-separated keywords, or set complex rubrics via the API:

POST /api/sessions/{id}/policy
{
  "policy": {
    "name": "guard",
    "judge_model": "anthropic/claude-haiku-4-5",
    "rubric": "In-scope: api.example.com and *.internal.example.com. Out of scope: production databases. No credential brute-forcing."
  }
}

The default safety rubric is always included unless you explicitly replace it. It denies:

• Destructive system operations (rm -rf, drop database, process kills outside the agent’s own)
• Credential/secret exfiltration to external destinations
• Privilege escalation without clear in-scope justification
• Network calls to destinations not implied by the task
• Writes outside the agent’s working directory that the user did not request

Note

When a scope preset is active, the rendered scope text tells the judge it overrides the default safety rules where they conflict. A preset like red_team can relax restrictions that the default rubric would otherwise deny.

To replace the default entirely (use with caution):

/policy guard judge_model=anthropic/claude-haiku-4-5 rubric=read-only,no-writes,no-deletes,no-network replace_default_rubric=true

Preset + rubric combined

A preset sets the baseline; a freeform rubric adds project-specific rules on top:

/policy guard judge_model=anthropic/claude-haiku-4-5 preset=standard_pentest rubric=target:10.0.1.0/24,avoid:10.0.1.5

Interactive scope editor (/policy scope)

For fine-grained control, open the scope modal:

/policy scope

This opens a three-view editor:

1. Preset picker — start from recon_only, standard_pentest, red_team, or a blank slate.
2. Capability editor — expand 11 capability categories (reconnaissance, exploitation, credential access, privilege escalation, lateral movement, persistence, defense evasion, collection, exfiltration, impact, operations) and toggle each subcategory to allow, deny, or ask using the a/d/s keys.
3. Rubric preview — see the scope portion of the rubric rendered as natural language. Note: this shows only the scope rules, not the default safety rubric or any freeform rubric text.

Caution

Submitting scope changes from the modal rebuilds the AgentJudge config with only the judge model and scope — any previously set rubric, transcript_strategy, on_deny, on_judge_error, and cache values are dropped. Re-apply those settings with /policy guard after using the scope modal.

Transcript strategies

The transcript strategy controls how much of the agent’s conversation the judge sees alongside the proposed tool call. More context helps the judge make better decisions but costs more tokens and increases the risk of prompt injection from tool outputs.

/policy guard judge_model=anthropic/claude-haiku-4-5 preset=standard_pentest transcript_strategy=intent_only

Strategy	What the judge sees	When to use it
rubric_only	Only the rubric and the proposed tool call. No conversation context.	Cheapest option. Works when your rubric is self-contained and the judge doesn’t need to understand the task to make a decision (e.g. blanket “no network calls”).
intent_only	System prompt + user-authored messages. No assistant turns, no tool history.	When the task prompt alone gives enough context. Good balance of cost and accuracy for well-scoped tasks.
intent_plus_calls	System prompt + user messages + prior tool calls the agent made (tool names and arguments only — no tool outputs, no assistant reasoning). This is the default.	Best general-purpose choice. The judge sees what the agent is doing without being influenced by what it’s saying or what tools returned.
intent_plus_outputs_summary	Same as above + one-sentence LLM-generated summaries of each tool’s output (each unique output is summarized once and cached for the session).	When tool outputs contain information the judge needs (e.g. “the nmap scan found port 22 open, so SSH is now a reasonable next step”). Adds a summarization call per unique tool output.
full	The entire trajectory — system prompt, user messages, assistant reasoning, tool calls, and raw tool outputs.	Maximum context. Use only when the judge needs the full picture. Highest cost and highest prompt-injection risk.

Caution

In intent_plus_calls and intent_plus_outputs_summary, assistant messages are reduced to tool-call blocks only — the agent’s reasoning text is stripped. In rubric_only and intent_only, assistant messages are excluded entirely. This is deliberate — it prevents the agent from writing persuasive justifications that talk the judge into allowing a call. The full strategy is the exception: it surfaces assistant prose unchanged, which gives the judge maximum context but also maximum prompt-injection and manipulation surface.

Note

In intent_plus_outputs_summary, tool outputs are summarized by a separate LLM call that treats tool output as untrusted data and ignores any directives embedded in it. This defends against prompt injection from tool results (e.g. a web page the agent fetched containing “ignore all previous instructions”).

Context budget and trimming

The transcript is trimmed to fit the judge model’s context window. When trimming is needed, the oldest messages are dropped first — but two messages are always preserved:

1. The system message — so the judge always has its instructions.
2. The first user message — so the judge always sees the original task prompt.

This means even in a long session, the judge never loses sight of what the agent was originally asked to do.

Deny behavior

When the judge denies a tool call, the on_deny parameter controls what happens next:

/policy guard judge_model=anthropic/claude-haiku-4-5 preset=standard_pentest on_deny=retry

on_deny	Behavior
retry (default)	The agent sees a [POLICY DENIED] message with the judge’s reason and gets a chance to choose a different approach.
finish	The turn ends immediately. Use for strict enforcement where a single out-of-scope attempt should halt the agent. You can still send a follow-up message to continue.

Error handling

If the judge model itself errors (network failure, malformed response, context overflow):

/policy guard judge_model=anthropic/claude-haiku-4-5 on_judge_error=deny

on_judge_error	Behavior
deny (default)	End the turn immediately. Fail closed — the agent cannot continue past a judge error.
allow	Allow the tool call to proceed. Fail open — logs a warning but lets the agent continue. Use only if you prefer availability over safety.
fail	Raise a hard error and halt the turn.

Regardless of the setting, if the error is a judge model authentication failure, AgentJudge disables itself for the rest of the session to prevent a retry loop. Subsequent tool calls pass through ungated.

Updating AgentJudge config mid-session

AgentJudge config merges with the previous config, so you can change individual parameters without restating everything:

/policy guard judge_model=anthropic/claude-haiku-4-5 preset=recon_only
# ... agent does recon work ...

# Escalate scope to standard pentest without re-specifying judge_model
/policy guard preset=standard_pentest

# Tighten transcript strategy
/policy guard transcript_strategy=intent_only

Note

Setting preset clears any previous scope config, and vice versa. The judge_model persists across updates unless you change it.

Choosing a judge model

The judge runs on every tool call, so model choice directly affects cost, latency, and how many violations slip through. ScopeJudge benchmarks eight judge models across five transcript strategies on ~5,000 expert-labeled tool calls and recommends two operating points:

Operating point	Model	Strategy	Recall	F1	~$/1k calls	When to use
Cost-sensitive	glm-5.2	intent_plus_calls	0.87	0.64	$3.22	General use — catches most violations at low cost. Best overall F1 of any judge tested.
Recall-first	anthropic/claude-opus-4-8	intent_only	0.94	0.55	$4.20	High-stakes deployments that prioritize catching violations over false positives.

An open-weight recall-first alternative is glm-5.2 with the full strategy (86% recall, F1=0.66, $6.01/1k calls) for teams that want high recall without a proprietary model.

Tip

anthropic/claude-haiku-4-5 is the cheapest proprietary option and works for simple rubrics, but it was not on the Pareto frontier in ScopeJudge testing. For stronger enforcement, use one of the models above.

The judge adds at least one LLM call per tool call. With intent_plus_outputs_summary, each previously unseen tool output also triggers a summarization call (summaries are cached per session, so repeated outputs are free). Factor this into long-running autonomous sessions.

Caching

AgentJudge prompt caching is on by default (cache=true) — cache-control markers are attached to the rubric, system instructions, and the rolling transcript prefix so that repeated content can be served from cache. Actual cache behavior depends on your model provider. Disable it if you need to:

/policy guard judge_model=anthropic/claude-haiku-4-5 cache=false

Quick-reference examples

Recon-only sweep of a target network:

/policy guard judge_model=anthropic/claude-haiku-4-5 preset=recon_only rubric=target:192.168.1.0/24,dns:internal.corp.local

Standard pentest with explicit boundaries:

/policy guard judge_model=anthropic/claude-haiku-4-5 preset=standard_pentest rubric=scope:web.target.com+api.target.com

Red team with minimal guardrails:

/policy guard judge_model=anthropic/claude-haiku-4-5 preset=red_team on_deny=retry

Strict mode — halt on first policy violation:

/policy guard judge_model=anthropic/claude-haiku-4-5 preset=standard_pentest on_deny=finish on_judge_error=fail

Budget-conscious — rubric-only strategy with minimal context:

/policy guard judge_model=anthropic/claude-haiku-4-5 rubric=no-network-outside-10.0.0.0/8 transcript_strategy=rubric_only

Dropping back to other modes

AgentJudge is a session policy like any other. Switch away at any time:

/interactive     # back to manual approval
/auto            # autonomous, no judge
/policy guard    # re-enable guard