Can a high Agent safety eval score pass acceptance?

01What does the paper actually evaluate?

The audit studies released trajectories and step records, then tests selected edits against guards configured as pre-execution judges. Its evidence concerns evaluation validity. The study executes no recorded tool calls and measures no end-to-end attack success rate in a deployed environment.

Hasan Demirkiran and Jens Ernstberger separate three claims that are often collapsed in discussions of benchmark shortcuts. A weak rule may predict labels, yet that alone doesn't show that a guard uses the same cue. Observational association supplies a candidate for a paired intervention that tests how a particular guard responds. Each claim requires its own evidence.

The distinction matters. A tool name may reveal whether an action can modify state. An external-content marker may communicate untrusted provenance. Either cue can carry legitimate safety information, even when it also reveals how benchmark records were collected.

The authors audit completed LinuxArena trajectories and two TS-Bench-derived step datasets, AgentDojo-Traj and AgentHarm-Traj. Under their strict mapping, any nonzero risk label counts as unsafe. The distinction between a candidate action and suspicious history matters when interpreting what a released safety label measures.

After exact-action deduplication, LinuxArena contributes 576 shell-bearing trajectories across 10 environments. Negative mean shell-command length alone produces an AUC of 0.910, exposing a predictive collection signature. The study also checks whether a monitor remains informative after trajectories are grouped into coarse length bands. It retains information within those bands. Label predictability therefore doesn't establish that the monitor merely follows the command-length ranking when judging a trajectory.

02What breaks when tool names cross domains?

Exact-tool lookup performs well when test records share domains with training records. Holding out an entire domain exposes its limited vocabulary coverage. This tests a diagnostic rule's transfer, without establishing how a capable guard behaves.

The AgentDojo-Traj diagnostic uses 1,220 steps from 4 domains to store each tool's majority training label. For an unseen tool, it uses the training fold's overall majority label. It's a data audit. The rule explicitly consumes evaluation labels and supplies no evidence of a deployable safety classifier's performance.

Split	Exact-tool lookup F1	Interpretation
Interaction-grouped cross-validation, mixed domains	0.901	Tool names predict labels within shared domains
Leave one domain out	0.495	That lookup has poor domain transfer

Of 51 exact tool names, 40 are label-pure and cover 776 rows. Under domain holdout, only 17.3% of rows use names seen in training domains. You need the fallback rule alongside the headline score, because unfamiliar tools dominate this test.

We checked the published arithmetic on October 18, 2026, using the confusion counts in Appendix Table 5. Its TP=137, TN=804, FP=64 and FN=215 sum to 1,220 held-out rows. Computing 2TP/(2TP+FP+FN) yields F1=0.495 after rounding, and balanced accuracy rounds to 0.658. Both match the paper. We used published counts without rerunning the dataset or calling a guard to generate new predictions.

For procurement, we would ask suppliers to show held-out-domain scores alongside the unknown-tool coverage of each split. A random split can provide useful evidence, but if unfamiliar tools dominate a holdout, readers need to see how the fallback changes predictions before deciding whether the aggregate supports their acceptance claim.

03Why does interaction weighting change F1?

Row weighting gives longer interactions more influence. Equal interaction weighting answers a different question using the same released predictions. A lower F1 after reweighting therefore needs interpretation alongside class balance and discrimination metrics.

The weighting audit uses 731 AgentHarm-Traj steps from 197 interactions whose lengths differ substantially by label. Its 72 harmful interactions contribute 7.29 rows on average, while 125 benign interactions contribute 1.65. The row-level sample gives harmful interactions more weight because they contain more recorded steps in the released artifact.

TS-Guard is ToolSafe's pre-execution tool-safety model. Its released predictions yield F1=0.902 under row weighting and 0.719 under equal interaction weighting. Balanced accuracy moves in the opposite direction, rising from 0.754 to 0.776 on the same released predictions. The statistical unit changes what the headline measures. A claim about interactions should therefore include interaction metrics even when the original report emphasizes steps.

An all-unsafe baseline reaches F1=0.836 on this dataset, despite balanced accuracy of 0.500. We recomputed that baseline from the published 525 unsafe and 206 safe steps, obtaining the same rounded result. Positive prevalence supports a high F1 even without useful discrimination between safe and unsafe records in this artifact.

If your acceptance claim concerns tasks, request interaction-level metrics alongside step metrics. If it concerns approval at each action, retain the step-level view too. Neither unit should silently stand in for the other when long attacks contribute repeated rows.

04Can a representation edit change guard decisions?

Claude Haiku 4.5 responds to representation edits under the tested configuration, with changes in its average blocking probability. Marker rewriting reduces blocking and opaque tool aliases increase it on a fixed cohort. The changes concern false positives against inherited safe labels, while unsafe calls remain blocked under every tested condition.

The authors freeze 64 marker-present steps from 58 source interactions, balancing 8 safe and 8 unsafe steps per domain. All 4 domains contribute equally. This deterministic panel was selected before scoring edited inputs, after released labels and earlier outputs had already been inspected. The selection doesn't provide a probability sample of deployment traffic from which population effects can be estimated.

Each step has five conditions: original, marker rewrite, tool alias, combined edit and formatting placebo. Replacing INFORMATION with UNTRUSTED_EXTERNAL_CONTENT retains the enclosed content, while tool aliases retain descriptions and arguments. The placebo changes only outer JSON whitespace, with equality of all field values checked after parsing the records.

Each model receives three calls per condition, interleaved deterministically across the panel for 960 calls in total. Repeats expose variation on identical inputs. The placebo measures sensitivity to a simpler serialization change, while repetition helps distinguish an edit's average effect from disagreement between individual hosted calls, even at temperature zero.

Edit	Haiku overall blocking change	Luna overall blocking change
Marker rewrite	-8.33 percentage points	0.00 percentage points
Tool alias	+11.46 percentage points	+1.04 percentage points

Haiku denotes Claude Haiku 4.5; Luna denotes GPT-5.6 Luna in this comparison of configurations.

These are marginal block-probability differences. Interpret them as percentage-point contrasts between repeated conditions, keeping them separate from relative changes and one-call flip rates. Every unsafe call remains blocked, so the overall Haiku changes come from the safe half of the panel. Safe-only contrasts are -16.67 and +22.92 percentage points respectively, twice the corresponding overall contrasts because the panel is balanced.

Haiku's formatting placebo changes blocking by -1.56 percentage points, which provides a comparison for the targeted edits. Subtracting that contrast gives -6.77 percentage points for the marker and +13.02 percentage points for aliases. Exploratory 95% ranges exclude zero. They use domain-stratified source-interaction resampling without multiplicity adjustment, and do not establish population effects for deployment traffic outside this fixed panel.

We also checked the observational screen separately using the safe-row counts in Appendix Table 3 on October 18, 2026. Haiku blocks 239/321 marked safe rows and 2/547 unmarked safe rows, giving a rounded 74.09-point difference. Requests and histories differ between these groups. That contrast selects an intervention target, without estimating the effect of rewriting a marker on a fixed step.

GPT-5.6 Luna doesn't reproduce the Haiku-sized paired effects under the same cohort and schedule of repeated calls. Its marker contrast is 0.00 percentage points and its alias contrast is +1.04, limiting claims of generality. The ranges don't prove equivalence. A near-zero estimate under one configuration also leaves uncertainty about other prompts, domains or input representations.

05What evidence should a supplier deliver?

Request the records needed to interpret each score, including grouping, vocabulary coverage, evaluation units and paired-edit protocols. The paper motivates those checks. Your team still needs acceptance thresholds tied to the tools and authorization policy covered by the procurement claim.

Deliverable	What we would inspect
Deduplication and split manifest	Whether related interactions or trajectories cross the split
Held-out-domain report	Unknown-tool coverage, fallback behavior and per-domain metrics
Unit comparison	Step-weighted and equal-interaction counts and metrics
Paired intervention record	Changed fields, authorization preservation, repeats and placebo

This table is our procurement recommendation. Its effectiveness hasn't been tested here, so apply it as a reporting proposal rather than a validated purchasing protocol. You still need acceptance criteria tied to the tools and authorization boundaries you expect the data to cover.

Semantic preservation needs independent attention. Rewriting the marker makes untrusted provenance more explicit to the guard, and per-sample aliases may add uncertainty; because the authors performed no blinded human equivalence check, interpreting a changed decision as spurious reliance requires additional evidence that relevant safety information survived the edit.

The deterministic Q8 TS-Guard reproduction flips 3/64 marker-edit decisions under a separate local quantized execution path. That result applies to its MLX configuration, with no claim of equivalence to the official BF16/vLLM path. Configuration is part of the finding.

Ask where the score holds before accepting it. Domain holdouts, explicit weighting and controlled edits make the evaluation claim easier for a buyer to inspect. End-to-end deployment safety requires separate evidence.

Paper: Shortcut Sensitivity in Agent-Safety Evaluations, version 1. We checked arithmetic from published counts without performing any model reproduction.

Related reading: 大厂采购的标的变了：从买数据集，到买环境和验证能力; Anthropic 关掉了内部评测的联网权限，原因写在自己的博客里：环境有漏洞.