SkillSonar generalizes zero-shot to external safety benchmarks, achieving the lowest ASR of 52.90% on SkillSafetyBench and highest safety score of 54.00% on WildClawBench compared to prompt, permission, and allowlist baselines.Whole claim: Experiment blocked; see the specific reason0/9 measurements supported · 0 assessed9measurementsExplore results

No experiment has been scheduled for this empirical claim; feasibility remains to be established.

Measurement results

0/9 assessed

0 supported0 disputed or conflicting9 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Metric and conditions	Reported	Latest observation	Evidence judgment
defense: No-guard · benchmark: SkillSafetyBench	93.3 percentage_points	—	Not assessed
defense: No-guard · benchmark: WildClawBench	38 percentage_points	—	Not assessed
defense: AcceptEdits · benchmark: WildClawBench	41 percentage_points	—	Not assessed
defense: AcceptEdits + allowlist · benchmark: SkillSafetyBench	64.3 percentage_points	—	Not assessed
defense: AcceptEdits + allowlist · benchmark: WildClawBench	45 percentage_points	—	Not assessed
defense: System prompt · benchmark: SkillSafetyBench	83.3 percentage_points	—	Not assessed
defense: System prompt · benchmark: WildClawBench	43 percentage_points	—	Not assessed
defense: SkillSonar (ours) · benchmark: SkillSafetyBench	52.9 percentage_points	—	Not assessed
defense: SkillSonar (ours) · benchmark: WildClawBench	54 percentage_points	—	Not assessed

Open claim details

On benign tasks containing no malicious skill behavior, SkillSonar preserves high utility on Claude Code (0.817 on GLM-5, 0.836 on Claude Haiku 4.5, and 0.724 on GPT-5.4), demonstrating minimal false-positive disruption of safe execution.Whole claim: Experiment blocked; see the specific reason0/15 measurements supported · 0 assessed15measurementsExplore results

No experiment has been scheduled for this empirical claim; feasibility remains to be established.

Measurement results

0/15 assessed

0 supported0 disputed or conflicting15 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Metric and conditions	Reported	Latest observation	Evidence judgment
model: GLM-5 · defense: No-guard	0.815 score	—	Not assessed
model: Claude Haiku 4.5 · defense: No-guard	0.827 score	—	Not assessed
model: GPT-5.4 · defense: No-guard	0.793 score	—	Not assessed
model: GLM-5 · defense: System prompt	0.833 score	—	Not assessed
model: Claude Haiku 4.5 · defense: System prompt	0.788 score	—	Not assessed
model: GPT-5.4 · defense: System prompt	0.836 score	—	Not assessed
model: GLM-5 · defense: AcceptEdits	0.796 score	—	Not assessed
model: Claude Haiku 4.5 · defense: AcceptEdits	0.813 score	—	Not assessed
model: GPT-5.4 · defense: AcceptEdits	0.74 score	—	Not assessed
model: GLM-5 · defense: AcceptEdits + allowlist	0.792 score	—	Not assessed
model: Claude Haiku 4.5 · defense: AcceptEdits + allowlist	0.819 score	—	Not assessed
model: GPT-5.4 · defense: AcceptEdits + allowlist	0.764 score	—	Not assessed

Open claim details

Over five matched evaluation rounds, SkillSonar reduces ASR across all individual risk families, achieving reductions of 50.0 pp on Resource & Reliability, 42.9 pp on Execution Safety, 35.3 pp on Specification Integrity, 18.3 pp on Operational Side Effects, 61.5 pp on Privacy & Data Flow (OOD), and 44.8 pp on Capability Control (OOD).Whole claim: Experiment blocked; see the specific reason0/18 measurements supported · 0 assessed18measurementsExplore results

No experiment has been scheduled for this empirical claim; feasibility remains to be established.

Measurement results

0/18 assessed

0 supported0 disputed or conflicting18 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Metric and conditions	Reported	Latest observation	Evidence judgment
split: id_test · defense: No Guard · risk_family: Resource & Reliability	61.3 percentage_points	—	Not assessed
split: id_test · defense: SkillSonar · risk_family: Resource & Reliability	11.3 percentage_points	—	Not assessed
split: id_test · risk_family: Resource & Reliability	50 percentage_points	—	Not assessed
split: id_test · defense: No Guard · risk_family: Execution Safety	48.6 percentage_points	—	Not assessed
split: id_test · defense: SkillSonar · risk_family: Execution Safety	5.7 percentage_points	—	Not assessed
split: id_test · risk_family: Execution Safety	42.9 percentage_points	—	Not assessed
split: id_test · defense: No Guard · risk_family: Specification Integrity	41.2 percentage_points	—	Not assessed
split: id_test · defense: SkillSonar · risk_family: Specification Integrity	5.9 percentage_points	—	Not assessed
split: id_test · risk_family: Specification Integrity	35.3 percentage_points	—	Not assessed
split: id_test · defense: No Guard · risk_family: Operational Side Effects	36.7 percentage_points	—	Not assessed
split: id_test · defense: SkillSonar · risk_family: Operational Side Effects	18.3 percentage_points	—	Not assessed
split: id_test · risk_family: Operational Side Effects	18.3 percentage_points	—	Not assessed

Open claim details

Packaging the safety policy as a modular skill-native artifact provides substantial gains over flattening the identical policy into a system prompt: SkillSonar reduces ID ASR from 0.353 to 0.104 and OOD ASR from 0.437 to 0.109, increases ID task utility from 0.655 to 0.815, and lowers token consumption from ~239K to ~188K (a ~21% reduction).Whole claim: Experiment blocked; see the specific reason0/12 measurements supported · 0 assessed12measurementsExplore results

No experiment has been scheduled for this empirical claim; feasibility remains to be established.

Measurement results

0/12 assessed

0 supported0 disputed or conflicting12 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Metric and conditions	Reported	Latest observation	Evidence judgment
split: id_test · configuration: Flattened prompt	0.353 fraction	—	Not assessed
split: ood_test · configuration: Flattened prompt	0.437 fraction	—	Not assessed
split: id_test · configuration: SkillSonar	0.104 fraction	—	Not assessed
split: ood_test · configuration: SkillSonar	0.109 fraction	—	Not assessed
split: id_test · configuration: Flattened prompt	0.655 score	—	Not assessed
split: ood_test · configuration: Flattened prompt	0.672 score	—	Not assessed
split: id_test · configuration: SkillSonar	0.815 score	—	Not assessed
split: ood_test · configuration: SkillSonar	0.789 score	—	Not assessed
configuration: Flattened prompt	0.748 score	—	Not assessed
configuration: SkillSonar	0.763 score	—	Not assessed
configuration: Flattened prompt	239 thousands_of_tokens	—	Not assessed
configuration: SkillSonar	188 thousands_of_tokens	—	Not assessed

Open claim details

Compared to action-level runtime defenses AgentSpec and TS-Guard, SkillSonar achieves superior safety–utility operating points on Claude Code with GLM-5: AgentSpec achieves 0.294 ID ASR but drops task utility to 0.581, while TS-Guard achieves 0.412 ID ASR, whereas SkillSonar reaches 0.104 ID ASR with 0.815 task utility.Whole claim: Experiment blocked; see the specific reason0/10 measurements supported · 0 assessed10measurementsExplore results

No experiment has been scheduled for this empirical claim; feasibility remains to be established.

Measurement results

0/10 assessed

0 supported0 disputed or conflicting10 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Metric and conditions	Reported	Latest observation	Evidence judgment
split: id_test · defense: AgentSpec	0.294 fraction	—	Not assessed
split: ood_test · defense: AgentSpec	0.576 fraction	—	Not assessed
split: id_test · defense: AgentSpec	0.581 score	—	Not assessed
defense: AgentSpec	0.642 score	—	Not assessed
split: id_test · defense: TS-Guard	0.412 fraction	—	Not assessed
split: ood_test · defense: TS-Guard	0.492 fraction	—	Not assessed
defense: TS-Guard	0.703 score	—	Not assessed
split: id_test · defense: No-guard	0.477 fraction	—	Not assessed
split: id_test · defense: No-guard	0.829 score	—	Not assessed
defense: No-guard	0.757 score	—	Not assessed

Open claim details

Human audit of the LLM judges confirms high reliability: the data construction judge achieves 94% balanced audit agreement (88% success precision, 100% failure confirmation) and the final evaluation judge achieves 98% balanced agreement (96% success precision, 100% failure confirmation).Whole claim: Experiment blocked; see the specific reason0/6 measurements supported · 0 assessed6measurementsExplore results

No experiment has been scheduled for this empirical claim; feasibility remains to be established.

Measurement results

0/6 assessed

0 supported0 disputed or conflicting6 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Metric and conditions	Reported	Latest observation	Evidence judgment
stage: Data construction	88 percentage_points	—	Not assessed
stage: Data construction	100 percentage_points	—	Not assessed
stage: Data construction	94 percentage_points	—	Not assessed
stage: Final evaluation	96 percentage_points	—	Not assessed
stage: Final evaluation	100 percentage_points	—	Not assessed
stage: Final evaluation	98 percentage_points	—	Not assessed

Open claim details

Under an adaptive attacker performing three iterations of targeted adversarial skill refinement against the final guard, SkillSonar reduces overall ASR from 45.1% to 22.9% (-22.2 percentage points), with reductions across all six individual risk families.Whole claim: Experiment blocked; see the specific reason0/21 measurements supported · 0 assessed21measurementsExplore results

No experiment has been scheduled for this empirical claim; feasibility remains to be established.

Measurement results

0/21 assessed

0 supported0 disputed or conflicting21 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Metric and conditions	Reported	Latest observation	Evidence judgment
defense: No Guard · risk_family: Capability Control	55 percentage_points	—	Not assessed
defense: SkillSonar · risk_family: Capability Control	26.7 percentage_points	—	Not assessed
risk_family: Capability Control	-28.3 percentage_points	—	Not assessed
defense: No Guard · risk_family: Specification Integrity	53.3 percentage_points	—	Not assessed
defense: SkillSonar · risk_family: Specification Integrity	43.3 percentage_points	—	Not assessed
risk_family: Specification Integrity	-10 percentage_points	—	Not assessed
defense: No Guard · risk_family: Resource & Reliability	45 percentage_points	—	Not assessed
defense: SkillSonar · risk_family: Resource & Reliability	26.7 percentage_points	—	Not assessed
risk_family: Resource & Reliability	-18.3 percentage_points	—	Not assessed
defense: No Guard · risk_family: Privacy & Data Flow	43.3 percentage_points	—	Not assessed
defense: SkillSonar · risk_family: Privacy & Data Flow	8.3 percentage_points	—	Not assessed
risk_family: Privacy & Data Flow	-35 percentage_points	—	Not assessed

Open claim details

Optimized SkillSonar guards exhibit bidirectional zero-shot transfer across distinct model families without target-model fine-tuning: the GLM-5-optimized guard reduces Kimi-K2.6 ASR to 0.199 (ID) and 0.161 (OOD), while the Kimi-K2.6-optimized guard achieves 0.112 ID ASR on GLM-5, 0.059 on Claude Haiku 4.5, and 0.019 on GPT-5.4.Whole claim: Experiment blocked; see the specific reason0/7 measurements supported · 0 assessed7measurementsExplore results

No experiment has been scheduled for this empirical claim; feasibility remains to be established.

Measurement results

0/7 assessed

0 supported0 disputed or conflicting7 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Metric and conditions	Reported	Latest observation	Evidence judgment
split: id_test · source_model: GLM-5 · target_model: Kimi-K2.6	0.199 fraction	—	Not assessed
split: ood_test · source_model: GLM-5 · target_model: Kimi-K2.6	0.161 fraction	—	Not assessed
split: id_test · source_model: Kimi-K2.6 · target_model: GLM-5	0.112 fraction	—	Not assessed
split: id_test · source_model: Kimi-K2.6 · target_model: Claude Haiku 4.5	0.059 fraction	—	Not assessed
split: id_test · source_model: Kimi-K2.6 · target_model: GPT-5.4	0.019 fraction	—	Not assessed
split: id_test · source_model: GLM-5 · target_model: Kimi-K2.6	0.799 score	—	Not assessed
split: ood_test · source_model: GLM-5 · target_model: Kimi-K2.6	0.799 score	—	Not assessed

Open claim details

In the OpenClaw agent environment, SkillSonar operates without modifying host runtime mechanics, reducing GLM-5 ASR to 0.245 (ID) and 0.232 (OOD), and Claude Haiku 4.5 ASR to 0.265 (ID) and 0.259 (OOD), demonstrating cross-harness portability.Whole claim: Experiment blocked; see the specific reason0/30 measurements supported · 0 assessed30measurementsExplore results

No experiment has been scheduled for this empirical claim; feasibility remains to be established.

Measurement results

0/30 assessed

0 supported0 disputed or conflicting30 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Metric and conditions	Reported	Latest observation	Evidence judgment
model: GLM-5 · split: id_test · defense: No-guard · environment: OpenClaw	0.596 fraction	—	Not assessed
model: GLM-5 · split: ood_test · defense: No-guard · environment: OpenClaw	0.621 fraction	—	Not assessed
model: GLM-5 · split: id_test · defense: No-guard · environment: OpenClaw	0.893 score	—	Not assessed
model: GLM-5 · split: ood_test · defense: No-guard · environment: OpenClaw	0.92 score	—	Not assessed
model: GLM-5 · defense: No-guard · environment: OpenClaw	102,398 tokens	—	Not assessed
model: GLM-5 · split: id_test · defense: System prompt · environment: OpenClaw	0.577 fraction	—	Not assessed
model: GLM-5 · split: ood_test · defense: System prompt · environment: OpenClaw	0.453 fraction	—	Not assessed
model: GLM-5 · split: id_test · defense: System prompt · environment: OpenClaw	0.807 score	—	Not assessed
model: GLM-5 · split: ood_test · defense: System prompt · environment: OpenClaw	0.835 score	—	Not assessed
model: GLM-5 · defense: System prompt · environment: OpenClaw	93,352 tokens	—	Not assessed
model: GLM-5 · split: id_test · defense: SkillSonar (ours) · environment: OpenClaw	0.245 fraction	—	Not assessed
model: GLM-5 · split: ood_test · defense: SkillSonar (ours) · environment: OpenClaw	0.232 fraction	—	Not assessed

Open claim details

Explicit safety responsibility assignment is necessary for effective runtime guarding: without explicit invocation, ordinary skill selection yields high ASR (0.400 ID, 0.582 OOD), whereas explicit invocation reduces ASR to 0.104 ID and 0.109 OOD while maintaining comparable benign utility (0.756 vs. 0.767).Whole claim: Experiment blocked; see the specific reason0/6 measurements supported · 0 assessed6measurementsExplore results

No experiment has been scheduled for this empirical claim; feasibility remains to be established.

Measurement results

0/6 assessed

0 supported0 disputed or conflicting6 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Metric and conditions	Reported	Latest observation	Evidence judgment
split: id_test · configuration: No explicit invocation	0.4 fraction	—	Not assessed
split: ood_test · configuration: No explicit invocation	0.582 fraction	—	Not assessed
split: id_test · configuration: Explicit invocation	0.104 fraction	—	Not assessed
split: ood_test · configuration: Explicit invocation	0.109 fraction	—	Not assessed
configuration: No explicit invocation	0.756 score	—	Not assessed
configuration: Explicit invocation	0.767 score	—	Not assessed

Open claim details

During Monte Carlo Tree Search guard evolution, full-evaluation malicious ASR decreases from 0.300 at iteration 1 to 0.078 by iteration 9, demonstrating that feedback-driven refinement effectively optimizes guard policy efficacy.Whole claim: Experiment blocked; see the specific reason0/2 measurements supported · 0 assessed2measurementsExplore results

No experiment has been scheduled for this empirical claim; feasibility remains to be established.

Measurement results

0/2 assessed

0 supported0 disputed or conflicting2 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Metric and conditions	Reported	Latest observation	Evidence judgment
iteration: 1	0.3 fraction	—	Not assessed
iteration: 9	0.078 fraction	—	Not assessed

Open claim details

On Claude Haiku 4.5 and GPT-5.4, SkillSonar generalizes zero-shot across model architectures, reducing ID/OOD ASR to 0.096/0.254 on Claude Haiku 4.5 and 0.019/0.034 on GPT-5.4, while preserving higher utility than default AcceptEdits.Whole claim: Experiment blocked; see the specific reason0/50 measurements supported · 0 assessed50measurementsExplore results

No experiment has been scheduled for this empirical claim; feasibility remains to be established.

Measurement results

0/50 assessed

0 supported0 disputed or conflicting50 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Metric and conditions	Reported	Latest observation	Evidence judgment
model: Claude Haiku 4.5 · split: id_test · defense: No-guard	0.481 fraction	—	Not assessed
model: Claude Haiku 4.5 · split: ood_test · defense: No-guard	0.707 fraction	—	Not assessed
model: Claude Haiku 4.5 · split: id_test · defense: No-guard	0.764 score	—	Not assessed
model: Claude Haiku 4.5 · split: ood_test · defense: No-guard	0.737 score	—	Not assessed
model: Claude Haiku 4.5 · defense: No-guard	179,918 tokens	—	Not assessed
model: Claude Haiku 4.5 · split: id_test · defense: System prompt	0.451 fraction	—	Not assessed
model: Claude Haiku 4.5 · split: ood_test · defense: System prompt	0.508 fraction	—	Not assessed
model: Claude Haiku 4.5 · split: id_test · defense: System prompt	0.661 score	—	Not assessed
model: Claude Haiku 4.5 · split: ood_test · defense: System prompt	0.662 score	—	Not assessed
model: Claude Haiku 4.5 · defense: System prompt	160,672 tokens	—	Not assessed
model: Claude Haiku 4.5 · split: id_test · defense: AcceptEdits	0.118 fraction	—	Not assessed
model: Claude Haiku 4.5 · split: ood_test · defense: AcceptEdits	0.051 fraction	—	Not assessed

Open claim details

On repeated runtime evaluation over N=10 runs with GLM-5 on Claude Code, SkillSonar reduces ID ASR from 0.482 to 0.104 and OOD ASR from 0.606 to 0.115 while maintaining 0.779 ID and 0.715 OOD task utility, significantly outperforming system-prompt and permission-based baselines.Whole claim: Experiment blocked; see the specific reason0/25 measurements supported · 0 assessed25measurementsExplore results

No experiment has been scheduled for this empirical claim; feasibility remains to be established.

Measurement results

0/25 assessed

0 supported0 disputed or conflicting25 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Metric and conditions	Reported	Latest observation	Evidence judgment
split: id_test · defense: No-guard	0.482 fraction	—	Not assessed
split: ood_test · defense: No-guard	0.606 fraction	—	Not assessed
split: id_test · defense: No-guard	0.811 score	—	Not assessed
split: ood_test · defense: No-guard	0.839 score	—	Not assessed
defense: No-guard	179.3 thousands_of_tokens	—	Not assessed
split: id_test · defense: System prompt	0.414 fraction	—	Not assessed
split: ood_test · defense: System prompt	0.489 fraction	—	Not assessed
split: id_test · defense: System prompt	0.783 score	—	Not assessed
split: ood_test · defense: System prompt	0.807 score	—	Not assessed
defense: System prompt	175.6 thousands_of_tokens	—	Not assessed
split: id_test · defense: AcceptEdits	0.122 fraction	—	Not assessed
split: ood_test · defense: AcceptEdits	0.073 fraction	—	Not assessed

Open claim details

SCOPE-R contains 206 attack-confirmed malicious skill instances partitioned into 95 train instances (46.1%), 52 ID-test instances (25.2%), and 59 OOD-test instances (28.6%), with Capability Control and Privacy & Data Flow entirely held out from training.Whole claim: Experiment blocked; see the specific reason0/8 measurements supported · 0 assessed8measurementsExplore results

No experiment has been scheduled for this empirical claim; feasibility remains to be established.

Measurement results

0/8 assessed

0 supported0 disputed or conflicting8 not assessed

The complete result group stays with its claim. Assessed rows come first; their judgments do not replace the whole-claim conclusion. Expand a row for conditions, sources, and evidence history.

Each row is a measurement with its reported value, latest available observation, and cumulative evidence state.
Metric and conditions	Reported	Latest observation	Evidence judgment
split: train	95 count	—	Not assessed
split: train	46.1 percentage_points	—	Not assessed
split: id_test	52 count	—	Not assessed
split: id_test	25.2 percentage_points	—	Not assessed
split: ood_test	59 count	—	Not assessed
split: ood_test	28.6 percentage_points	—	Not assessed
split: total	206 count	—	Not assessed
split: total	100 percentage_points	—	Not assessed

Open claim details

Defense-as-Skill: Evolving Runtime Guard Skill for Skill-Augmented Agents

Research claims

Reproduction results

No reproduction results yet

Key metricsPaper-reported → reproduced · click a row to expand

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results

Measurement results