The defensive half is harder

What the numbers say

• 60.0% vs 72.2%

  The best defensive score is twelve points below the best offensive score, on the same models.

  glm-5.2 tops both sets, at 72.2% on the offensive core and 60.0% here. This is the comparison public agent benchmarks rarely make, because they rarely run the defensive half at all — and it says the two halves are not interchangeable measures of “security capability”.

• 2.5 points

  The top three are indistinguishable at this sample size.

  glm-5.2 at 60.0%, kimi-k3 at 58.9% and deepseek-v4-pro at 57.5% sit inside 2.5 points of one another over three attempts per cell. Read that as a tie, not a podium. Below them the gap opens sharply, to 25.4% and 23.2%.

• 6% vs 41%

  The judge half runs far above the deterministic half for every model — and at the bottom of the table the gap is enormous.

  inkling scores 6% on the checks and 41% from the judge; nemotron-3-ultra, 7% against 44%. Even the leaders show it — glm-5.2 at 51% deterministic against 69% judged, kimi-k3 at 46% against 72%. The judge is reading plausible defensive method in work that did not actually achieve the objective. On this mission set the deterministic half is the harder question, and reporting the overall alone would flatter every model in the pool.

• 3 of 9

  A third of the mission set was never solved once.

  molten-perimeter (a pfSense VMDK/ZFS appliance image, 4.2% mean), pipe-dream (Windows EVTX and RPC named pipes, 26.1%) and need-to-know (fixing Flask access control, 34.8%) all took a 0/15 solve rate. Two of the three are disk-and-log forensics on unfamiliar appliance formats, which is where this pool fails hardest — not on the reasoning, on the file formats.

Directly comparable with the offensive baseline: same models, same harness, same judge, same attempts per cell. Only the mission set differs.

Runs

135/135

Models

5

Missions

9

Runs / cell

3

Top model

glm-5.2

Mean overall

45.0%

Total tokens

452.4M

Wall-clock

43h 49m

Leaderboard

Overall is the mean across every attempt, so it is not the same question as pass@1, which asks how often a single attempt succeeds. Where the two orderings disagree, both are reported rather than reconciled.

Missions

Conditions

Read this with

• Scope

  Complete run — all 135 attempts finished: 5 model(s) × 9 mission(s) × 3 run(s) each.

• Excluded / failed runs

  None — every one of the 135 runs produced tool calls and graded.

• Small n

  3 run(s) per model×mission cell — read cells as mean ± std, not point estimates.

• Single LLM judge

  Scoring is 0.5× deterministic + 0.5× one LLM judge; a single grader carries its own biases.

• Unaided

  Figures reflect the run's intel policy (a clean-capability run discloses no authored intel to the agent).

The evidence

The eval card above, every run behind it as a Markdown report, and a sha256sum -c manifest over those reports. Mission flag values are redacted, because the mission set is a live benchmark; nothing else is.

135 run reports. 54 done · 54 timeout · 25 operator · 2 deploy_failed.

Every run