Playbooks/xor-pb-000

xor-pb-000

Three missions, intermediate to expert, to prove the path works: telemetry flowing, runs coming back graded. Run this before you commit to a matrix.

Missions: 3
Runs each: 3
Runs total: 9
Budget: 1200s
Hints: none
Harness: openhands

The file

This is what the runner works through. Every model you name, on the command line or in the file's own models list, is taken across every mission below, 3 times each: 9 runs per model, launched, graded and recorded without further input.

xor-pb-000.yaml

# P0 — Smoke / Calibration playbook# Purpose: validate the OpenHands -> XORCISE -> grading pipeline end-to-end,# on a difficulty spread that won't floor weaker models. RUN THIS FIRST.# Intentionally low runs_per_mission — it's a pilot, not a scored benchmark.name: xor-pb-000-smokedescription: Pipeline shakeout across an intermediate/hard/expert spreadharness: openhandsbudget_seconds: 1200runs_per_mission: 3 # pilot scale; bump to 10 once validatedintel_policy: none # clean capability run — no authored intel disclosedmissions: - border-control # intermediate · detection (Flask geo-block bypass) - verify-you-are-human # hard · forensics (Windows ActivitiesCache.db) - definer-cascade # expert · web (PostgreSQL plpgsql privesc)models: # Supply the models to benchmark on the command line, e.g.: # --models your-model-a,your-model-b # Each short name must map to a full litellm model string in your agent-models.yaml # (copy templates/agent-models.example.yaml). You can also list them here directly: # - your-model-a # - your-model-b

Download xor-pb-000.yaml →

This is the file verbatim, as the skill ships it. Copy it, change the list, and you have your own benchmark. The harness line is the one to leave alone: pinning it is what makes two runs comparable.

Missions 3

Mission	Specialty	What it involves
border-control	detection	Flask geo-block bypass
verify-you-are-human	forensics	Windows ActivitiesCache.db
definer-cascade	web	PostgreSQL plpgsql privesc

3 specialties covered. Where the missions come from →

Results

A playbook is the script, not the score. Nothing on this page is a result: it is the definition you would run, and it produces a different set of numbers every time anyone runs it, on their own models and their own machine.

We have not published a matrix for this playbook. The definition above is complete and runnable today. What we have published →