Playbooks
Playbooks
A playbook is a means to automate multiple XORCISE mission runs and analyse the results in aggregate. The YAML declares the missions, attempts per mission, a per-run time budget, hint policy and a pinned agent harness; you name the models at invocation. It ships in the xorcise-playbooks Claude Code skill, which expands the models × missions × attempts matrix, drives every run through the XORCISE API, and aggregates the rows into summary.csv and an HTML eval card.
Playbooks are an experimental feature. The skill ships outside XORCISE's own release process, so its interface can change between versions and nothing here carries the compatibility guarantees the engine does. A benchmark also spends real money: the matrix is models × missions × attempts, and the judge is billed once for every run in it. Start with xor-pb-000, which is the cheap way to find out what a run actually costs before committing to a matrix.
git clone https://github.com/xorcise-ai/xorcise-skills.gitxorcise-playbooks/SKILL.md is a plain Markdown skill file: point whichever skill-aware agent you use at it, or copy that directory into your agent's skills folder — however that agent loads skills. Scripts and templates: no credentials, nothing phoning home.
Apache-2.0, like the engine and the mission library. Read it before you point an agent at it, because the agent is about to run it — the repository is the only thing we publish it from, so what you clone is the source rather than a copy of it.
The version to trust is the one attached to a tagged release, diffable against the history it came from. Those releases are not yet signed; that is on the list, with the rest of what we have not done.
Playbooks drive an engine you already have. Install XORCISE first and have Docker running. The skill inspects your install, it will not create one. Install XORCISE →
Catalogue
- xor-pb-000mixed
Pipeline shakeout
Three missions, intermediate to expert, to prove the path works: telemetry flowing, runs coming back graded. Run this before you commit to a matrix.
- Missions
- 3
- Attempts each
- 3
- Runs per model
- 9
- Budget
- 1200s
- xor-pb-001expert
Generalist coverage
One expert-tier mission per specialty, web and pwn through forensics, detection and AI security. Measures how a model holds up across the spectrum, not just the part it is good at.
- Missions
- 15
- Attempts each
- 10
- Runs per model
- 150
- Budget
- 1800s
- xor-pb-002expert
Red team
Binary exploitation, web, crypto, reverse engineering, malware analysis: the classic offensive ground, run against live contained environments rather than a static corpus.
- Missions
- 8
- Attempts each
- 10
- Runs per model
- 80
- Budget
- 1800s
- xor-pb-003expert
Blue team
Forensics, detection and threat hunting, live network defence, secure coding: the half of the job public agent benchmarks almost entirely omit.
- Missions
- 9
- Attempts each
- 10
- Runs per model
- 90
- Budget
- 1800s
These are scripts, not scores. A playbook produces a different set of numbers every time anyone runs it, on their own models and their own machine. What we got when we ran them is written up separately. Our results →