// XORCISE
Control centre for cyber AI evaluation.
One local process: the console, the fence, the collector and the grader. Register any agent, send it on a contained mission, and read back the whole record of what it did.
// Features
Five views, one record.
The console is not a dashboard bolted onto a test runner. Each view is a different cut of the same evidence, and every number in it is traceable to something that happened.
Dashboard
Counts, system health, and how your fleet is performing.
Agents registered, missions installed, runs completed and evaluated, with the pass rate and score spread across everything you have run. Module, catalog and topology health sit alongside them, so a bad number and a broken component are never confused for one another.
Agents
Every agent you have registered, and how it has held up.
Register any agent that speaks OTLP: Claude Code, Codex, OpenHands, or your own harness. Each carries a track record across runs and model versions, so a single good result is never mistaken for a stable one.
Mission Runs
The full record of what the agent actually did.
Reasoning, shell commands, file touches and tool calls, timestamped and attributed, streaming as the run happens rather than reconstructed afterwards. Deploy state and teardown are part of the record, not a footnote to it.
Results
Deterministic checks and the judge, scored apart.
Hard checks anchor the score: flag match, artefact presence, asserts. The judge scores method against a rubric fixed before the run. Every deduction names what was missing and links back to where in the trace it should have happened.
Configure
The conditions a result has to be read in.
The judge model and its key, the terrain attribution model, token caps, the mission library connection and the runtime environment. These are the settings a score depends on. Change one and every result after it is read differently, so the record carries them.
// Technology
Architecture
You run XORCISE. It never runs your agent. Your agent works in your own sandbox and your own model grades the evidence. XORCISE provides the target, the isolation, the recording and the verdict, and puts a network fence between the agent and the mission that holds by construction, not by trust: a per-run Headscale ACL, enforced at the network layer, that lets the agent reach its one mission and nothing else. Rendering a policy that would allow more fails closed. how the fence is built.
- from XORCISE (prompt · deploy)
- live data from agent (trace · attack)
- your model (trace ↑ / eval ↓)
Live data streams across the fence: the agent's trace back to XORCISE, and its traffic out to the target.
What leaves your machine
Three answers, because “local” on its own is not one. A run needs nothing of ours to be reachable.
- Nothing to us, everNo phone-home and no usage telemetry. App state, the trace store and every result stay in
~/.xorciseon your machine. The OTel mirror is off by default. - The mission catalogue, when you ask for it
xorcise mission pullandxorcise catalog statusfetch from our registry on demand. Once a mission is on disk, running and grading it never touches us again. - Your judge modelHalf the grade comes from a model you configure, so the submitted artifacts and a distilled transcript of the run go to that endpoint, which can be a self-hosted or gateway one, since any OpenAI-compatible URL will do. Configure no judge and that half scores zero rather than being quietly redistributed.
Stack
Boring where it counts. Standards at the seams, no heavyweight frameworks in the open-source grader.
Prove it on your own agent.
pip install xorciseApache-2.0 · Python 3.12+ · runs locally · no phone-home.