// Evaluations
A score you can argue with.
Half of the grade is arithmetic and half of it is judgement, and the result says which is which. Every number resolves back to something recorded during the run (a check that passed, a rubric criterion a judge scored, a span in the trace) so a score comes with its working attached rather than asking to be taken on trust.
// Methodology
Two scores, kept apart.
Deterministic checks and the judge are weighted evenly and reported separately. Collapsing them into a single figure would hide the thing you most need to see: whether an agent got the answer, or whether it got there in a way you would accept.
- 50% deterministicHard checks scored by the control plane: flag match, artefact presence, asserts over the trace. No model is involved and the answer is the same every time.
- 50% LLM judgeA model reads the full record of the run and scores it against the mission's rubric, criterion by criterion. Where no judge is configured, this half scores zero rather than being quietly redistributed.
An agent can hit the flag and still lose half the marks. That is the point.
How the judge scores
The judge follows an LLM-as-judge approach adapted from G-Eval: the mission's rubric criteria serve as the fixed evaluation steps, and the model returns a per-criterion score that is weighted into [0,1].
// The rubric
Written before the run, not after.
The grading rubric is not ours to set at scoring time. It ships inside the mission pack. Pulling a mission installs the weighted criteria the judge will score against, along with the deterministic checks, and both sit on your machine before the first agent attempts it.
Because it is versioned with the mission, every agent that attempts it is measured against the identical standard, and you can read that standard yourself, in advance, and disagree with it before a single run is scored.
// The judge
Swap the judge. Regrade. Compare.
Bring your own judge. Any OpenAI-compatible model will do: set the base URL, the model name and a key in Settings, and the key is stored locally in ~/.xorcise/.env and never returned. A Test button calls the model live, so you find out it works before a run depends on it rather than after.
Judges are swappable, and a graded run can be graded again. Point the console at a different model, regrade the same runs, and compare. The agent's record is fixed, so the only variable is the marking. That is how you tune a judge for accuracy and consistency before you trust it to mark anything that matters.
// Results
Every grade points at its evidence.
A result is not a number and a shrug. Every run exports as a self-contained Markdown or HTML report: metadata, both scores, the check table, the judge rubric with the evidence and deduction behind each criterion, artifacts, terrain coverage, telemetry, and the conditions the run was carried out under.
Held onto, these become the track record for a given agent version. The rubric is fixed and versioned with the mission, so a report from March and a report from July are measuring the same thing; the difference between them is the agent, not the marking. Regressions show up as a criterion that used to score and no longer does.
A real export from a real run: a Claude agent on the claude-code harness, graded by gpt-5.2. Different vendors on each side, on purpose: a model should not mark its own homework. The flag, the pivot answer, the byte offsets and the agent's writeup are redacted here because this mission is in the public library; the reports you generate carry all of them in full.
Prove it on your own agent.
pip install xorciseApache-2.0 · Python 3.12+ · runs locally · no phone-home.