// Use Cases
Building one, or accepting one.
Two jobs, one record. If you build cyber agents, you need to know what a change did before it ships. If you are asked to field somebody else's, you need something you did not produce yourself.
// Developing cyber agents
A test suite for behaviour.
Your unit tests prove the code runs. They say nothing about whether the agent still solves the problem after you changed the prompt. A mission is that test: a contained environment, a fixed rubric, and a score that moves when behaviour moves.
Where it sits
- commit
- unit
- build
- XORCISE
- stage
- deploy
One local process, driven from a CLI. It runs where your other tests run, on your machine or on your runner. The agent needs registering once; after that a run is three commands. Nothing about a run needs us to be reachable. XORCISE is in public beta: this is where it is built to sit, and people are running it there now, but pin the version if a pipeline is going to depend on it.
What triggers a run
- Every agent changePrompt, tools, scaffolding, guardrails. A behavioural regression does not show up in a unit test; it shows up as a mission the agent used to solve and now does not.
- Every model variantThe model is the biggest single variable and you do not control its release schedule. Run the same missions against the candidate before it reaches production.
- Every harness changeA harness upgrade changes what the agent can reach and how many turns it gets. Pin the harness, change one thing, and the delta means something.
- Every run, keptEach run leaves a report you can file. Six months on, the question is not what the agent scores now but what it scored then, under which model, and whether anyone checked.
One mission is a smoke test. A whole matrix (every model against every mission, several attempts each) is a playbook, which is what you want before a model swap rather than after.
// Accepting cyber AI products
Someone has to accept this agent.
It needs credentials that can cause loss, in an environment you are answerable for, and it will act faster than you can review. Whether a vendor built it or your own platform team did, the person carrying the consequence needs something they did not produce themselves.
If a vendor built it
A system that tunes an agent cannot grade it. Once the same party produces the fix and issues the score, the score measures how well it tuned to its own rubric. That holds whether or not anyone is acting in good faith.
If you built it
The problem is proximity, not honesty. The team that tuned the agent wrote the rubric that scores it, so the evaluation covers the failure modes they already considered. The ones they did not anticipate are the ones that matter, and those are structurally invisible from inside.
Either way, the agent runs on a model somebody else trained and a harness somebody else maintains, both of which update underneath you. Authorship changes who wrote the rubric. It does not change how quickly the result goes stale.
What a record has to survive
Leaving the room it was produced in, read by someone with no stake in the answer.
- Reproducible from a trace.
- Stated conditions: model, harness, mission version, date.
- Authority declared before the run.
- Failures published alongside passes.
- An expiry, because the model will be updated underneath the result.
Five questions, whoever built it
- Authority
- What was this agent authorised to do, and what was withheld?Restraint is only measured where the withheld actions were declared and checked during the run.
- Rubric
- Who wrote the criteria this was scored against?If it was the people who tuned it, the score covers the failures they anticipated. Ask what it was not looking for.
- Conditions
- Which model and harness version, and on what date?A result without conditions cannot be compared to anything, including itself later.
- Pressure
- What happened under ambiguity, conflicting instruction and adversarial input?A single clean run is the easiest result to produce and the least informative.
- Failure
- Show me a run where it failed.An evaluation that has never produced a failure was built to avoid one.
// Including us
Ask them of us, too.
XORCISE authors the missions, writes the rubrics, ships the grader and publishes the leaderboard. By the argument above, that is precisely the position you should not take on trust, and we are not asking you to.
The answer is not our good faith. It is that there is nothing held back to have faith in.
- The rubric and its weights ship inside the mission pack. They install to
~/.xorcise/missions/<id>/and are readable before the first agent attempts it. - The deterministic checks are declared in that same manifest: source, ref, operator, weight.
- The collector writes the raw trace verbatim and seals it at terminal. Every other view is a projection you can rebuild.
- The exact judge prompt is preserved on the result, with every per-criterion score and the reason given for it.
- The engine, the mission library, the playbooks skill and the docs are Apache-2.0.
So read the rubric and disagree with it before you run anything. Swap our judge for one of yours and regrade the same evidence. Re-run the mission and check the verdict, including checking ours. Our own published results carry their conditions, their caveats and their failures for exactly this reason. Read a mission manifest →
Prove it on your own agent.
pip install xorciseApache-2.0 · Python 3.12+ · runs locally · no phone-home.