Technology · AI and automation · Component
AI evaluation record
The test set, the measured quality, the thresholds accepted, and how performance is watched once the system is live.
The deliverable
What it is
AI systems are evaluated as other measurements are: against cases with known answers, with the types of error kept apart because they do not cost the same. A summarisation tool that omits a clause and one that invents a clause have different consequences, and a single accuracy figure conceals which is happening.
Evaluation continues after launch. Providers update models, inputs drift, and a prompt changed to fix one case can break others; re-running the same test set after each change is the practical safeguard.
One level in
What it is made of
Each element is a constituent part of the component. Follow one to see the attributes it carries.
Test set
A fixed collection of real cases with agreed good answers, including the difficult and adversarial ones.
3 attributes: Cases · Drawn from · Version
LearnQuality thresholds
The error rates accepted before launch, set separately for each kind of error according to what it would cost.
4 attributes: Error type · Accepted rate · Human baseline · Basis
LearnLive monitoring
How performance is watched in use: sampled review, user reports, and a re-run of the test set after each change.
3 attributes: Review cadence · Current error rate · Within threshold
Learn
Keep the test set fixed and re-run it after every change. Without that, nobody can say whether an update improved anything.
The other components in ai and automation
AI use-case register
Every place AI or automation is used, with its task, owner, provider, level of autonomy and risk classification.
LearnAI acceptable-use policy
What staff may and may not do with AI tools, which tools are sanctioned, and what material may be given to them.
LearnHuman oversight design
For each use, where a person reviews, approves or can stop the system, and what they need in order to do so meaningfully.
Learn