Applied AI / Capability
AI evaluation & testing.
Measure whether an AI feature is good enough, using your own examples, before and after it goes live.
Describe the task
When to bring us in
Sound familiar?
- Our AI feature looked great in the demo but behaves differently with real data.
- We can’t tell whether a new model or prompt is better or just different.
- Nobody has defined what an acceptable error rate is for this task.
The cost of leaving it
Nobody can say whether a model change made things better or just different. Problems show up as user complaints, after the change is live, with no baseline to compare against.
Why us
What we bring to ai evaluation & testing.
A person stays in charge
Models work on bounded tasks inside your systems, and anything that matters goes to a named person to confirm.
Tested on your examples first
We measure a model on real cases from your work before it reaches staff, so you see where it fails while the stakes are low.
We say when AI is the wrong tool
Not every task should be automated. When conventional software or a person is the better answer, we tell you.
What the work includes
Decide what good enough means, then measure it.
We collect real examples of the task with agreed correct answers, including the awkward cases. Different models, prompts and approaches are compared against that set for quality, speed and cost. The same set runs again whenever something changes, so a regression shows up before your users find it.
Evaluation set
Representative examples with agreed correct outputs, including difficult and edge cases.
Pass criteria
Measures that match the task, and the level each must reach before launch.
Approach comparison
Models, prompts and non-AI options compared on quality, latency and cost.
Ongoing checks
The evaluation runs again on every change, with results recorded over time.
Use cases
Projects we take on.
Checking readiness before launch
An AI feature that works in demos, measured against real cases to decide whether customers should see it yet.
Comparing models for one task
Several models run on the same examples to see which meets the quality bar at a cost you can accept.
Catching regressions after changes
An evaluation that runs whenever a prompt, model or retrieval setting changes, so a drop in quality is caught.
What you get
Outputs you can review.
- Evaluation set with agreed answers
- Pass criteria for each measure
- Comparison report across approaches
- Repeatable evaluation script
- Review of failure cases
Technologies we use for this
Where it matters
- Health & social care Staff can look up care policies and procedures with the source shown; decisions about a person’s care stay with people.
- Medical education Educators can draft revision questions or case summaries from approved teaching material, then review every item before learners see it.
- Fintech & professional services Statements and application forms can be read into your systems field by field, and any value below a set confidence goes to a reviewer.
What’s changing
Shifts to plan for.
July 2026
EU high-risk rules move to 2027
The AI Omnibus entered into force on 27 July 2026, moving obligations for Annex III high-risk systems, such as those used in employment, credit scoring and education, to 2 December 2027.
August 2026
EU AI transparency duties apply
Article 50 obligations, such as telling people when they are dealing with an AI system, apply from 2 August 2026.
February 2026
New UK rules on automated decisions
The Data (Use and Access) Act 2025 widened when significant decisions can be automated, provided people are told, can contest the outcome and can ask for a human review.
Process
How the work runs.
Define the task precisely
We write down the input, the expected output and what counts as a wrong answer for this use.
Build the example set
Real cases are sampled across the range you see, with correct answers agreed by people who know the work.
Score, then read failures
Results are scored automatically where possible, and a person reads the failed cases to find patterns.
Automate the rerun
The evaluation is scripted into your release process, and results are stored so changes over time are visible.
Cost
What shapes the estimate.
Agreement on answers
Tasks where experts disagree about the right output take longer to label.
Scoring method
Exact-match checks are cheap to run, but judging written answers needs a rubric and human review.
Approaches compared
Each model or configuration in the comparison adds runs and analysis.
Ways to work together
A first, fixed stage
Discovery, a prototype or a technical review with agreed outputs, so the next decision is made with something concrete in hand.
Delivery in phases
The build split into releases, each with a scope you review and approve before it starts.
Ongoing engineering
An agreed scope for support and improvements after launch, with response expectations written down.
Common questions
Before you get in touch.
Can you evaluate an AI feature we already have?
Yes. We build an evaluation set around what it is meant to do and test the current version, which gives you a baseline for any changes.
What if we don’t have labelled examples?
We can help create them from past work, with people who know the task agreeing the correct answers. That effort is part of the scope.
What affects the effort?
The number of tasks, how hard it is to agree a correct answer, the volume of examples needed and how many approaches you want compared.
Further reading
- Answers staff can trust: an assistant built on approved documents How to build a document assistant that cites its sources, respects permissions and says when it does not know.
- Where AI fits: choosing a first task in an operations workflow How to choose a first AI task that is bounded, checkable and worth measuring, and when not to use AI at all.