The Evaluation Process
Not a black box. A scored process.
Every solution we evaluate is scored against the same five dimensions. Here is exactly what we look at — and what gets something rejected immediately.
The evaluation exists because most AI and operational solutions that reach a client recommendation have passed through a vendor sales process designed to minimise friction — not identify fit. We run a separate process. Independent of vendor input.
The five dimensions below are not a checklist. They are the structural questions we ask before any solution is introduced into a client context. A fail on a Critical dimension ends the evaluation — there is no weighting that compensates for a fundamental deployment problem.
Scoring Dimensions
What we score a solution on.
Can the solution operate outside a controlled demo environment? We test against real operational conditions — inconsistent data, user variability, edge cases that vendors don't cover in their pitch.
Does it behave consistently when inputs vary from expected format?
What happens when an edge case occurs — does it fail gracefully or break?
How long before it requires significant re-configuration in a live environment?
Passes when
Passes when it holds up under documented edge cases and doesn't require continuous vendor intervention post-deployment.
Fails when
Fails when it requires prompt engineering maintenance, clean structured data that doesn't exist, or breaks under real conversation or workflow load.
Non-Negotiable
Immediate disqualifiers.
These conditions end an evaluation regardless of performance on any other dimension. They are not negotiable and they do not have workarounds.
Who Scores It
Not a committee.
Not procurement.
The evaluation is conducted by the same person managing the engagement. No intermediary layers. No procurement committee that doesn\'t understand the operating context. The person who owns the diagnosis runs the evaluation.
Lead Evaluator
The same person who manages the client engagement. Not a procurement team. Not a vendor-facing account manager. The evaluator has operational context for the specific mandate.
Live Environment Test
Where possible, solutions are tested against a simplified version of the client's actual operating conditions — not the vendor's demo environment.
Written Verdict
Every evaluation that reaches scoring produces a written verdict — pass, conditional pass, or fail — with rationale against each dimension. This is available to the client.
Want to know if a specific solution passes?
Submit a mandate through the intake process and we will run it through this framework. If the solution doesn\'t pass, we will tell you before you allocate budget to it.