Twelve questions to ask an AI vendor before signing
Vendor selection is consequential. MIT's GenAI Divide (July 2025) found externally built tools succeeding roughly twice as often as internal ones, which makes the choice of external partner one of the higher-leverage decisions in an AI programme. Gartner, in June 2025, estimated that only around 130 of thousands of self-described agentic AI vendors were genuinely building agentic systems, and named the rebranding practice "agent washing".
Twelve questions. Each includes what a good answer sounds like, and what evasion sounds like.
1. What is your straight-through processing rate in production, on documents or cases like ours? Good: a specific number, with the document type, the volume, and the conditions under which it was measured. Evasive: a field-level accuracy figure offered instead. Accuracy is a component; STP is the outcome.
2. What is your exception rate, and what happens to an exception? Good: a rate, a routing rule, and a description of the reviewer's interface. Evasive: "exceptions go to your team." That is your cost, undisclosed.
3. What is the per-transaction cost at our volume, including retries and failures? Good: a model showing cost at three volume points, with the assumption about difficulty mix stated. Evasive: a per-seat or per-month price with no unit economics. Gartner named escalating costs first among the reasons agentic projects get cancelled.
4. Where does inference run, and what leaves our network? Good: a specific architecture — in your cloud tenant, on-premise, or a named region — with the data flow described. Evasive: a contract clause quoted in place of an architecture. "Your data doesn't train our models" is an architectural property, not a promise.
5. What is logged, at what granularity, and for how long? Good: input, model and rule versions, tools called, confidence, reviewer identity, timestamp, action taken, retention period. Evasive: "full audit logging" with no field list.
6. What enforces the system's boundaries? Good: scoped credentials, allowlisted actions, approval gates before consequential writes. Evasive: a description of prompt instructions. A prompt is a request, not a control.
7. Show me a production failure and what happened next. Good: a specific incident, the detection path, the remediation, and what changed afterwards. Evasive: no example available. A vendor with production deployments has incidents; one who claims none has neither.
8. What is the rollback path, including for actions already actioned downstream? Good: code rollback, data rollback, and a reconciliation procedure for downstream effects. Evasive: answering only the first. The third is the one that turns a rollback into a project.
9. When a human corrects the system, what changes? Good: a described feedback path with a cadence for retraining or rule updates. Evasive: corrections are stored. Stored is not learned, and this is precisely the "learning gap" MIT identified behind stalled pilots.
10. Who owns the model, the prompts, the rules and the integration code at the end of the contract? Good: clear allocation, in writing, with an exit provision. Evasive: deferral to the master agreement. Ask before signing, not at renewal.
11. What certifications do you hold, and what do they cover? Good: named standards with named scope. If ISO/IEC 42001 is offered, note what it is: published in December 2023 as the first international AI management system standard, structured on a plan-do-check-act cycle, and certifiable by accredited bodies. It is a management-system standard, not a technical specification for an individual system, so it does not by itself establish EU AI Act compliance. A vendor who claims otherwise is either mistaken or hoping you are. Evasive: certification named without scope.
12. What would you tell us not to automate? Good: a specific answer, ideally with an example from another client. Evasive: everything is a candidate. Gartner's own analysis noted that many use cases positioned as agentic do not require agentic implementations; a vendor who has never declined work has not been thinking about your return.
Ask us these too
We would fail a strict reading of some of these, and it is more useful to say so than to pretend otherwise. Our published outcome figures are anonymised, our sector scenarios for logistics, healthcare and financial services are labelled as illustrative because they are not delivered work, and our per-seat and per-transaction models are built during discovery rather than quoted from a rate card.
The twelfth question is the one we would most want a buyer to ask us, because the answer is in writing in every roadmap we produce: a section listing the workflows where we recommend against automation, and why.