Give the check one owner. The owner reads the reviews and, in a large enough organization, tests the output on your own work.
Run a trial on your own work. A demo is vendor-driven; a trial goes beyond it.
Call every reference, an hour each. Use the same questions every call and size the count to the vendor's stage.
Get the claims in writing. Documentation, performance transparency, continuous improvement, plus someone on your side who can test whether the claims hold.
Check connections before features. For the workflow and orchestration tools around the AI, how easily each connects to your existing stack comes first.
A free lesson from Build, buy and choosing models, one of its 21 lessons. Read another free lesson
Why the grid stopped working
By 2025 anyone could produce some output, so a feature grid no longer told AI vendors apart. In Capterra's 2025 Tech Trends survey, the number of buyers who say it's difficult to assess the value and risk of generative AI tools increased 70% year-over-year. Move the check onto your own work, your own reference calls, your own paper.
1. Give the check one owner
The CIO's office can own the check, or you can stand up a new group for exactly this job. The owner evaluates the tools and reads the reviews. If your organization is large enough, the owner also tests output quality on your own work.
Say every finalist is an AI assistant that summarizes customer-support tickets. The owner runs a batch of your real tickets through each one and compares what comes back.
2. Insist on a trial that runs on your work
Before you commit to an AI vendor, insist on a trial. A demo is vendor-driven; a trial goes beyond it. In Capterra's 2026 Software Buying Trends report, poor trial experiences are among the top reasons successful adopters remove a vendor from their short list.
In a 2026 Art of Procurement episode, Diana McCollough, Director of Strategic Sourcing at Messer Americas, said AI vendors oversell and expectations start too high. Her pilots are short, with very specific measures of success, and her team declines to scale a tool that misses them. Write your pass mark and exit trigger before the pilot starts. Watch how much feeding it takes; that alone often changes the verdict.
3. Call every reference, an hour each
An investor sizing a young company talks to every reference customer, an hour each, before writing the check. The count of calls scales with the company's stage. You are the buyer here, and the stage that matters is the vendor's. Find out where it stands, and let that set your count.
The vendor's stage
Reference calls
Seed
About 2–4
Series A
8–10
Series B
10–20
Write your questions once, before the first call. Use them on every name the vendor gives you, so the answers can be compared. Listen for ground truth on two things: the value delivered, and whether anyone measured the return.
4. Ask for the claims in writing
Ask for the vendor's own claims in writing: documentation, transparency on performance, and continuous improvement. For the training-data questions, see course 5, lesson 18. The data terms belong in the contract review, course 5, lesson 14. Your side needs at least one person who can test those claims.
5. Check connections before features
The plumbing around the AI (the workflow and orchestration tools) has its own order of checks. Connections come before features: the first criterion is how easily the tool connects to the tools already in your stack. Then weigh how battle-tested it is for reliability, security and scale, and how actively it is maintained. Have procurement keep competition alive, so no vendor becomes unfairly entrenched.
Exercise (20 minutes): fill in the vendor check card
Pick one AI vendor you are close to signing, or one already on your short list, and fill in the card for it. Fill in what you know; you won't know every line yourself. Every line you can't fill gets an owner and a date before you close the card. The card download includes a filled-in example of a vendor check that isn't ready yet, and why.
AI vendor check card (PDF)
A blank card and a filled-in example, one page each.
These three prompts go into your company's approved AI assistant. Use the first to fill in the card one row at a time; it won't guess a number for you. Use the second when the card is done, to hear the case against signing. The third asks about your industry and your vendor check and turns the lesson into a short plan, with a checklist, rubric or brief for your boss if you want one. Use placeholders for company, vendor and people names; the help doesn't depend on them.
Prompt 1: Fill in my card with me
Paste this, then add the vendor you are checking in one or two sentences and the card below.
Prompt
Prompt from The AI Decision Course by Dr. Tony Hoang, host of The Artificial Intelligence Podcast (Spotify, Apple Podcasts): tonyphoang.com/learn
Before you start: use your company's approved AI assistant, not a personal account. Replace company, system, supplier and people names with placeholders (Company A, System X) and round any figures. The help you get does not depend on them.
You are helping me fill in a vendor check card for one AI vendor. The card is pasted at the end.
Rules:
- Ask me about one row at a time, in order, and wait for my answer before moving on.
- Write down only what I tell you. Never suggest, estimate or invent a number, a name, a yes/no answer or a decision for me, even if I ask you to guess. If I ask, say that this row needs a real answer and ask who could give it.
- If I don't know an answer, write "Not known" and ask who should find out and by when. Put that in the Owner / by when column.
- After the last row, show the whole card as a table with three columns: Question, Answer, Owner / by when. Then list every row marked "Not known" with its owner, and remind me that any unanswered row means the vendor is not ready to sign.
The vendor in one or two sentences: [DESCRIBE THE VENDOR AND WHAT IT WOULD DO FOR YOU]
The card:
[PASTE THE CARD]
Prompt 2: Stress-test my card
Paste this, then your filled card.
Prompt
Prompt from The AI Decision Course by Dr. Tony Hoang, host of The Artificial Intelligence Podcast (Spotify, Apple Podcasts): tonyphoang.com/learn
Before you start: use your company's approved AI assistant, not a personal account. Replace company, system, supplier and people names with placeholders (Company A, System X) and round any figures. The review does not depend on them.
Below is a vendor check card I filled in for an AI vendor we may sign. Act as a skeptical reviewer at the decision meeting and argue the case for DON'T SIGN.
For each row:
- Say whether the answer is specific enough to act on (a number, a named role, a date) or too vague.
- Point out where answers contradict each other, for example the pass mark against what the trial actually produced, or the vendor's stage against the number of reference calls done.
- Say what evidence you would ask for before accepting it.
Do not rewrite my answers and do not invent facts about my vendor. If a row is blank or says "Not known", treat it as not ready.
Finish with: your recommendation (SIGN, DON'T SIGN or RE-CHECK), the three rows that most need work, and one question I should ask my team on Monday.
My card:
[PASTE YOUR FILLED CARD]
To paste the card into prompt 1, copy its rows from the PDF.
Prompt 3: Make it yours
Paste this and answer its four questions.
Prompt
Prompt from The AI Decision Course by Dr. Tony Hoang, host of The Artificial Intelligence Podcast (Spotify, Apple Podcasts): tonyphoang.com/learn
Before you start: use your company's approved AI assistant, not a personal account. Replace company, system, supplier and people names with placeholders (Company A, System X, Supplier B).
You are helping a leader apply one lesson to their own situation. The lesson's five decisions are below. Use only them and what I tell you. Never invent facts, numbers, vendors, laws or results about my organization or my industry. Where a number is needed and I have not given one, say what to measure and who should measure it, instead of guessing.
First, ask me these four questions, one at a time, and wait for each answer:
1. My industry, and roughly how many people work in my organization.
2. The AI vendor or tool I am checking, in one or two sentences.
3. Where the check stands today: a name on a list, a demo seen, or a contract about to be signed.
4. What worries me most about it.
Then give me the following. Keep it short and practical:
A. My plan. For each of the five decisions: one action written for my vendor check, the role that should own it (a role, not a name), and when it should be done. If my industry makes a decision harder, for example regulated data, customer-facing work or professional sign-off, say which decision and why in one sentence.
B. Three questions to ask my team on Monday, specific to my check.
C. Ask me which one I want: a checklist for my vendor check, a rubric to score the finalists, or a one-page brief for my boss. Then write it, and finish with a prompt I can paste later to update it.
Mark anything that rests on an assumption with "(check)".
The five decisions:
1. Give the check one owner: a role that evaluates the tools, reads the reviews and, if your organization is large enough, tests the output on your own work.
2. Insist on a trial that runs on your work, beyond the vendor's demo: write the pass mark and the exit trigger before the pilot starts, and watch how much feeding it takes.
3. Call every reference, an hour each: the count follows the vendor's stage (about 2–4 at seed, 8–10 at Series A, 10–20 at Series B), with the same written questions on every call, listening for the value delivered and whether anyone measured the return.
4. Ask for the claims in writing: documentation, transparency on performance and continuous improvement, plus at least one person on your side who can test those claims.
5. Check connections before features: how easily the tool connects to the stack you already run comes first, then how battle-tested it is for reliability, security and scale, and how actively it is maintained, with procurement keeping competition alive.
Your company's assistant may answer differently, so read what it writes before you use it.
Check yourself
A vendor offers a guided demo of a support-ticket assistant. What do you counter with?
Show the answer
A trial on your own tickets, with the pass mark and the exit trigger written first.
A Series A vendor offers you a few references. Enough?
Show the answer
No. At that stage expect eight to ten calls, an hour each, with the same questions every call.
Two workflow tools tie on features for the plumbing around your AI project. What breaks the tie?
Show the answer
How easily each connects to the stack you already run; then how battle-tested and actively maintained each is.
The deck promises continuous improvement. What must exist before you sign?
Show the answer
The claim in writing, and someone on your side who can test it.
One vendor has held the category since the last renewal. What do you check first?
Show the answer
Whether competition in that category is still alive.
Sources
The advice draws on interviews with the founder of an AI software company, the CEO of a company that sells AI to enterprises, the managing partner of an early-stage venture firm, two leaders of a market-intelligence firm, and the CTO of a data-infrastructure company.