How to evaluate an AI voice agent: a buyer’s checklist

A practical checklist for buyers — past voice quality, into the console, the handoff, the write-back, and the compliance record that separate a product from a demo.

Most voice-agent demos sound great, because voice quality is nearly solved and every vendor leads with it. That makes the demo the least useful part of an evaluation. The questions that actually separate products are about what happens after the call connects — whether an operator can see what happened, fix what is wrong, and prove an outcome landed. Here is what to test.

Don’t score the voice; score the console

Assume the voice is good — it usually is. Then ask who the product is built for. A console built for the engineer who built the bot means every change is a ticket. A console built for the operator who owns the phone line means the person responsible for outcomes can act. Ask to change a script live in the demo, test it, and ship it. If that takes a vendor ticket, you have found your first constraint.

Test the handoff, including the failure case

A real agent hands off to a human the moment anyone asks. But the part vendors gloss over is failure: what happens when the human doesn’t pick up? A system that emails call context minutes later has been fairly called a glorified voicemail system. Insist on a warm transfer and an explicit report when the human didn’t answer, so no caller silently falls through.

• Ask “can I talk to a person?” and confirm the transfer is immediate
• Force the human to not answer, and see whether the system tells you
• Check that the caller isn’t dumped or looped when the handoff fails

Demand the write-back and a verifiable id

A call that doesn’t update your system of record is a call that didn’t happen, operationally. Ask how outcomes are written back and whether each one carries a verifiable id you can trace. Without that, you cannot prove a booking was made, a promise-to-pay was logged, or a claim was opened — and you will be reconciling by hand.

Read the compliance record

Regulated calling lives or dies on the record. The agent should disclose it’s AI, capture recording consent where required, never take a card by voice, and maintain a ledger of consent, disclosure, recording, and cross-channel contact frequency. Ask to see that ledger. If the vendor can’t show you a verifiable audit trail, assume there isn’t one.

• AI self-disclosure at the top of the call
• Recording consent, all-party where the jurisdiction requires it
• Never captures a card by voice — a secure payment link instead
• A contact-frequency ledger across every channel, not just this one

The demo proves the voice; it proves nothing else. Score the console, the handoff failure case, the write-back id, and the compliance ledger — those decide whether you bought a product or a good-sounding demo.