Five free questions separate agencies running real AI systems from agencies with "AI-powered" on a website: read or write access to your accounts, per-person query logging, a written rulebook defining every reported number, a named verification gate before numbers reach you, and one live definition test. Real systems answer each in a sentence.
"AI-powered" is a claim on a website. Whether anything sits behind it is knowable in one call, because real systems leave evidence and claims do not. These five questions are free to ask, take ten minutes, and every one of them has a short answer if the system exists. Long answers are data too.
Why it matters: an AI that can look is a diagnostic tool; an AI that can change bids, budgets, or campaigns is a different risk class entirely, and you deserve to know which one is attached to your money. What a good answer sounds like: a one-sentence policy, plus the enforcement mechanism, technical scope or permission level, not a promise of restraint. Red flag: the question has never come up before.
Why it matters: "the system pulled it" is not accountability. When a number in your report is wrong, someone specific ran the query that produced it, and the log either says who or it does not. What a good answer sounds like: yes, per-user, and here is what the log captures. Red flag: logging exists at the tool level but everything runs under one shared login, which means the audit trail names a robot.
Why it matters: the most expensive reporting failures are not bugs, they are undefined terms. A "conversions" figure that silently blends purchases with sign-ups is technically correct and practically fiction, and only a written definition catches it. What a good answer sounds like: a document, per metric, stating source field, inclusions, exclusions, and source-of-truth system. Red flag: the definitions live in someone's head, which means they change when the someone does.
Why it matters: "assigned" and "done" are different claims, and the gap between them is invisible until it costs you. We once pulled an account where conversion goals had been marked complete on a checklist while the live account contained zero of them, with spend already leaning on the assumption. What a good answer sounds like: a named check, on a named cadence, against the live system, before client delivery. Red flag: the verification step is "we double-check things."
Why it matters: this is the live test the first four questions build to, and it cannot be prepared for. An agency with a rulebook answers in a sentence, because the sentence is written down. What a good answer sounds like: immediate, specific, boring. Red flag: the answer starts with "so, basically" and ends somewhere else.
Notice what none of these ask: which model, which tools, how many agents. Vendors can match each other on those in an afternoon. What cannot be improvised is governance, and governance is what these five surface. Any agency running a real system will enjoy the questions. That reaction is itself the sixth signal.
Our own answers, in the open: the access policy and logging are documented on the systems pages the live-query stack hub, the metric rules live on the methodology page, and the verification cadence is described where the catches are. If you would rather have the whole exercise run against your current setup, agency or in-house, that is the AI-Stack Evaluation.
The whole exercise run against your current setup — agency or in-house.
Get in touchIs it rude to ask these?
It is procurement. You are attaching a system to your revenue; asking how it is governed is the minimum, and the agencies with real systems answer these happily because the answers are their differentiation.
What if the answers are all "we're working on that"?
Then you have an honest agency without a system yet, which is a legitimate thing to be. The problem is only when the website said otherwise.
Can I ask these of my current agency?
Especially of your current agency. Question 5 costs nothing and tells you the most.