| What we tested (Oct 13, 2026) | “Best SEO audit tools for small business” | “Best Shopify SEO apps” | “Best website builder for small business” |
|---|---|---|---|
| Same prompt, asked again and again | 8 runs, 8 different lists | 8 runs, 1 identical answer | 5 runs, 1 identical answer |
| Same intent, reworded 5 ways | 5 different lists | 5 different lists | 3 different lists |
| Named in every single answer | 4 tools | 1 app | 4 builders |
| Linked sites that were also in Google’s top 10 | 24% to 42% | 76% to 100% | 8% to 24% |
Rand Fishkin and SparkToro showed in January that AI tools almost never give the same brand list twice when hundreds of different people ask. I wanted to know something narrower, because it’s how most tracking tools actually work: what happens when one account asks Google AI Mode the same buyer questions over and over, and then rewords them?
Below: how we ran it, the three things that surprised me, what it means before you pay for an AI visibility tool, and a free way to run the same test on your own category.
- Two of our three prompts returned the exact same AI Mode answer on every repeat. The third changed every time. A tracker that only re-runs one prompt can report stability or chaos for reasons that have nothing to do with your brand.
- Rewording the question changed the brand list in 13 of 15 rewordings, but a core set of brands appeared every time.
- How closely AI Mode’s links matched the classic top 10 depended heavily on the category: from 8% to 100% in our runs.
- Track mention frequency across many reworded prompts. Ignore “AI rank position”.
What is AI visibility tracking?
AI visibility tracking is the practice of running a fixed set of prompts through AI tools such as Google AI Mode, AI Overviews, ChatGPT or Perplexity, and recording which brands and websites the answers mention or link to. Most tools turn that into a share-of-voice percentage, and some into a “rank” for each brand.
The idea is sound. The problem is that AI answers aren’t fixed lists. Google says in its AI features documentation that AI Mode and AI Overviews may use a “query fan-out” technique, “issuing multiple related searches across subtopics and data sources” to build a response. So the same intent can be answered from different searches each time.
If you’re new to the measurement side, Google’s own numbers for AI features live in Search Console. I explained how to read them in my guide to the Search Console AI performance report.
Why we ran this test
Three people whose work I follow shaped this test, and they don’t fully agree with each other, which is why it was worth running.
- Rand Fishkin (SparkToro) published research with Gumshoe.ai in which 600 volunteers ran 12 prompts through ChatGPT, Claude and Google’s AI a combined 2,961 times. They found a less than 1 in 100 chance of getting the same brand list in any two responses. His conclusion: visibility percentage across many prompts is a reasonable metric, but a “ranking position in AI” isn’t. Search Engine Journal’s write-up covers the method in short.
- Chris Long (Nectiv) has been studying the searches behind AI answers. He spoke at Ahrefs Evolve this week on an analysis of more than 17,000 ChatGPT fan-out queries. If the answer depends on hidden sub-searches, wording should matter a lot.
- Cyrus Shepard often frames search debates around evidence. This week he was surveying SEOs because, in his words, he wants “to look at what the data actually says.” That’s the spirit here: a small test you can repeat, not a big claim.
SparkToro’s study used hundreds of different people. Most tracking tools use one setup that asks the same prompt on a schedule. So my question was: what does that single-setup view look like?
How we tested Google AI Mode
We ran 36 Google AI Mode searches from one signed-in Google account in Chrome, with US English settings, on October 13, 2026, within about an hour. Each search was a fresh page load, not a follow-up in the same conversation.
| Part | What we did | Searches |
|---|---|---|
| Exact repeats | The same prompt, word for word, run again and again | 21 (8 + 8 + 5) |
| Rewordings | Five other ways a buyer might ask the same thing, for each prompt | 15 |
| Classic results | The top 10 web results for each original prompt, for comparison | 3 (not counted in the 36) |
The three prompts are things our clients’ customers actually ask: “best seo audit tools for small business”, “best shopify seo apps” and “best website builder for small business”. Rewordings included “which shopify app should i use for seo” and “recommend a website builder for my small business”.
From each answer we recorded the brands named (in bold text or in the comparison table), the order they first appeared and every external site the answer linked to. We matched brands by name, by hand checked the list, and dropped generic phrases.
Finding 1: the same prompt can give a frozen answer, or a new one every time
Repeating a prompt in Google AI Mode didn’t behave the same way across our three prompts. Two returned the identical answer on every repeat: same brands, same order, same length. The third returned a different list every time.

- “Best Shopify SEO apps”: 8 out of 8 runs returned the same five apps in the same order: Tiny SEO, SEO Lab, SearchPie, SEOAnt and Booster AI SEO.
- “Best website builder for small business”: 5 out of 5 runs returned Wix, Shopify, Squarespace, Hostinger and WordPress, in that order.
- “Best SEO audit tools for small business”: 8 runs, 8 different lists, between 8 and 11 tools each, in a different order every time. On the first run the page said it was “Thinking a little longer” before answering.
Google doesn’t explain when AI Mode reuses an answer and when it writes a new one, so I won’t guess at the cause. What matters is the effect on tracking.
Finding 2: rewording changes the list, but a core set survives
Rewording a prompt changed which brands Google AI Mode named in 13 of our 15 rewordings. Only two rewordings, both for website builders, returned the same five builders as the original. Yet in every category, a small core set of brands showed up no matter how we asked.
| Prompt family | Named every time | Sometimes | Different brands seen |
|---|---|---|---|
| SEO audit tools (13 asks) | SEOptimer, SE Ranking, Screaming Frog, Google Search Console | Semrush, Ahrefs, PageSpeed Insights, Bing Webmaster Tools, WooRank and others | 14 |
| Shopify SEO apps (6 asks) | Tiny SEO | SearchPie, Booster AI SEO, Plug in SEO, Yoast SEO for Shopify, Avada, Smart SEO and others | 11 |
| Website builders (6 asks) | Squarespace, Shopify, Wix, Hostinger | WordPress, GoDaddy, one other | 7 |
This lines up with what SparkToro found at a much bigger scale: individual answers vary a lot, but how often a brand appears across many answers is far more stable. The Shopify category was the loosest. Two apps from the “frozen” original answer, SEO Lab and SEOAnt, didn’t appear in any of the five rewordings.
Finding 3: AI Mode’s sources only sometimes match the top 10
The websites Google AI Mode linked to overlapped with the classic top 10 results to very different degrees depending on the category. That fits Google’s own description: AI features can show “a wider and more diverse set of helpful links” than a classic web search.
| Prompt family | Linked sites also in the top 10 (original prompt) | Same, for rewordings |
|---|---|---|
| Shopify SEO apps | 100% (6 of 6, every repeat) | 76% (32 of 42 links) |
| SEO audit tools | 42% (27 of 65 links across 8 runs) | 24% (13 of 55) |
| Website builders | 8% (2 of 24, every repeat) | 24% (12 of 49) |
For rewordings we compared against the top 10 of the original prompt, so part of the drop is simply that a reworded search has its own results. Still, the lesson holds: ranking in the top 10 helps in some categories much more than others, and AI answers also lean on sites that don’t rank on page one at all.
That’s why strong classic SEO still matters for AI visibility, but it isn’t the whole job. Which brings me to what I’d actually measure.
What to track instead of “AI rank”
The metric to track is how often your brand appears across a large, varied set of prompts, measured over time, next to your normal search data. Here’s how I’d weigh what most AI visibility tools report:
| Metric a tool shows you | Trust it? | Why |
|---|---|---|
| Your “position” in an AI answer | No | Order changed on every regenerated answer in our test |
| Day-to-day change on one prompt | No | Could be a stored answer or a new one; you can’t tell which |
| Mention rate across 50+ reworded prompts, monthly | Yes, as a trend | The core-set pattern held in every category we tested |
| Which sites AI answers cite in your category | Yes | Tells you where to earn coverage and links |
| AI impressions in Search Console | Yes | Google’s own data, by page (impressions only) |
If you want help building a prompt set and turning it into content and PR priorities, that’s what our generative engine optimization (GEO) work covers.
How to run this test on your own category (free, about an hour)
You can run a small version of this AI visibility test yourself with a browser and a spreadsheet. This is the routine we used:
- Write 3 buyer prompts in your customers’ words (“best [category] for [type of business]”).
- Write 5 rewordings of each. Ask a colleague to write two of them; people phrase things differently than you think.
- Run each original prompt 5 times in Google AI Mode, each as a fresh page, not a follow-up.
- Run each rewording once.
- Log the brands named and the sites linked in a sheet: one row per answer, one column per brand.
- Check whether repeats were identical. If they were, count them once.
- Calculate mention rate: answers that named you divided by answers in that prompt family.
- Note which cited sites cover your category but not your brand. That’s your outreach list.
- 3 prompts written in customer language
- 5 rewordings per prompt
- Fresh page for every search
- Identical repeats counted once
- Brands and linked sites logged per answer
- Mention rate per prompt family
- Core set vs “sometimes” brands marked
- Cited sites without your brand listed for outreach
- Same test repeated next month
And start with the basics on your own site: AI features draw on Google’s index, so pages that aren’t indexed or are slow and thin won’t help you. Our free SEO audit checks that, including an AI citability score.
Questions to ask before you buy an AI visibility tool
Before paying for an AI visibility tracking tool, ask the vendor these questions. A good vendor will have clear answers.
- How many times do you run each prompt, and do you report the spread or just one result?
- Do you detect identical, reused answers and count them once?
- How many rewordings per topic do you track, and can I add my own?
- Do you report “position”? If so, how do you handle answers that reorder every run?
- Which accounts, locations and devices do your runs use?
- Do you show the sites each answer cites, not only brand mentions?
FAQ
Is AI visibility tracking worth paying for?
It can be, if the tool runs many varied prompts many times and reports mention rates as trends. It isn’t worth paying for a single “AI rank” or daily swings on one prompt. Our test showed one prompt can look frozen while another changes every run.
Does Google AI Mode give the same answer every time?
Sometimes. In our test, two of three prompts returned an identical answer on every repeat, while the third gave a different list each time. Rewording the question changed the brands in 13 of 15 cases.
How do I check if my brand shows up in Google AI Mode?
Search your buyers’ real questions in AI Mode, each as a fresh page, plus several rewordings, and note if your brand is named. For Google’s own data, check the Generative AI report in Search Console, which shows AI impressions by page.
Do AI Mode answers use the same sites as the top 10?
Partly. In our test the overlap ranged from 8% to 100% depending on the category. Google says AI features can show a wider set of links than classic search, so ranking well helps but isn’t the only route.
What is a good AI visibility metric?
Mention rate across a large, varied set of prompts, tracked monthly, plus the list of sites AI answers cite in your category. Both point to actions: content to create and sites to earn coverage on.
Related: GEO services, digital PR and SEO services.

