A study by SparkToro makes the point that AI “answers” are for the most part random making AI visibility KPIs based on citations and mentions a poor indicator of an SEOs performance!
Why do AI KPIs Need to be Re-evaluated?
- Most AI platforms provide no prompt or click data to use in calculating/analysing performance
- Analytics platforms have problems identifying clicks from AI providers and often identify as direct traffic
- AI results are random/probablistic so tracking attribution in results as KPIs was ill conceived
- Counting citations and mentions in results or doing large numbers of prompts to get visibility share are at best guesstimates
No Prompt or Click Data to Calculate AI KPIs
AI KPIs based on SEO metrics like ranking reports were ill conceived because there is no data to back them up and AI results are too random. You could own the AI result and still not be mentioned or cited.
SEO KPIs like traffic, CTR and rankings have end to end data points. SEO KPIs started with keyword research (query volume) and every click from the SERP to conversion was counted and auditable.
Analytics Platforms have Problems Identifying Clicks from AI Providers
Lately there have been some regex and MCP’s shared that do help in tracking clicks and other data in GA4 and GSC, however, there’s no “for sure” referrer data outside of web logs. Seems like a lot of effort to measure clicks and related data from minor players in search.AI Results are Random so Tracking Attribution as KPIs was Ill Conceived
Venture Capitalists are over represented in the AI Visibility and SEO industry. I believe most AI visibility tools were ill conceived by VCs fashioning AI visibility tools on deterministic platform metrics (rankings).
AI visibility tools are measuring AI attribution by what is displayed in the result not really knowing all contributors to the training and grounding data that goes into results!
Citations and Mentions or doing Large Numbers of Prompts to get Visibility Share are Guesstimates
Practioner service KPIs shouldn’t be guesstimates! They should be based on auditable data! In my business I went so far as to not only provide the performance data I also provided an accounting for the activity on every hour I invoiced clients for.
The method of doing a large number of prompts to determine visibility share is the biggest guesstimate of all because you don’t even know if users are using the prompt since AI platforms don’t provide that data. Lastly Google has by far the most AI searches and and say queries are getting longer, however, user habits die hard so a wide majority still use traditional keyword phrases!
Alternative AI KPIs
An AI reference that has a Good List of KPIs for Multiple Environments
The best reference I’ve found for alternative AI KPIs is Duane Forrester’s book The Machine Layer. He outlines 12 on his substack in 12 New KPIs for the GenAI Era: The Death of the Old SEO Dashboard. Currently many of the suggestions have no tools to gather the data points needed for the KPIs to monitor.
Current Available AI KPIs
For the most part the only AI platforms providing any data for KPIs is Google and Bing through AI reporting in Google’s GSC and Bing’s Webmaster Tools. Google only provides:
- pages
- impressions
Bing’s Webmaster Tools provides data for both AI search and Copilot:
- grounding queries
- total citations (site)
- page level citations
- daily average of cited pages
- visibility trends (tracking variations in citation frequency over time)
Traditional Search KPIs Applicable to AI
Below are discovery KPIs that can be monitored through GSC and Bing Webmaster Tools:
- is the content crawlable
- is the content indexable
- is the content added to the index
- is JavaScript or AJAX indexed
In Google Search Console run pages through the URL Inspection Tool at the top of the console. Click crawled page and inspect the html for any content loaded by JS or AJAX. In general client side execution of JS and AJAX will fail for AI crawlers.
AI KPIs for Measuring Relevance Used BY AI Platforms
There are a few KPIs for scoring use of vector embeddings and relevance that will help with inclusion in AI results:
- Semantic density score
- Vector index presence rate
- Embedding relevance score
- Chunk retrievable frequency
- Chunk extraction rate
Semantic Density Score
Semantic density scores calculate the ratio of explicit non-redundant information and concepts relative to content length. Semantic density plays a role in LLM attention mechanisms prioritizing passage “chunks” with high semantic density because they lower cognitive extraction uncertainty. Pages with high semantic density are more likely to be chosen from the vector pool (RAG fan-out results) into “grounding chunks”, mentions and citations in AI results.
Vector Presence Rate
Vector presence rate (VPR) measures how frequently a brand, domain or core concept appears within a RAG retrieval pool during grounding. Unfortunately there is only one AI platform providing this data. Google’s Gemini Grounding API which is the data pipleine used by this site to measure visibility and citation possibility share.
You should measure VPR at the page level and domain level at least monthly but I’d recommend weekly updates on keywords and prompts that have a VPR score. IMO, this is the only true measure of visibility and conversation share. Our data isn’t scrapped it is calculated down to the byte and comes directly from Google who own the greatest share of AI search!
Embedding Relevance Score
Embedding Relevance Score (ERS) is mathematical calculation that quantifies how conceptually close content is to the users query. ERS is important for RAG vector engines because it is used to determine whether to pull content into the LLM’s context window during the initial search pass.
Chunk Retrieval Frequency
Chunk retrival frequency (CRF) is a measurement of how often an AI platform retrieves chunks from your content for a group of similar prompts. Duane suggested using LngChain. I would prefer to use a combination of similar prompts and grounding fan-out queries run through the Gemini Grounding Expolorer. This method reveals whether your passage is a systemic fixture in the vector index or just an isolated match for a single long-tail phrasing.
Chunk Extraction Rate or Passage Yield
The Gemini Grounding explorer also calculates “chunk” extraction rate or passage yield to determine conversation or visibility share. This calculation is done at the byte level by counting the characters in each “grounding chunk” and dividing it by the total number of characters in the result. Like CRF it can also be calculated for topic clusters doing the same calculations over a topic cluster rather than a page.
Conclusion: Why Changing AI KPIs is Needed
The toughest part of any practioners job is managing client expectations! First explain to clients that AI platform results are probablistic rather than deterministic which means that using traditional search KPIs isn’t a true reflection of the work you performed. Introduce them to these new KPIs and why they are important for getting cited and mentioned in AI results.
Clients have a hard time believing your performance was adequate when the metrics they understand like clicks, conversion and traffic are decreasing and trending lower. Clients need to understand the game has changed so your performance must also be gauged differently. By explaining how AI platforms choose which sites to mention and cite. Clients understanding that these are the new tools and KPIs to grade performance. At that point managing client expectations will be much easier resulting in less client churn.
