When someone claims, “I searched this in ChatGPT and we didn’t show up,” it’s tempting to shrug it off. But in the messy world of AI-powered search, those offhand comments can spark legitimate concerns, and they need a structured response.
I put together a checklist after a board member raised exactly this question. When I emailed it to colleagues, it saved a lot of back-and-forth - so here’s a version for anyone facing similar conversations.
Why Asking the Right Questions Matters
AI chat-based tools like ChatGPT, Perplexity, Gemini, and Copilot are growing fast in visibility. That means marketing should still prioritise traditional SEO - AI chat tools aren’t replacing Google or Bing anytime soon.
Still, someone asking “why don’t we appear in ChatGPT?” deserves a decent response. The same phrasing or tool can produce very different results depending on a few key variables:
- Prompt wording
- Follow-up queries
- Model version and training cut-off
- Any stored memory or past chat sessions
- Plugins, retrieval tools, or browsing enabled
So instead of guessing, we ask the right questions.
Questions to Ask When Auditing an AI Search Claim
Here’s a quick checklist to get the right context before we jump to conclusions:
-
Which AI tool did they use?
ChatGPT, Claude, Perplexity, Gemini, Harvey… If they mention self-hosted tools (e.g. Ollama, DeepSeek) or say things like “13b model,” tell them to keep reading - I explain it below. -
What model version was used?
GPT-4, GPT-4o, Claude 3.5, Gemini 1.5, etc. -
What was the initial prompt?
The first thing they typed or said. Get a copy if possible. -
Were there follow-up questions?
AI is conversational. One follow-up can change the whole thread. - Was there any stored memory or chat context?
If they’re not sure, they can ask the AI:“What context or memory are you using for this answer?”
-
Were there any system settings or preferences active?
Things like tone of voice, safe-search filters, or restricted sources. -
Was it a fine-tuned or company-trained model? If yes, that’s a different game entirely - and probably beyond what most marketers need to worry about.
- Can they share the full transcript or a chat link?
Most paid tools (like ChatGPT Plus) allow sharing.
Model Size ≠ Output Quality
Marketers sometimes hear terms like “7b,” “13b,” or “70b” and assume bigger = better. That’s not quite right.
Those numbers refer to the number of parameters in a model: the internal “knobs” it uses to understand language. A 70b model has more of these than a 13b one. But that doesn’t guarantee better answers.
Think of it like engine size: more power doesn’t always mean a better ride. My son James is a talented racing driver, and I’ve watched him hustle a Citroen C1 around Silverstone against BMW Z4s, Mazda RX8s, and Honda Type Rs. It’s about training, tuning, and context.
Why This All Matters
- Google has begun surfacing Gemini activity in Search Console, but only for AI Overview impressions. I was at Google’s offices in June for the Amadeus Google Summit and spoke directly with a Senior Product Manager about this - it’s early days, and far from comprehensive.
- There are now a few unofficial tools that let you peek into Gemini or ChatGPT behaviour. I spoke with a large SEO agency who’d just built and launched one - they were honest that it’s useful for investigation but fragile, manually operated, and nowhere near scalable.
- Crucially: AI models don’t retrieve results like search engines. They generate responses based on data, prompts, and model behaviour - not a ranked list of pages.
Bottom Line for Marketers
- Traditional SEO still matters most. It powers both human and AI understanding.
- Treat AI visibility checks as qualitative research, not a KPI.
- If someone flags something from an AI chat, ask for the right context before jumping in.
- And if they say “7b” like it’s gospel, smile and tell them to come talk to me.