2 September 2026 · 6 min read
This is the single most common technical misunderstanding in AI search right now, and acting on it incorrectly can remove a business from Google Search altogether.
Short answer
Google-Extended is a crawler control that determines whether your content is used to improve Gemini and Vertex AI. It does not control AI Overviews in Google Search, which are served through ordinary Googlebot indexing. Blocking Google-Extended will not remove you from AI Overviews; blocking Googlebot would, along with removing you from Search entirely.
01 The premise
When Google introduced Google-Extended, a great deal of coverage described it as a way to opt out of Google’s AI. That is half right, and the half that is wrong is the half businesses act on. Publishers concerned about AI Overviews summarising their content read that they could block it, added a rule to robots.txt, and waited for a change that was never going to come.
The distinction is between training and serving. Google-Extended governs whether your content is used to improve Gemini and Vertex AI models. AI Overviews inside Google Search are a Search feature, built on Google’s existing index — the same index that decides whether you rank at all. They are reached through Googlebot, not through Google-Extended.
The practical consequence is that there is no setting which keeps you in Google Search and out of AI Overviews. Blocking Googlebot achieves the second by achieving something far worse: disappearing from Search. Every business that has asked me about this wanted the first and would have been horrified by the second.
02 Crawlers
Worth checking against the official documentation for each provider, because these change.
| Agent | What blocking it does |
|---|---|
| Googlebot | Removes you from Google Search, and therefore from AI Overviews as a side effect. Almost never what anybody wants. |
| Google-Extended | Stops your content being used to improve Gemini and Vertex AI. Does not affect AI Overviews in Search. |
| GPTBot | Stops OpenAI using your content for model training. Does not remove you from ChatGPT’s live browsing. |
| OAI-SearchBot | Removes you from ChatGPT’s search and browsing results. This one does affect what users see today. |
| PerplexityBot | Removes you from Perplexity, which cites sources prominently. Usually worth allowing. |
| ClaudeBot | Anthropic’s crawler. Same reasoning as the others. |
03 Your policy
Four questions that settle your robots.txt policy properly.
— Questions
There is currently no supported way to do that. AI Overviews are built on Google’s Search index, so opting out of them means opting out of Search. Google-Extended does not provide this control despite being widely described as though it does.
Not from live browsing. GPTBot is the training crawler; OAI-SearchBot serves ChatGPT’s search results. Blocking the first affects future model training, blocking the second removes you from answers users see today.
Most businesses that want to be recommended should allow the retrieval crawlers, because you cannot be cited by a system that cannot read you. Training crawlers are a separate judgement with no direct effect on current visibility.
Open yoursite.com/robots.txt in a browser. Many sites carry rules inherited from a previous developer or a plugin default that nobody has reviewed, and some of them block more than the owner realises.
— More
AEO & GEO · 8 min
Almost every argument about generative engine optimisation comes from conflating two completely different things: what a model learned when it was built, and what it fetches when it answers.
ReadSEO · 7 min
It is still one of the most commonly sold local SEO tactics in India, it works briefly, and it is a named spam pattern in Google’s own documentation.
ReadGoogle Ads · 6 min
Almost every underperforming account I open has the same three problems. None of them are exotic, all of them are visible within an hour, and fixing them usually matters more than any bid strategy.
ReadFree consultation · No obligation
The reasoning on this page is public because I would rather be judged on it than on a claim. If you want it applied, that is the service.
Name, number, email, what you need.