If you want ChatGPT, Claude, Perplexity and Google's AI features to be able to name and cite your pages, their crawlers have to be able to fetch them. Each provider now runs more than one bot, and they do different jobs: one builds a search index, one fetches a page when a user asks a question, and one collects text for training models. You can allow the first two and block the third, and many sites do exactly that.
This guide lists each crawler as its provider documents it on 2026-10-20, what blocking it changes, and a robots.txt you can copy.
We make Seen42, an AI visibility tool, so read our product notes with that in mind. Everything else in this guide works without it.
The crawlers, by provider
OpenAI documents four user agents. OAI-SearchBot is "used to surface websites in search results in ChatGPT's search features"; OpenAI says sites that block it "will not be shown in ChatGPT search answers, though can still appear as navigational links". GPTBot collects content that may be used to train OpenAI's models, and disallowing it "indicates a site's content should not be used in training generative AI foundation models". ChatGPT-User fetches a page when a user asks ChatGPT a question; because the visit is user-initiated, OpenAI says robots.txt rules "may not apply", and this agent "is not used to determine whether content may appear in Search". OAI-AdsBot checks pages submitted as ads. The settings are independent: you can allow OAI-SearchBot and disallow GPTBot. Changes take about 24 hours to reach search.
Anthropic documents three. ClaudeBot collects web content that may be used to train Anthropic's models. Claude-SearchBot indexes the web for Claude's search results; Anthropic says blocking it means your content "isn't indexed for search, which may reduce your visibility". Claude-User fetches pages when a Claude user asks a question. Anthropic says its bots honour robots.txt and support the non-standard Crawl-delay directive, and publishes its IP ranges at claude.com/crawling/bots.json. It advises against blocking by IP, because that can stop the bots from reading your robots.txt at all.
Perplexity documents two. PerplexityBot is "designed to surface and link websites in search results on Perplexity" and "is not used to crawl content for AI foundation models"; Perplexity recommends allowing it. Perplexity-User fetches a page when a user asks a question, and "since a user requested the fetch, this fetcher generally ignores robots.txt rules". Changes take up to 24 hours.
Google uses Googlebot for Search, including AI Overviews and AI Mode. Google says "there are no additional requirements to appear in AI Overviews or AI Mode" and that robots.txt rules for Googlebot are the control. Google-Extended is a separate token that controls whether your content may be used for training Gemini models and for grounding in Gemini apps and Vertex AI; Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". Blocking Google-Extended does not remove you from AI Overviews, and allowing it does not get you in.
What each choice does
| You want | Allow | Blocking it means |
|---|---|---|
| Appear in ChatGPT search answers | OAI-SearchBot | Not shown in ChatGPT search answers |
| Appear in Claude search results | Claude-SearchBot | Not indexed for Claude search |
| Appear in Perplexity answers | PerplexityBot | Not surfaced in Perplexity results |
| Appear in Google AI Overviews and AI Mode | Googlebot | Not in Google Search at all |
| Let assistants open your page for a user | ChatGPT-User, Claude-User, Perplexity-User | Mostly ignored; user requests may still be fetched |
| Keep your text out of training sets | Disallow GPTBot, ClaudeBot, Google-Extended | No effect on search or AI Overviews, per the providers |
A robots.txt that allows search and blocks training
This file lets every search crawler and user-initiated fetcher in, and asks the three training crawlers to stay out. Adjust the Disallow lines for your own private paths.
User-agent: *
Allow: /
Disallow: /app/
Disallow: /login
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Sitemap: https://www.example.com/sitemap.xml
If you are happy for your public pages to be used for training, remove the three blocks; the first rule already allows everything else. Seen42's own robots.txt allows all of these crawlers.
Where sites get this wrong
The most common problem is not robots.txt at all. Some CDNs, firewalls and bot-management products block AI crawlers by default or with a single setting, and the site owner never sees it. Check your CDN's bot rules and your server logs for the user agents above before you touch robots.txt.
The second is blanket blocking. A rule written in 2023 to keep GPTBot out of training sets often reads "User-agent: GPTBot" plus a wildcard that also catches OAI-SearchBot, which did not exist yet. OpenAI's search crawler launched after its training crawler, and many robots files have not caught up.
The third is content that is not in the HTML. A crawler that fetches the page and finds a loading spinner and a script tag has nothing to cite. Keep the text that answers the question in the delivered HTML.
How to check
Fetch your own robots.txt and read it as each bot would. Then search your access logs for OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot over the last 30 days. If a search crawler never appears, it is being blocked somewhere, or it has not found a reason to visit. Google Search Console reports traffic from AI features under the Web search type, which is a slow but reliable signal that Google can reach you.
Where Seen42 fits
Seen42 checks the prompts your buyers ask on ChatGPT, Perplexity, Gemini, Claude and Google AI Overviews, shows who is named and cited instead of you, and builds and hosts comparison, facts and guide pages from your own facts. Paid plans start at $39 a month for 15 prompts; the Free plan checks 10 prompts once on ChatGPT and Gemini. Every site Seen42 hosts ships a robots.txt that allows these crawlers, a sitemap, llms.txt and JSON-LD on each page. Being reachable is the first gate, and the only one we can promise; what the assistants do once they have read the page is measured, not guaranteed.
