Does allowing GPTBot and OAI-SearchBot mean ChatGPT will mention my business?

No. It means ChatGPT may read you. Being named depends on being in Bing and its own index, on a first paragraph that answers the question, and on who else mentions you.

Can I refuse training and still appear in ChatGPT answers?

Yes. The six training crawlers in the table and the fetchers and index builders are separate names, and robots.txt treats each name on its own, so refusing one job need not refuse the others. The sites that lose ChatGPT answers over training are the ones where a plugin or a copied list bundled all three of ChatGPT's names into one block.

Do I need to name all 32 when a general allow-everything rule already covers them?

Not for the crawlers; a general rule covers them. Naming them is for you: it makes the decision visible, lets you treat training separately, and stops a future plugin adding a blanket block unnoticed. The risk in a named group written carelessly is that it stops reading the general one, which is where most owners keep the lines that matter.

Which AI crawlers should I allow in robots.txt?

Most sites missing from AI answers never decided this; a plugin or a copied block list decided for them. The live fetchers (ChatGPT-User, Perplexity-User, Claude-User, Meta-ExternalFetcher, DuckAssistBot, MistralAI-User) only arrive when a real person has just asked and the assistant chose your page, so a block on them refuses an audience already at the door. The index builders decide whether you can be found at all. Only the training crawlers are a judgement call. And none of it counts if something in front of your server is overruling the file.

Quotable
A robots.txt that blocks GPTBot but allows OAI-SearchBot keeps you eligible for ChatGPT's search results today and out of the model's memory tomorrow.

Four jobs a crawler can be doing on your page

A crawler is a program that fetches your pages. Each AI company runs several, each under its own name (its user agent) with one job, and robots.txt answers each name separately.

Answering. A person has just asked and the assistant fetched your page, right then, to read it; ChatGPT does this as ChatGPT-User. Each fetch is an answer served: you were read, and usually drawn on, whether or not anyone clicks.
Indexing. Building the assistant's own search index so it can find pages later: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity.
Search. The ordinary search engines. Bingbot and Googlebot build the Bing and Google indexes, which the assistants read too: Copilot and in part ChatGPT look things up in Bing, Gemini and AI Overviews in Google.
Training. Collecting text for the next model; GPTBot is OpenAI's. What it collects now is what the model knows later without looking anything up.

Index first, then most of the answers, is what we have seen. On our address-tracing site, launched on 29 September 2026, OAI-SearchBot took 35 pages on 9 October 2026, and 21 of the 24 ChatGPT answers served in the site's first fifteen days arrived in the four days after that (server logs, 29 September to 13 October 2026). In those same fifteen days, with no backlinks (no other website yet linked to it), the index builders and search engines came anyway: Amazon's crawler fetched 151 pages, Bing's 105 and Google's 58 (same logs). All were allowed by name.

The crawlers, by job

These nineteen decide whether an assistant can read and quote you.

User agent	Who	Job
`ChatGPT-User`	OpenAI, ChatGPT	Answers: fetches a page while ChatGPT replies to someone
`Perplexity-User`	Perplexity	Answers
`Claude-User`	Anthropic, Claude	Answers
`Meta-ExternalFetcher`	Meta AI	Answers
`DuckAssistBot`	DuckDuckGo AI	Answers
`MistralAI-User`	Mistral, Le Chat	Answers
`OAI-SearchBot`	OpenAI, ChatGPT search	Index: builds the index ChatGPT searches
`Claude-SearchBot`	Anthropic, Claude search	Index
`PerplexityBot`	Perplexity	Index
`Applebot`	Apple	Index
`Amazonbot`	Amazon, Alexa	Index
`Bingbot`	Microsoft: Bing, Copilot, DuckDuckGo	Search: Copilot and in part ChatGPT look things up here
`Googlebot`	Google: Search, AI Overviews, Gemini	Search: AI Overviews and Gemini look things up here
`GPTBot`	OpenAI	Training
`ClaudeBot`	Anthropic	Training
`Google-Extended`	Google, Gemini	Training: a token, not a crawler; whether Gemini may train on what Googlebot fetched
`Meta-ExternalAgent`	Meta	Training
`Bytespider`	ByteDance	Training
`CCBot`	Common Crawl	Training: an open web archive many models train from

Block an answers fetcher and that assistant quotes someone else when a person asks about you. Block an index builder and it cannot find you on its own. Block Bingbot or Googlebot and you have left search and every assistant that reads those indexes. Block a training crawler and the next model learns nothing about you from your own site.

Our robots.txt names 32 crawlers. The other thirteen (anthropic-ai, GoogleOther, Google-CloudVertexBot, Applebot-Extended, FacebookBot, DuckDuckBot, cohere-ai, YouBot, Kagibot, PetalBot, Diffbot, Timpibot, AI2Bot) are smaller search engines, Google's and Apple's extra tokens, and AI companies' research fetchers; Google-CloudVertexBot (Vertex AI grounding) began reading investigationbureau.uk in its first eight days (server logs to 13 October 2026). We allow them all.

The GPTBot mistake

The commonest robots.txt change is a two-line block on GPTBot, for a fair reason: the owner does not want their writing used to train a model, and a block on that one name does exactly that. Two things then go wrong.

First, people take GPTBot to be the whole of OpenAI. It is not: ChatGPT reaches your site under three names, GPTBot for training, OAI-SearchBot for its search index and ChatGPT-User when answering someone. Blocking GPTBot alone leaves the other two untouched, but copied block lists and plugins with a "block AI" tick box bundle all three, and a decision about training quietly becomes a decision to vanish from ChatGPT's search results and today's answers.

Second, people assume training has nothing to do with being mentioned. An assistant answers from two places: what it remembers and what it looks up. Training is the remembering. The London investigation firm in our case files has traded for three years, long enough to be in the models' training data, and ChatGPT sent it 43 visitors in September 2026 (the firm's own analytics, 24 July to 7 October 2026). Our two companies, weeks old, exist for the assistants only as something to look up; we think, though cannot prove, that history explains much of the difference.

GPTBot is also the busiest of OpenAI's three on our sites: 52 pages of our address-tracing site in fifteen days against 35 for OAI-SearchBot, and 150 pages of investigationbureau.uk in eight days (server logs to 13 October 2026). The 150 includes the Markdown copies, the plain-text twin of each article published for assistants to read. A robots.txt that blocks GPTBot but allows OAI-SearchBot keeps you eligible for ChatGPT's search results today and out of the model's memory tomorrow. That may be exactly what you want; in the files we read it is rarely a choice.

Cloudflare's switch overrules robots.txt

robots.txt is a request your server publishes. An edge block is a refusal made before the request reaches your server, usually by a content delivery network (CDN), the service that stands between the internet and your server. Cloudflare, in front of many UK business sites, has a one-click setting that blocks AI crawlers (labelled "Block AI bots" or "AI Scrapers and Crawlers"), on by default for some recently added sites; other CDNs and WordPress security plugins have the same switch.

When it is on, GPTBot, OAI-SearchBot, ChatGPT-User and the rest get a 403 (forbidden) or a challenge page from the edge. They never reach your server, your logs never show them, robots.txt is never read, and every Allow line in it is decoration. A business can spend a month writing answer-first pages and be read by nothing but browsers.

The tell is a split in status codes: a browser gets 200 from your home page and ChatGPT-User, GPTBot, ClaudeBot or PerplexityBot get 403, whatever robots.txt says. The free check fetches your home page as each of them and shows what each got; on a plan the Audit traces the refusal to its setting and writes the exact change for your developer. A firewall rule on user agent shows up the same way. Meanwhile the owner reads an empty log as a lack of interest.

What a broken robots.txt looks like

Two faults hide in a robots.txt that looks fine. A crawler that finds its own name reads that group only, so lines written once in the general group for an admin folder never reach it, and private pages end up in a public index. And a second group added later for the same name, to refuse training, merges with the first; the earlier allow generally wins and the training block never takes effect. On a plan the Audit reads your live file against the 32 names ours lists and writes the exact change for your developer.

What robots.txt cannot do

It is a request, not a lock: the large companies say their crawlers read it; a scraper pretending to be a browser does not. It is permission, not an invitation: on investigationbureau.uk every crawler was allowed from day one, yet after eight days ChatGPT's search index held only the home page and Bing four pages (server logs to 13 October 2026); index builders arrive by links and search engines, and a new site has few of either (see how long it takes to get cited). And it says nothing about what they find on arrival: a page blank until JavaScript runs is allowed and unreadable at once, since no AI crawler we have logged executes JavaScript (Vercel and MERJ crawler study, 2025; our own logs, 2026). See do AI assistants read JavaScript websites?