The bots that decide whether you exist
Lesson 1: how AI search picks local businesses ends with a simple dependency. Retrieval needs a copy of your page, and that copy arrives through a crawler. If the crawler never comes, or comes and gets a 403, none of your other work reaches an answer — the retrieval step described in how RAG decides which local business to name has nothing to pull.
These are the named agents worth knowing, what they do, and how to recognise them.
Training crawls versus search crawls
The distinction matters more than the list itself.
A training crawl collects content used to train or update a model. It affects what a model knows generally. It does not fetch your page at the moment a customer asks a question.
A search crawl fetches pages to support live answers, either by maintaining a search index or by retrieving a URL on demand during a conversation. These are the agents that decide whether your current hours can appear in an answer today.
Blocking the first slows the model’s general familiarity with you. Blocking the second removes you from live answers. People who block “AI bots” in one sweep usually mean to do the first and accidentally do both.
The named agents
| User agent | Operator | Type | What it does |
|---|---|---|---|
| GPTBot | OpenAI | Training | Collects content for model training |
| OAI-SearchBot | OpenAI | Search | Fetches pages to support search features |
| ChatGPT-User | OpenAI | On-demand | Fetches a URL when a user’s prompt requires it |
| PerplexityBot | Perplexity | Search | Builds and refreshes Perplexity’s index |
| Perplexity-User | Perplexity | On-demand | Fetches a page during a live answer |
| ClaudeBot | Anthropic | Training | Collects content for model training |
| Claude-SearchBot | Anthropic | Search | Supports search-connected answers |
| Claude-User | Anthropic | On-demand | Fetches a page at a user’s request |
| Google-Extended | Training | Controls Gemini training use, separate from Googlebot | |
| Applebot-Extended | Apple | Training | Controls Apple’s AI training use |
| CCBot | Common Crawl | Crawl corpus | Feeds an open dataset many models use |
Note that Google-Extended is not a crawler in the usual sense. It is a control token: Googlebot still crawls your site for Search, and Google-Extended governs whether that content may be used for generative training.
Spotting them in your logs
Open your server access log and filter for the user-agent strings above. You are checking three things: whether they arrive, what response code they receive, and which pages they reach.
A healthy pattern looks like periodic visits returning 200 responses, reaching your service pages rather than only the homepage. Warning signs include no AI user agents at all, a wall of 403s, or crawlers only ever hitting the homepage, which usually points at internal linking or sitemap problems.
If your host does not expose raw logs, most CDNs and bot-protection layers have a traffic view that can be filtered by user agent.
Verifying a crawler is real
User-agent strings are trivial to fake, and scrapers do fake them. Each major operator publishes the IP ranges its crawlers use, so the reliable check is to confirm the requesting IP belongs to the operator’s published range, or to run a reverse DNS lookup and then a forward lookup on the result.
This matters if you are considering rate limits or blocks. Blocking a spoofed agent is fine. Blocking a real search-side fetcher because it was noisy is a decision you should make deliberately.
What to do about it
For nearly every local business, allow them all. You are trying to be cited, and the fetchers that build those citations need access. Write the rules explicitly rather than relying on a default, and keep the file honest: the robots.txt guide for AI crawlers has copy-paste blocks for both allow and block policies, plus the syntax errors that silently make a file do nothing.
Then confirm it worked in the logs, which is the one part of this subject you can prove. Lesson 5 covers the rest of the technical floor these crawlers depend on.
What blocking each one would actually cost
Before you decide a policy, it helps to price the decision honestly.
Blocking GPTBot or ClaudeBot removes your content from future training. The effect is diffuse and slow: models become less familiar with your business in general. For a local service business with no proprietary archive, the protection is close to worthless and the cost is real.
Blocking OAI-SearchBot, PerplexityBot or Claude-SearchBot removes you from live, cited answers. This is the expensive one. Ask an assistant about your business after blocking these and it will answer from directory listings, review sites and whatever it held before, none of which you control.
Blocking ChatGPT-User, Perplexity-User or Claude-User prevents a page being fetched when a customer explicitly asks about it. That is someone typing your name into an assistant and being told the page cannot be read.
Blocking Google-Extended limits generative training use while leaving classic Search untouched. It is the narrowest option and the most defensible if you want one, because it costs the least.
The businesses with a genuine case for blocking are publishers whose written archive is the product being sold. A plumber, a clinic or a cafe is in the opposite position: obscurity costs more than copying.