# AI Crawlers Explained: GPTBot, PerplexityBot, ClaudeBot > What each named AI crawler is, who runs it, the difference between training and search crawls, and how to spot them in your server logs. URL: https://aiseocourse.net/guide/which-ai-crawlers-read-your-site/ Last-Modified: 2026-09-20 Author: Adam Yong would turn main into a scroll container and break every \`position: sticky\` inside it. --> Skip to main content [#main-content →](#main-content) definition Free guide # Which AI Crawlers Read Your Site What each named AI crawler is, who runs it, the difference between training and search crawls, and how to spot them in your server logs. update Updated September 20, 2026 schedule 5 min read school Part of How AI Search Picks Local Businesses [/lessons/how-ai-search-picks-local-businesses/ →](/lessons/how-ai-search-picks-local-businesses/) ![Table-style illustration of four named AI crawler bots with their operators](/images/featured/clean-table-style-illustration-of-four-named-ai-cr.webp) ## The bots that decide whether you exist Lesson 1: how AI search picks local businesses [/lessons/how-ai-search-picks-local-businesses/ →](/lessons/how-ai-search-picks-local-businesses/) ends with a simple dependency. Retrieval needs a copy of your page, and that copy arrives through a crawler. If the crawler never comes, or comes and gets a 403, none of your other work reaches an answer — the retrieval step described in how RAG decides which local business to name [/guide/how-rag-decides-which-local-business-to-name/ →](/guide/how-rag-decides-which-local-business-to-name/) has nothing to pull. These are the named agents worth knowing, what they do, and how to recognise them. ## Training crawls versus search crawls The distinction matters more than the list itself. A **training crawl** collects content used to train or update a model. It affects what a model knows generally. It does not fetch your page at the moment a customer asks a question. A **search crawl** fetches pages to support live answers, either by maintaining a search index or by retrieving a URL on demand during a conversation. These are the agents that decide whether your current hours can appear in an answer today. Blocking the first slows the model’s general familiarity with you. Blocking the second removes you from live answers. People who block “AI bots” in one sweep usually mean to do the first and accidentally do both. ## The named agents | User agent | Operator | Type | What it does | | --- | --- | --- | --- | | GPTBot | OpenAI | Training | Collects content for model training | | OAI-SearchBot | OpenAI | Search | Fetches pages to support search features | | ChatGPT-User | OpenAI | On-demand | Fetches a URL when a user’s prompt requires it | | PerplexityBot | Perplexity | Search | Builds and refreshes Perplexity’s index | | Perplexity-User | Perplexity | On-demand | Fetches a page during a live answer | | ClaudeBot | Anthropic | Training | Collects content for model training | | Claude-SearchBot | Anthropic | Search | Supports search-connected answers | | Claude-User | Anthropic | On-demand | Fetches a page at a user’s request | | Google-Extended | Google | Training | Controls Gemini training use, separate from Googlebot | | Applebot-Extended | Apple | Training | Controls Apple’s AI training use | | CCBot | Common Crawl | Crawl corpus | Feeds an open dataset many models use | Note that Google-Extended is not a crawler in the usual sense. It is a control token: Googlebot still crawls your site for Search, and Google-Extended governs whether that content may be used for generative training. ## Spotting them in your logs Open your server access log and filter for the user-agent strings above. You are checking three things: whether they arrive, what response code they receive, and which pages they reach. ![Server log excerpt in monospace type highlighting AI user-agent strings](/images/content/server-log-excerpt-panel-in-monospace-type-highlig.webp) A healthy pattern looks like periodic visits returning 200 responses, reaching your service pages rather than only the homepage. Warning signs include no AI user agents at all, a wall of 403s, or crawlers only ever hitting the homepage, which usually points at internal linking or sitemap problems. If your host does not expose raw logs, most CDNs and bot-protection layers have a traffic view that can be filtered by user agent. ## Verifying a crawler is real User-agent strings are trivial to fake, and scrapers do fake them. Each major operator publishes the IP ranges its crawlers use, so the reliable check is to confirm the requesting IP belongs to the operator’s published range, or to run a reverse DNS lookup and then a forward lookup on the result. This matters if you are considering rate limits or blocks. Blocking a spoofed agent is fine. Blocking a real search-side fetcher because it was noisy is a decision you should make deliberately. ## What to do about it For nearly every local business, allow them all. You are trying to be cited, and the fetchers that build those citations need access. Write the rules explicitly rather than relying on a default, and keep the file honest: the robots.txt guide for AI crawlers [/guide/robots-txt-for-ai-crawlers/ →](/guide/robots-txt-for-ai-crawlers/) has copy-paste blocks for both allow and block policies, plus the syntax errors that silently make a file do nothing. Then confirm it worked in the logs, which is the one part of this subject you can prove. Lesson 5 [/lessons/build-a-local-site-ai-can-find/ →](/lessons/build-a-local-site-ai-can-find/) covers the rest of the technical floor these crawlers depend on. ## What blocking each one would actually cost Before you decide a policy, it helps to price the decision honestly. **Blocking GPTBot or ClaudeBot** removes your content from future training. The effect is diffuse and slow: models become less familiar with your business in general. For a local service business with no proprietary archive, the protection is close to worthless and the cost is real. **Blocking OAI-SearchBot, PerplexityBot or Claude-SearchBot** removes you from live, cited answers. This is the expensive one. Ask an assistant about your business after blocking these and it will answer from directory listings, review sites and whatever it held before, none of which you control. **Blocking ChatGPT-User, Perplexity-User or Claude-User** prevents a page being fetched when a customer explicitly asks about it. That is someone typing your name into an assistant and being told the page cannot be read. **Blocking Google-Extended** limits generative training use while leaving classic Search untouched. It is the narrowest option and the most defensible if you want one, because it costs the least. The businesses with a genuine case for blocking are publishers whose written archive is the product being sold. A plumber, a clinic or a cafe is in the opposite position: obscurity costs more than copying. Common questions ## Questions readers ask Should I block AI crawlers? expand\_more For most local businesses, no. Blocking the search-side crawlers removes you from the answers you are trying to appear in, and blocking the training crawlers removes your content from a channel where being known helps. Publishers with paid archives have a different calculation. Is GPTBot the same as the bot behind ChatGPT search results? expand\_more No. GPTBot collects content for model training. OAI-SearchBot fetches pages to support search features, and ChatGPT-User fetches a page when a user's prompt requires it. They are separate user agents with separate purposes. How do I verify a crawler is genuine? expand\_more Check the published IP ranges for each operator rather than trusting the user-agent string, which anyone can spoof. Each major operator publishes its ranges, and a reverse DNS check on the requesting IP is the usual verification. Why do I see no AI crawler traffic at all? expand\_more Check robots.txt first, then any bot-protection or firewall rule in front of your site. Aggressive bot filtering blocking legitimate AI fetchers is common and invisible until you look at the logs. ## Guides in this cluster Short, evergreen answers that go deeper than the lesson itself. comparison ### AEO vs GEO vs Local SEO: Which One Are You Actually Doing? One-line definitions of each, where they overlap, what a local business should actually care about, and why most vendor distinctions are marketing. [AEO vs GEO vs Local SEO: Which One Are You Actually Doing? →](/guide/aeo-vs-geo-vs-local-seo/) comparison ### ChatGPT Search vs Google AI Overviews vs Perplexity: Where Each One Gets Local Business Facts Which sources each engine leans on for local queries, what a citation looks like in each, and where to spend your effort first. [ChatGPT Search vs Google AI Overviews vs Perplexity: Where Each One Gets Local Business Facts →](/guide/chatgpt-vs-ai-overviews-vs-perplexity-local-facts/) process ### How Retrieval-Augmented Generation Decides Which Local Business to Name Query, retrieval, grounding, answer: walk the four steps an AI assistant takes, and see exactly where your business can intervene. [How Retrieval-Augmented Generation Decides Which Local Business to Name →](/guide/how-rag-decides-which-local-business-to-name/) definition ### What Is Generative Engine Optimization (GEO)? A plain definition of GEO, how it differs from classic SEO, and what a local business actually changes, without the vendor jargon. [What Is Generative Engine Optimization (GEO)? →](/guide/what-is-generative-engine-optimization/)