# Build a Local Site AI Can Find | Lesson 5

> Crawlability, rendering, speed, sitemaps and crawler permissions: the technical floor that lets AI crawlers reach and read your local business site at all.

URL: https://aiseocourse.net/lessons/build-a-local-site-ai-can-find/

would turn main into a scroll container and break every \`position: sticky\` inside it. --> 

Skip to main content

[#main-content →](#main-content)

Lesson 5 of 8

# Lesson 5: Build a Local Site AI Crawlers Can Actually Reach

Last updated: September 20, 2026

The technical baseline that lets AI crawlers reach and read your site.

schedule 12 min read lock\_open Free, no signup person Written by Adam Yong

Next: Lesson 6 arrow\_forward

[/lessons/reviews-mentions-and-citations/ →](/lessons/reviews-mentions-and-citations/)

 

All 8 lessons

[/lessons/ →](/lessons/)

![Crawler request reaching a server and returning rendered HTML, with sitemap and robots file markers](/images/misc/flat-editorial-illustration-of-a-crawler-request-r.webp)

★★★★★ 4.9 (200+ Verified Business Audits)

verified 100% Free, no signup, no paywall

group 9,400+ readers

schedule 8 lessons · ~80 minutes

What it fixes

[#what-it-fixes →](#what-it-fixes)

 

The lesson

[#lesson →](#lesson)

 

Key takeaways

[#takeaways →](#takeaways)

 

Figures

[#figures →](#figures)

 

Steps

[#steps →](#steps)

 

Guides

[#guides →](#guides)

 

FAQ

[#faq →](#faq)

What it fixes

## Technical problems that make AI crawlers give up

gpp\_maybe

### Something is blocking AI crawlers and nobody noticed

A default plugin setting or an inherited robots file is enough to shut GPTBot, ClaudeBot and PerplexityBot out of a site whose owner is actively trying to win AI answers.

code\_off

### Your content only appears after JavaScript runs

A page that looks fine in your browser can arrive empty to a fetcher. Some crawlers render, some do not, and the ones that do not simply move on.

speed

### The page is too slow to finish fetching

Crawlers give up. A slow time to first byte on mobile data means your best page never gets read, regardless of what is written on it.

## What this lesson covers

check

Crawlability and rendering basics

check

Page speed and mobile as retrieval prerequisites

check

Sitemap and internal link structure

check

Crawler permissions for the named AI bots

check

Disclosed note: LocusPilot is used as an example in this lesson

## None of the previous lessons matter if the page cannot be fetched

A clean profile, a quotable service page and a valid JSON-LD block are all worth nothing if the crawler times out, receives an empty shell, or was told not to come. This lesson is the floor underneath the rest of the course.

**Disclosure, stated here because this lesson is where it applies.** LocusPilot, which publishes this course, builds websites for local businesses and is one of the platforms discussed below. Treat the platform comparison accordingly, and read the 

website builder versus WordPress guide

[/guide/website-builder-vs-wordpress-for-ai-findable-site/ →](/guide/website-builder-vs-wordpress-for-ai-findable-site/)

, which carries the same disclosure and gives a decision rule rather than a recommendation.

## Crawlability and rendering

Start with what the crawler actually receives. Open your most important service page, view the rendered source, and search for a sentence from the middle of your service description. If it is not in the HTML, your content depends on JavaScript executing, and not every fetcher executes JavaScript reliably.

Googlebot renders JavaScript, eventually. Several AI fetchers are less patient. Server-side rendering or static generation removes the question entirely, which is why it is the default recommendation for a local business site.

Then check the plumbing:

-   Every important page returns a 200, not a redirect chain or a soft 404.
-   One canonical version of the site, so `https://www.example.com` and `https://example.com` do not both serve content.
-   No `noindex` left behind from a staging environment, which is a surprisingly common cause of total invisibility.
-   Reasonable crawl budget use, meaning you are not generating thousands of near-duplicate URLs through filters or session parameters.

## Speed and mobile

Speed matters here as a prerequisite rather than a score. A crawler that waits three seconds for a first byte may give up, and a customer arriving from an AI answer on mobile data will certainly give up. Time to first byte under roughly half a second, images sized for the web, and scripts that do not block rendering will get you past the bar.

Core Web Vitals are a reasonable proxy, but do not turn them into a project. The realistic priority order for a small local site is: fix the slow server response, compress the oversized hero image, remove the tracking scripts nobody reads, and stop there.

## Sitemap and internal links

An XML sitemap is how you tell search infrastructure which pages exist. Keep it current, include every page you want cited, exclude anything that returns a redirect or an error, and submit it in Google Search Console.

Internal links matter more than owners expect. A page reachable only from a dropdown five levels deep gets crawled rarely and cited less. Two rules cover most sites: every service page is reachable within two clicks of the homepage, and related pages link to each other with descriptive anchor text rather than “click here”. That structure also helps an engine understand which of your pages is the authority on which topic, which is the same clustering logic this course uses between lessons and guides.

## Crawler permissions, on purpose

AI crawlers announce themselves with user agents. The named ones that matter for a local business:

-   **GPTBot**, OpenAI’s training crawler.
-   **OAI-SearchBot**, OpenAI’s search-side fetcher.
-   **ChatGPT-User**, fetches a page when a user’s prompt requires it.
-   **PerplexityBot** and **Perplexity-User**, Perplexity’s index and live fetchers.
-   **ClaudeBot** and **Claude-SearchBot**, Anthropic’s equivalents.
-   **Google-Extended**, which governs Gemini and related training use, separately from Googlebot.

| Crawler type | What blocking it costs you | Typical local business policy |
| --- | --- | --- |
| Search (OAI-SearchBot, PerplexityBot, Claude-SearchBot) | Removal from live AI answers | Allow |
| On-demand (ChatGPT-User, Perplexity-User, Claude-User) | Your page cannot be fetched mid-conversation | Allow |
| Training (GPTBot, ClaudeBot, Google-Extended) | Less general model familiarity with your brand | Allow |

The 

guide on which AI crawlers read your site

[/guide/which-ai-crawlers-read-your-site/ →](/guide/which-ai-crawlers-read-your-site/)

 explains who operates each one and how to tell training crawls from search crawls in your logs.

For most local businesses the sensible policy is to allow all of them. You are trying to appear in AI answers, and blocking the fetcher that builds those answers is self-defeating. A minimal permissive file looks like this:

```
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap-index.xml
```

The 

robots.txt guide for AI crawlers

[/guide/robots-txt-for-ai-crawlers/ →](/guide/robots-txt-for-ai-crawlers/)

 has the full copy-paste set, including block variants if you have a genuine reason, plus the syntax errors that quietly make a file do nothing.

While you are in this territory, you will hear about llms.txt, a proposed markdown file describing your site for language models. It is not an adopted standard and support is thin. It is also harmless. The 

llms.txt guide

[/guide/llms-txt-for-local-businesses/ →](/guide/llms-txt-for-local-businesses/)

 gives the honest verdict on whether that hour is better spent elsewhere.

## Confirm it worked

This is the one part of AI SEO you can verify directly. Open your server access logs and filter for the user agents above. You are looking for three things: that they arrive at all, that they receive 200 responses rather than errors or blocks, and that they are reaching your important pages rather than only the homepage.

If a named crawler never appears, check robots.txt, then any firewall or bot-protection rule in front of your site. Aggressive bot protection blocking legitimate AI fetchers is a common and invisible failure.

## What to do next

Work through the checks in order: rendering, speed, sitemap, internal links, robots, logs. Most sites need one or two fixes rather than a rebuild.

Then move to 

Lesson 6

[/lessons/reviews-mentions-and-citations/ →](/lessons/reviews-mentions-and-citations/)

, which turns to the signals you do not control directly: what other sites and other people say about your business.

![Monospace robots.txt panel with AI crawler user-agent directives highlighted](/images/content/monospace-robots-txt-panel-with-ai-crawler-user-ag.webp)

Monospace robots.txt panel with AI crawler user-agent directives highlighted

Ask AI about this lesson:

travel\_explorePerplexity

[https://www.perplexity.ai/search?q=Read%20https%3A%2F%2Faiseocourse.net%2Flessons%2Fbuild-a-local-site-ai-can-find%2F%20and%20summarise%20what%20it%20says%20about%20Build%20a%20Local%20Site%20AI%20Can%20Find%20for%20a%20local%20business%20owner%2C%20then%20list%20the%20three%20actions%20it%20recommends%20first. →](https://www.perplexity.ai/search?q=Read%20https%3A%2F%2Faiseocourse.net%2Flessons%2Fbuild-a-local-site-ai-can-find%2F%20and%20summarise%20what%20it%20says%20about%20Build%20a%20Local%20Site%20AI%20Can%20Find%20for%20a%20local%20business%20owner%2C%20then%20list%20the%20three%20actions%20it%20recommends%20first.)

chatChatGPT

[https://chatgpt.com/?q=Read%20https%3A%2F%2Faiseocourse.net%2Flessons%2Fbuild-a-local-site-ai-can-find%2F%20and%20summarise%20what%20it%20says%20about%20Build%20a%20Local%20Site%20AI%20Can%20Find%20for%20a%20local%20business%20owner%2C%20then%20list%20the%20three%20actions%20it%20recommends%20first. →](https://chatgpt.com/?q=Read%20https%3A%2F%2Faiseocourse.net%2Flessons%2Fbuild-a-local-site-ai-can-find%2F%20and%20summarise%20what%20it%20says%20about%20Build%20a%20Local%20Site%20AI%20Can%20Find%20for%20a%20local%20business%20owner%2C%20then%20list%20the%20three%20actions%20it%20recommends%20first.)

auto\_awesomeGemini

[https://gemini.google.com/app?q=Read%20https%3A%2F%2Faiseocourse.net%2Flessons%2Fbuild-a-local-site-ai-can-find%2F%20and%20summarise%20what%20it%20says%20about%20Build%20a%20Local%20Site%20AI%20Can%20Find%20for%20a%20local%20business%20owner%2C%20then%20list%20the%20three%20actions%20it%20recommends%20first. →](https://gemini.google.com/app?q=Read%20https%3A%2F%2Faiseocourse.net%2Flessons%2Fbuild-a-local-site-ai-can-find%2F%20and%20summarise%20what%20it%20says%20about%20Build%20a%20Local%20Site%20AI%20Can%20Find%20for%20a%20local%20business%20owner%2C%20then%20list%20the%20three%20actions%20it%20recommends%20first.)

Visual reference

## Figures from this lesson

Diagrams you can screenshot and keep beside you while you work through the steps.

 

Why it matters

## Why the technical floor decides whether any of this lands

foundation

### It is the floor everything else stands on

A perfect profile and a beautifully written service page are invisible if the page returns a timeout or renders empty to a bot.

build

### Most of it is one-off work

Sitemap, rendering, robots rules and internal links are set once and checked occasionally, rather than maintained weekly.

fact\_check

### It is testable

Unlike most of AI SEO, you can prove this part. Server logs show which crawlers arrived and what they got.

gpp\_maybe

### It prevents accidental self-harm

Plenty of small sites block AI crawlers without knowing it, through a default plugin setting or an inherited robots file.

Steps

## How to make your local site reachable by AI crawlers

1.  1 visibility
    
    ### Check what the crawler sees
    
    View the rendered source of your key pages. If the service description is not in the HTML, a crawler may never read it.
    
2.  2 speed
    
    ### Fix the obvious speed problems
    
    Oversized images, blocking scripts and a slow server response are the usual causes. You are aiming for adequate, not perfect.
    
3.  3 map
    
    ### Publish a current sitemap
    
    Every page you want cited should be listed, and every listed URL should return 200. Submit it in Search Console.
    
4.  4 linked\_services
    
    ### Link internally with intent
    
    Each service page should be reachable within two clicks of the homepage and linked from related pages with descriptive anchors.
    
5.  5 policy
    
    ### Set crawler permissions on purpose
    
    Write robots.txt rules for the named AI bots, publish it at the site root, and confirm it serves as plain text.
    

## Site reachable? Now work on what others say about you.

Lesson 6 covers reviews, structured citations and brand mentions, which is the part of your footprint you do not write yourself.

play\_arrow Continue to Lesson 6

[/lessons/reviews-mentions-and-citations/ →](/lessons/reviews-mentions-and-citations/)

 

Open the 90-day checklist

[/checklist/ →](/checklist/)

## Guides that go deeper on AI crawlers, robots.txt and rendering

Short, evergreen answers that go deeper than the lesson itself.

definition

### llms.txt for Local Businesses: What It Is and Whether It Helps

What the proposed standard is, which engines consume it today, a minimal example, and an honest verdict on effort versus payoff.

[llms.txt for Local Businesses: What It Is and Whether It Helps →](/guide/llms-txt-for-local-businesses/)

process

### robots.txt for AI Crawlers: Allowing GPTBot, ClaudeBot and PerplexityBot

Per-bot user-agent strings, allow and disallow blocks to copy, what blocking actually costs you, and how to verify the file is live.

[robots.txt for AI Crawlers: Allowing GPTBot, ClaudeBot and PerplexityBot →](/guide/robots-txt-for-ai-crawlers/)

comparison

### Website Builder vs WordPress for an AI-Findable Local Site

What each platform makes easy or hard for schema, speed and control, migration cost, and a decision rule by situation.

[Website Builder vs WordPress for an AI-Findable Local Site →](/guide/website-builder-vs-wordpress-for-ai-findable-site/)

Reader feedback

## What readers said about the technical lesson

Feedback from owners and SEOs working through the lessons. These are reader comments, not Google Business Profile reviews.

★★★★★

Rated 5 out of 5

> "We found a client blocking PerplexityBot through an inherited robots file nobody had read in two years. Ten minute fix, and it explained a lot."

Owen B.

Small agency principal

★★★★★

Rated 5 out of 5

> "The log check was the part I had never done. Seeing which bots actually arrived made the whole thing feel less like guesswork."

Priya S.

Manager, mobile detailing business

Common questions

## AI crawlers and site technicals: common questions

Should a local business block GPTBot? expand\_more

Generally no. Blocking the training crawler removes your content from a channel you are trying to win, and blocking the search-side crawlers removes you from live answers entirely. Publishers with paid archives have a different calculation. A local service business usually does not.

Does my site need to be server-rendered? expand\_more

It needs to return your content in the HTML a crawler receives. Server rendering or static generation both do that reliably. A single-page app that paints content only after JavaScript runs may work for Google and fail for other fetchers.

How fast does my site need to be? expand\_more

Fast enough not to be abandoned mid-fetch. A time to first byte under about half a second and a page that loads on mobile data without stalling is the practical bar. Chasing a perfect score is not necessary.

Where does robots.txt go? expand\_more

At the site root, reachable at yourdomain.com/robots.txt, served as plain text. A file in a subfolder does nothing, and a syntax error can silently allow or block everything.

Do I need llms.txt? expand\_more

It is a community proposal, not an adopted standard, and support is limited today. It is harmless to add and unlikely to change anything on its own. The guide on llms.txt gives the honest verdict.

Can I confirm AI crawlers actually visited? expand\_more

Yes, in your server access logs. Filter for the named user agents and check the response codes they received. Verify against each operator's published IP ranges, since user-agent strings can be spoofed.

## Related lessons

Lesson 4

### Schema and Structured Data for Local AI SEO

Machine-readable facts so AI engines can resolve your business as an entity.

[Schema and Structured Data for Local AI SEO →](/lessons/schema-and-structured-data-for-local-ai-seo/)

Lesson 7

### Measure Your AI Search Visibility

A way to tell whether any of the previous work is landing.

[Measure Your AI Search Visibility →](/lessons/measure-your-ai-search-visibility/)

![Portrait of Adam Yong](/images/squares/editorial-headshot-portrait-of-course-author-adam-.webp)

Adam Yong

Author & Founder, LocusPilot

Founder of LocusPilot and Agility Writer; leads GEO strategy at ADE Marketing.

verified Founder, LocusPilot (AI website builder for local businesses)

More about the author

[/author/adam-yong/ →](/author/adam-yong/)

Open access · Immediate start

## Ready for Lesson 6?

Reviews, Mentions and Citations. It picks up exactly where this lesson stops.

Read Lesson 6 arrow\_forward

[/lessons/reviews-mentions-and-citations/ →](/lessons/reviews-mentions-and-citations/)

 

View the 90-Day Checklist

[/checklist/ →](/checklist/)

No signup. No paywall. 8 lessons, ~80 minutes.