AI Crawlers Explained: GPTBot, ClaudeBot & PerplexityBot

Before an AI engine can cite you, its crawler has to reach your page—and be allowed in. If GPTBot, ClaudeBot, or PerplexityBot is blocked in your robots.txt, or your content only loads with JavaScript, you’re invisible to AI search no matter how good your copy is. This guide explains how the major AI crawlers behave, how to control them with robots.txt and llms.txt, and how the SellOnLLM extension checks all of this for you automatically.

Three types of AI bot (and why the distinction matters)

Not every AI bot does the same job, and you may want to treat them differently:

• Training crawlers fetch content to improve future models (e.g., OpenAI’s GPTBot).
• Search/answer crawlers fetch content to surface it in live answers and citations (e.g., OpenAI’s OAI-SearchBot, PerplexityBot).
• User-initiated fetchers retrieve a page because a user asked about it in real time (e.g., ChatGPT-User).

The key implication: blocking a training crawler is a data/copyright decision, but blocking a search/answer crawler directly removes you from AI citations. Many sites accidentally do the second while intending the first.

The major AI crawlers in 2026

Because user-agent names and policies change, always confirm against the source: OpenAI documents its bots in the OpenAI crawlers reference, Anthropic explains ClaudeBot in its crawler help article, and Perplexity lists its crawlers in its bots guide. Google’s AI features and the Google-Extended control are covered in its crawler overview.

Controlling AI crawlers with robots.txt

The robots.txt file at your site root is the primary access control. To allow AI answer engines to cite you (recommended for most brands), make sure you are not disallowing them:

To opt out of a specific crawler (for example, to exclude training while keeping search), disallow just that user-agent:

Common mistakes we catch in audits: a blanket Disallow: / under User-agent: * that silently blocks every AI bot, staging rules copied to production, and disallowing the answer crawler while trying to block only training. Each one quietly removes you from AI citations.

Why robots.txt isn’t enough: JavaScript rendering

Even with crawlers allowed, most AI bots read your initial HTML and don’t execute JavaScript the way a browser does. If your main content, prices, or FAQs only appear after client-side JS runs, the crawler sees an empty shell. Server-side rendering or static generation for your key content is the single biggest technical fix for many sites—it’s a core check in the AI Readiness Score.

llms.txt: guiding crawlers to your best content

Where robots.txt controls access, llms.txt guides attention. It’s a curated Markdown file at your root that points AI systems to your most important pages, following the llms.txt spec: a single H1, a blockquote summary, and H2 sections of Markdown links. It does not grant or deny access—it’s a recommendation layer that works alongside robots.txt. If you don’t have one, generate a spec-compliant file with our free llms.txt generator.

Keep the two straight:

• robots.txt — “you may/may not crawl these paths.”
• llms.txt — “here are the best pages to understand and cite.”
• sitemap.xml — “here is the full inventory of URLs.”

How to verify what crawlers actually see

Don’t assume—check. Three practical methods:

1. Read your robots.txt at /robots.txt and look for any rule affecting the AI user-agents above.
2. View source (not the rendered DOM) to confirm your main content exists in the raw HTML.
3. Run the extension. The SellOnLLM extension checks robots.txt directives, server-rendered content, structured data, and llms.txt in one pass and flags exactly what’s blocking AI access.

For server logs and site-wide crawl checks, pair it with the free multi-page LLM audit.

A sensible default policy for most brands

Unless you have a specific reason to opt out of training, the pragmatic 2026 stance for brands that want AI traffic is:

• Allow answer/search crawlers (OAI-SearchBot, PerplexityBot, ClaudeBot search) so you can be cited.
• Decide deliberately on pure training crawlers (GPTBot, Google-Extended) based on your content strategy.
• Publish llms.txt to steer models toward your best pages.
• Server-render your key content so allowed crawlers actually get it.

Then verify the whole setup with an audit—because a single stray robots.txt line can undo all of it. Learn the strategy behind this in the AEO Hub, and see how it feeds whether ChatGPT recommends you.

The extension checks this automatically

You shouldn’t have to remember every user-agent and edge case. Install the free SellOnLLM extension, open any page, and it verifies crawler access, rendering, structured data, and llms.txt—then tells you precisely what to fix so GPTBot, ClaudeBot, and PerplexityBot can read and cite you.