The ClaudeBot Directive: Two Lines That Change Your Score

Here's what Claude actually sees when it reads your robots.txt - and it's looking for its own name.

ClaudeBot checks robots.txt before touching any page. But Claude goes beyond basic access control. When it finds an explicit "User-agent: ClaudeBot / Allow: /" directive, it reads that as a deliberate policy decision - you've actively chosen to participate in Claude's content ecosystem. That's a governance signal.

No ClaudeBot directive? Ambiguity. If your robots.txt mentions GPTBot, PerplexityBot, and Google-Extended but skips ClaudeBot, Claude reads that as either oversight or intentional exclusion. Either way, the trust signal weakens. Claude still follows your general "User-agent: *" rules, but you've lost the bonus from explicit acknowledgment.

Granularity matters too. Full domain access ("/") generates a stronger trust signal than directory-restricted access. Crawl-delay directives? Respected without penalty - they signal thoughtful configuration, not hostility. But "Disallow: /" for ClaudeBot is an active block. That significantly reduces Claude's willingness to cite you, even from cached content.

Claude also checks for consistency. Some sites include directives for "anthropic-ai" but not "ClaudeBot." Claude recognizes both but weights the ClaudeBot-specific entry more heavily - it demonstrates awareness of the exact crawler name. Beyond robots.txt, Claude looks for complementary signals: does your llms.txt reference Anthropic? Does your content policy address AI citation? These signals compound into a holistic trust profile.

GPTBot is everywhere - millions of robots.txt files mention it. For ChatGPT, the absence of a GPTBot directive is the default state. Minimal penalty.

Claude operates in a different universe. ClaudeBot is less widely recognized than GPTBot. Sites that include ClaudeBot directives are self-selecting for above-average governance awareness. Claude factors this in.

Google AI Overviews doesn't use a separate AI crawler the same way. Google-Extended controls Gemini training data, but Googlebot builds the search index regardless. Different strategic game entirely.

The culprit behind the scoring gap: Claude's trust model is more sensitive to robots.txt signals than ChatGPT's. Adding ClaudeBot directives produces a larger marginal improvement on Claude than adding GPTBot directives produces on ChatGPT - because the relative signal value is higher for the less common directive.

Perplexity sits in the middle. PerplexityBot directives are less common than GPTBot but more common than ClaudeBot, so the trust signal carries moderate weight. Sites targeting multi-engine visibility should address all four major crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) to max out trust across every platform.

Tidio.com included explicit ClaudeBot Allow directives alongside GPTBot and PerplexityBot. Comprehensive AI crawler policy. Different crawl-delay values per bot, with ClaudeBot getting the same generous access as GPTBot. Claude read that as a deliberate governance decision. Tidio's Claude bonus: +14.

LiveChat.com addressed ClaudeBot with "Allow: /" and a 2-second crawl-delay. Here's what made it notable: LiveChat is a ChatGPT partner - yet they chose to also welcome Claude's crawler. Combined with strong Organization schema and semantic HTML, that contributed to LiveChat's +12 bonus.

HelpSquad.com? Zero AI-specific crawler directives. No GPTBot, no ClaudeBot, no PerplexityBot. That hurt everywhere, but it hit Claude hardest. HelpSquad scored 42 on Claude vs. 47 on ChatGPT - a -5 penalty. ChatGPT's wider default crawling corpus partially compensated. Claude's governance-first approach didn't forgive the absence.

Crisp.chat mentioned GPTBot but not ClaudeBot. Despite the partial omission, Crisp still pulled a +17 Claude bonus - primarily from other governance signals (schema depth, semantic HTML). But adding a ClaudeBot directive would've pushed the bonus higher. The directive is additive to other signals, not a prerequisite.

Across our full 20-site cohort, sites with explicit ClaudeBot directives scored an average of 8 points higher on Claude - controlling for other variables. Two lines in robots.txt. Eight points. That's one of the most cost-effective optimizations in AEO.

Start here: add ClaudeBot directives to your robots.txt right now. Two minutes, outsized impact. The minimum configuration is three lines: "User-agent: ClaudeBot", "Allow: /", and "Crawl-delay: 2". Put them in a clearly labeled section alongside other AI crawler policies.

Include both "ClaudeBot" and "anthropic-ai" as separate user-agent entries. While Claude primarily identifies as ClaudeBot, some docs reference anthropic-ai as the crawler identifier. Cover both. Use the same Allow/Disallow rules for consistency.

Set crawl-delay between 1 and 5 seconds. Claude respects it and doesn't interpret rate limiting as hostility. Two seconds is the sweet spot - prevents server load without slowing Claude's indexing. Don't go above 10 seconds - very high delays may cause Claude to deprioritize your site in its crawling queue.

Need to restrict certain directories? Use targeted Disallow rules. "Disallow: /admin/" and "Disallow: /checkout/" while keeping "Allow: /" for public content. That demonstrates sophisticated governance - you've thought about what should and shouldn't be accessible to AI crawlers.

Verify your robots.txt after changes. Common mistakes: file in a subdirectory instead of domain root, incorrect line endings causing parse failures, UTF-8 BOM characters corrupting the file. Test by requesting your-domain.com/robots.txt directly. Confirm the ClaudeBot section renders as expected. Add "Sitemap: your-domain.com/sitemap.xml" at the bottom so all crawlers discover your full content inventory.

• robotstxt.org - www.robotstxt.org
• Anthropic ClaudeBot Crawler Policy - support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler