Claude-SearchBot — Anthropic's ai search indexer

As a search indexer, Claude-SearchBot's crawling is oriented toward coverage and freshness of citable pages rather than bulk corpus collection.

Its index feeds Claude's web search feature, where answers cite and link sources — making this one of the AI crawlers whose access can return visibility rather than just consuming content.

Claude-SearchBot identifies itself with the following user-agent string:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com)

Never trust the user-agent string alone. Scrapers routinely impersonate well-known crawlers to inherit their access. Anthropic documents Claude-SearchBot in its help center. Treat user-agent claims as unverified until checked against DNS and observed behavior — search-indexer identities are commonly spoofed.

Note that most AI crawlers and fetchers, Claude-SearchBot included, do not execute JavaScript — so this traffic is invisible to GA4 and every script-based analytics tool. Server logs, CDN analytics, or a dedicated bot-analytics layer are the only places you will see it. For the full picture of measuring AI-driven visits, see our guide on how to track AI traffic.

To refuse Claude-SearchBot access to your entire site, add this to your robots.txt:

User-agent: Claude-SearchBot
Disallow: /

To restrict it from specific sections only (for example, premium content) while leaving the rest open:

User-agent: Claude-SearchBot
Disallow: /premium/
Disallow: /members/

The case for blocking: Blocking it removes you from Claude's search citations. That is only rational under a strict no-exposure paywall strategy or as deliberate negotiating leverage.

The case for allowing or monetizing: Like other AI search indexers, the play is citation share, not access fees: structured pages with direct answers get cited more. Sponsored placement inside AI answers is the emerging complement.