Should you allow GPTBot, ClaudeBot and PerplexityBot?

Who these crawlers are

Robots.txt decisions get made badly because the agent names are opaque and the vendor documentation is uneven. Some of these are bulk crawlers, some are search indexers, some are live fetches triggered by a user in the moment, and two of them are not crawlers at all but permission tokens. Blocking the wrong one has effects people do not intend.

The table below summarises the main agents as vendors describe them today. Where a vendor is vague, we say so rather than filling the gap. Names and behaviour change, so check the vendor documentation before making a decision you care about.

Two distinctions in that table do most of the work. The first is between training crawlers and search or retrieval agents. If your objection is to model training, the targeted controls are GPTBot, Google-Extended, Applebot-Extended and CCBot, and you can still allow the search-side agents. The second is between scheduled crawling and live user-triggered fetches. Blocking ChatGPT-User or Claude-User denies a page to someone who has already asked for it by name, which is rarely what anyone means to do.

The case for allowing them

The argument is mostly mechanical rather than ideological.

• Citation requires access. An assistant cannot quote a page it was not allowed to fetch. Every guide, price list and service page you wrote to persuade people is unavailable to the system now doing the recommending.
• Your description of yourself is the accurate one. Block your own site and the assistant assembles you from directories, review sites and old press. Those sources are typically stale and sometimes wrong, and you have no editorial control over them.
• Google-Extended does not affect Search. Google documents it as a control over generative product use, not over indexing. Blocking it costs you AI surfaces and buys you nothing in ordinary results.
• The blocking rarely achieves the stated goal. If the intent is to keep your text out of models entirely, a robots.txt rule does not reach content already crawled, syndicated, quoted or archived elsewhere.
• Retrieval is now a distribution channel. If a share of your prospective customers ask an assistant before they open a search engine, being unreadable in that channel is a commercial decision, not a technical one.

The honest case against, and who it applies to

There are real reasons to block, and pretending otherwise would be dishonest. They apply to a minority of sites.

• The content is the product. Paywalled journalism, paid research, course material and subscription databases sell access. Free summarisation competes with the thing being sold, and several publishers have chosen blocking or paid licensing deliberately.
• You do not own the rights. Licensed photography, syndicated text and third-party datasets may carry contractual limits on reuse. Here the block is a compliance requirement rather than a marketing choice.
• Legal or privacy sensitivity. Archives containing personal data, court material or anything under a takedown regime deserve a conservative default.
• Infrastructure cost. Some crawlers request heavily. If a specific agent is measurably hurting a small server, rate limiting it is legitimate. That is a capacity decision about one agent, not a reason to block the category.

Notice what is missing from that list. "AI might take our traffic" is a fear, not a mechanism, and blocking does not preserve the traffic. It just removes you from the answer while your competitors stay in it. If the concern is commercial, the useful response is to measure appearance and referral behaviour first, using the method in the fan-out queries guide.

One middle path is worth naming: allow the search and retrieval agents, restrict the training-oriented ones. It is a defensible position for publishers, and it keeps you citable while limiting bulk ingestion. It is more brittle than it sounds, because the line between the two moves as vendors reorganise their agents.

Here is the robots.txt

This is our own allow list, reproduced in full. Named agents come first so that the intent is unambiguous to both a parser and a human reading the file. Adapt the host and sitemap lines to your domain.

# Boost Search: a GEO consultancy. Everything here is crawlable, deliberately,
# including by AI assistants. That is the point.
# Machine-readable overview: https://boostsearch.ai/llms.txt
# Full corpus in one file: https://boostsearch.ai/llms-full.txt

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: Bytespider
User-agent: CCBot
User-agent: cohere-ai
User-agent: Meta-ExternalAgent
User-agent: DuckDuckBot
User-agent: YandexBot
Allow: /

User-agent: *
Allow: /
Disallow: /api/

Host: https://boostsearch.ai
Sitemap: https://boostsearch.ai/sitemap.xml

The live version is at boostsearch.ai/robots.txt, served from a single generated file so it cannot drift from what we recommend. A few details in it are deliberate.

• Stacked user agent lines. Consecutive User-agent lines followed by one Allow form a single group. That is valid, compact, and easier to audit than repeating a block twenty times.
• Named agents come first. Most parsers apply the most specific matching group. Naming agents explicitly means a later change to the wildcard group cannot quietly cut off an assistant.
• No crawl-delay for AI agents. Support for it is inconsistent, and slowing a crawler is not a goal here.
• Only low-value paths disallowed. The single exclusion is our API path, which has no content value. Nothing a reader or an assistant would want is blocked.
• Machine surfaces advertised in comments. The header points at llms.txt and a full-corpus text file. Robots comments are read by people evaluating you as well as by parsers.

Editing robots.txt is the cheapest item on the GEO checklist and the one that most often explains total invisibility. Check it before you spend money on anything else, and check it again after any platform migration, because hosting platforms and staging environments both like to ship a blanket disallow.

How to verify that crawlers actually visit

A permissive robots.txt is an invitation, not evidence. To know whether anything accepted it, read your server logs.

1. Filter access logs by user agent. Search the raw log for the agent strings in the table above. Group by agent, count requests per day, and note which URLs are being fetched. Most CDN and hosting dashboards will do this without a shell.
2. Check status codes, not just hits. A crawler receiving 403, 429 or a soft 404 has visited and got nothing. A bot-protection layer or a firewall rule can block an agent your robots.txt cheerfully allows, and this mismatch is common enough to check first.
3. Watch which pages are fetched. If crawlers only ever see your home page, your internal linking or your sitemap is the problem, not your robots file.
4. Do not trust the user agent string. Any client can send any user agent. Treat the counts as directional. Where vendors publish official IP ranges or support reverse DNS verification, use that to confirm the traffic is genuine before you draw conclusions from it.
5. Remember what logs cannot tell you. A crawl is not a citation. Log data proves access; whether you were named in an answer has to be measured separately by sampling the assistants, as described in how AI assistants choose businesses.

If you want the surrounding context for why any of this matters, start with what generative engine optimization is and GEO vs SEO. If you want the schema side of machine readability, that is schema for AI search. The remaining common questions are collected in the FAQ.