Your team wants its public documentation to appear in search answers.
It may not want that same documentation collected for potential model training.
Those are different preferences. Treating them as a single “allow AI” setting can create a configuration that does neither job properly.
The major providers do not all use the same crawler for every purpose. OpenAI separates search crawling from potential training collection. Anthropic documents separate bots for training, search and user requests. Perplexity distinguishes its search crawler from user-requested fetching.
This guide explains those differences, provides an illustrative policy, and shows how to review access without treating robots.txt as a security control.
The goal is not to let every bot through. It is to make an informed decision about each documented use.
The short answer
If your policy is to allow search discovery while limiting collection for potential training, start by identifying the correct user agents.
For OpenAI, OAI-SearchBot is associated with search, while GPTBot is associated with content that may be used for training.
For Anthropic, Claude-SearchBot supports search, while ClaudeBot collects content that may contribute to training.
Perplexity describes PerplexityBot as a search crawler, not a crawler for training foundation models.
Do not confuse permission with selection.
Allowing a crawler can remove an access obstacle. It does not guarantee that a page will be indexed, cited, recommended or visited.
First, understand what robots.txt can and cannot do
robots.txt communicates crawl rules to crawlers that follow the Robots Exclusion Protocol.
It belongs at the root of the host it governs:
https://example.com/robots.txtIf your documentation lives on a separate subdomain, review that host's configuration too:
https://docs.example.com/robots.txtGoogle's documentation explains that robots.txt rules apply to the host where the file is located.
It is not access control
A robots.txt rule does not authenticate visitors or prevent someone from requesting a public URL.
Google explicitly warns that the file cannot enforce crawler behavior.
Keep private dashboards, customer records, internal documents and administrative tools behind appropriate authentication and authorization.
Writing their paths in robots.txt does not make them private.
It is not a universal removal tool
For Google, blocking crawling is not the same as preventing a URL from appearing in search.
A blocked URL may still appear if Google discovers it through links.
Google recommends appropriate indexing controls, removal, or password protection depending on the situation.
That is why “block this bot” and “remove this information” should be treated as separate tasks.
Know which bot does what
The following summary reflects the provider documentation linked here. Check those sources again before changing production rules.
| Provider | User agent | Documented purpose |
|---|---|---|
| OpenAI | OAI-SearchBot | Search discovery for ChatGPT |
| OpenAI | GPTBot | Crawling content that may be used for model training |
| OpenAI | ChatGPT-User | Certain user-initiated page visits and actions |
| Anthropic | Claude-SearchBot | Crawling to improve search results |
| Anthropic | ClaudeBot | Collecting content that may contribute to model training |
| Anthropic | Claude-User | Retrieving pages in response to user requests |
| Perplexity | PerplexityBot | Surfacing and linking websites in search results |
| Perplexity | Perplexity-User | Retrieving pages for user-requested answers |
The names are easy to misread.
ClaudeBot is not the same as Claude-SearchBot. GPTBot is not the same as OAI-SearchBot.
A policy written for one should not be assumed to control the other.
Decide the policy before editing the file
Have the website owner, product team and relevant legal or security reviewer agree on the intended outcome.
A simple policy worksheet helps.
| Question | Decision to document |
|---|---|
| Should public marketing pages be available to search crawlers? | Yes / No / Selected pages |
| Should public documentation be available? | Yes / No / Selected sections |
| Do we want to restrict collection for potential training? | Provider-specific decision |
| How should user-requested page retrieval be handled? | Review each provider's behavior |
| Which resources must remain private? | Enforce through authentication |
| Who approves changes? | Named owner |
| Who checks the result? | Named technical reviewer |
This is not just a wording exercise.
“Allow search, restrict potential training” is a different policy from “block all automated access.”
It also does not mean private pages become public. Authentication should remain in place regardless of crawler preferences.
An illustrative robots.txt policy
The following example represents this limited policy:
- Public marketing and documentation pages may be crawled for search.
- Two example non-content areas should not be crawled by the named search crawlers.
GPTBotandClaudeBotare disallowed.- User-requested fetchers are not configured by this example.
Example policy only. Review against your current configuration. Do not replace an existing production robots.txt without checking every applicable rule.
# Example only.
# Replace the domain and paths with reviewed values.
# These path rules are not security controls.
User-agent: *
Disallow: /internal/
Disallow: /account/
User-agent: OAI-SearchBot
Disallow: /internal/
Disallow: /account/
User-agent: Claude-SearchBot
Disallow: /internal/
Disallow: /account/
User-agent: PerplexityBot
Disallow: /internal/
Disallow: /account/
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlThe sitemap line is appropriate only if that is the real location of your sitemap.
The example deliberately repeats the two path restrictions in the named search-bot groups.
Do not assume that a crawler will automatically combine the wildcard group with its more specific user-agent group. Google, for example, documents selecting the most specific matching group rather than combining it with the wildcard group.
Provider behavior should still be checked against current documentation and the configuration actually deployed.
Why there is no “Allow: /” line
In the ordinary robots.txt model, paths that are not disallowed are available for crawling.
You do not need a blanket allow line simply to make an otherwise unrestricted public page crawlable.
The larger issue is preserving the restrictions your website already needs. A short example cannot account for every production route, legacy rule or subdomain.
User-requested fetchers need a separate review
A person asking an assistant to read a URL is not necessarily the same event as an automatic crawler discovering pages.
Providers describe this differently.
ChatGPT-User
OpenAI says ChatGPT-User supports certain user actions and is not used for automatic web crawling.
Because those actions are user-initiated, robots.txt rules may not apply.
OpenAI also says OAI-SearchBot, not ChatGPT-User, is the control for search opt-outs and automatic search crawling.
Perplexity-User
Perplexity describes Perplexity-User as a user-requested fetcher rather than a training or automatic web crawler.
Its documentation says the fetcher generally ignores robots.txt because a user requested the page.
Claude-User
Anthropic says Claude-User retrieves pages for user requests and that its bots honor robots.txt directives.
It describes disabling Claude-User as preventing its system from retrieving content in response to a user query.
The practical lesson is simple: do not assume one robots.txt strategy governs every provider's user-directed retrieval.
And do not rely on robots.txt when the content needs to remain private.
robots.txt may allow a crawler while your firewall blocks it
A file can be correct while the website is still inaccessible.
A content delivery network, web application firewall, bot challenge, rate limit or application error may prevent the crawler from receiving the intended page.
OpenAI recommends allowing its published searchbot IP ranges where necessary. Perplexity provides published IP ranges and guidance for combining IP verification with user-agent matching.
Have the infrastructure owner inspect:
- Requests reaching the CDN or firewall
- Relevant block and challenge events
- HTTP response codes
- Whether the intended page content is returned
- Whether verified requests are being rate-limited
- Whether rules are scoped only to the intended resources
Do not disable bot protection for the entire website simply to make a crawler test pass.
A user-agent string is not proof of identity
A client can claim to be a known crawler.
Do not build a broad security bypass using only a user-agent substring.
Where a provider publishes verification guidance or official IP ranges, use the current documented method. Keep authentication and sensitive application protections intact.
Perplexity specifically recommends combining its user-agent checks with source-IP verification.
Test the deployed result, not just the repository
A robots.txt file in source control does not prove that production serves the same file.
Start with the public endpoint.
curl -i https://example.com/robots.txtReplace the domain with the real host.
Confirm:
- The request succeeds
- The response is the expected plain-text file
- The content matches the approved policy
- A cached or generated version has not replaced your edit
- You checked the correct hostname
Then inspect representative public pages.
A normal request or a request with a custom user-agent can help diagnose response behavior, but it does not prove that the request came from a provider or that a real crawler can pass IP-based verification.
A command-line client also does not enforce robots.txt merely because you supply a crawler name.
Use request tests, configuration review and verified logs together.
Separate crawling, indexing and security checks
These are related, but they are not interchangeable.
| Objective | Appropriate control or check |
|---|---|
| Communicate that a compliant crawler should not crawl a path | Applicable robots.txt rules |
| Keep confidential content private | Authentication and authorization |
| Prevent a public page from appearing in Google Search | Appropriate Google-supported indexing controls |
| Confirm a crawler can receive a page | Response checks and verified request logs |
| Improve the usefulness of a public page | Accurate content and accessible documentation |
| Observe citations or recommendations | A separate, documented answer sample |
Watch for the robots.txt and noindex conflict
For Google to process a noindex directive on a page, it needs to be able to crawl that page.
If robots.txt blocks the crawl, Google may not see the directive.
Do not combine controls without understanding the intended outcome.
And do not assume that Google's documented indexing behavior automatically describes every other provider.
Common mistakes worth checking
Blocking the wrong bot
A GPTBot rule is not an OAI-SearchBot rule.
A ClaudeBot rule is not a Claude-SearchBot rule.
Check the exact documented token rather than relying on a familiar brand name.
Copying a universal “AI bot list”
Lists become outdated, and providers assign different purposes to different agents.
Start with the services your policy actually covers.
Replacing existing restrictions
A copied template may omit paths your current configuration handles.
Review the complete file and the effect of specific user-agent groups.
Treating training restrictions as deletion requests
A crawler preference should not be presented as removing information from already trained models.
Anthropic, for example, describes restricting ClaudeBot as a signal about future materials and training collection.
Treating an allowed crawl as a guaranteed citation
A successful request proves access at that moment.
It does not prove selection, ranking, recommendation or future traffic.
Using llms.txt as an access-control file
A documentation guide is not a replacement for crawler rules, authentication or firewall configuration.
Use each mechanism for its actual job.
A safer rollout checklist
Before the change:
- Document the intended search and training preferences.
- Save the current configuration.
- Review each relevant hostname.
- Identify existing rules that must remain.
- Assign an approver and a rollback owner.
During the change:
- Make the smallest justified edit.
- Avoid unrelated firewall changes.
- Check that private areas remain protected.
- Confirm the deployed file matches the approved version.
After the change:
- Inspect representative public pages.
- Review verified crawler requests where available.
- Check firewall and application errors.
- Record the deployment time and exact changes.
- Revisit current provider documentation periodically.
OpenAI says search systems may take approximately 24 hours to adjust after a robots.txt update. Perplexity says its systems may take up to 24 hours to reflect changes.
Those are configuration-adjustment statements, not promises of indexing or citations within a day.
What should you measure afterward?
Keep operational checks separate from marketing outcomes.
Operational checks
- Is the intended file available?
- Can verified search crawlers reach the public pages?
- Are requests returning the intended content?
- Are private pages still protected?
- Did error rates or unwanted load change?
Visibility observations
If your team also tracks generated answers, use a fixed question set and record the platform, date, prompt, response and cited links.
An answer changing after a configuration update is an observation. It does not establish that the update caused the change.
This distinction keeps a useful technical fix from turning into an unsupported performance claim.
Frequently asked questions
Can I allow ChatGPT search while restricting GPTBot?
OpenAI documents OAI-SearchBot and GPTBot as independent settings.
That allows a publisher to express different preferences for search discovery and potential training collection.
Does blocking ClaudeBot also block Claude search?
Not by itself.
Anthropic documents ClaudeBot and Claude-SearchBot as separate agents with different purposes.
Review all applicable rules and infrastructure behavior.
Is PerplexityBot a training crawler?
Perplexity says PerplexityBot is designed to surface and link websites in search results and is not used to crawl content for foundation models.
Will blocking a crawler make my private content safe?
No.
Use authentication and authorization for private content. robots.txt cannot enforce access restrictions.
Should every SaaS company use the example in this article?
No.
It illustrates one policy. Your legal preferences, public routes, existing restrictions, infrastructure and provider requirements may be different.
Will allowing search crawlers make assistants recommend my product?
Not necessarily.
Access removes a possible obstacle. It does not establish product fit, source selection, ranking or recommendation.
Make the policy explicit
The useful question is not “Should we allow AI bots?”
It is:
“Which documented services should access which public resources, for which purposes, under what controls?”
Answer that first. Then review robots.txt, infrastructure behavior and private-page protection as separate parts of the implementation.
A clear policy is easier to maintain than a copied list of bot names. It also gives your team a better way to investigate problems when search access, crawler requests and answer visibility do not line up.
A practical guide to the llms.txt proposal: file structure, Markdown links, validation, maintenance, and the claims you should not rely on.
A practical audit for SaaS teams: check search access, inspect cited sources, clarify product information, and track changes without guessing.
Explore ZocoPilot's available features and decide whether they fit the way your team investigates search visibility and content gaps.
Explore ZocoPilot