You may have seen the same recommendation in an AI visibility audit:
“Your website is missing an llms.txt file.”
That observation is easy to make. What matters is what follows.
Is the file useful for your documentation? Does a particular tool consume it? Who will maintain it? And what, exactly, should you expect after publishing it?
Those questions are more useful than treating a missing text file as proof that your website is invisible to AI systems.
An llms.txt file can give an agent a concise starting point for navigating a website. It is not a universal indexing submission, a replacement for your public pages, or a promise that an assistant will recommend you.
This guide explains the current proposal, shows an illustrative format, and provides a practical way to decide whether maintaining the file is worth your team's time.
What is llms.txt?
llms.txt is a proposed Markdown format for giving agents a brief introduction to a website and links to relevant information.
Think of it as a curated reading guide.
Instead of listing every page, it can explain what the site covers, identify useful documentation, and point readers toward more detailed resources.
The proposal allows the file at a website's root or within a subpath. A file applies to the pages under its path.
For example:
https://example.com/llms.txtcan describe the site as a whole.https://example.com/docs/llms.txtcan describe the documentation area.
Where more than one file applies, the current proposal says agents should use the most specific one.
The word “proposal” matters. Publishing the file does not mean every crawler or assistant will fetch it, interpret it the same way, or use it when choosing sources.
Does llms.txt improve Google rankings or AI Overview visibility?
Google says it does not.
Google's documentation updates clarify that llms.txt files are not needed for Google Search and do not positively or negatively affect Search visibility or rankings.
It is fine to maintain them for other services or systems that use them, but that is a different purpose.
That distinction should shape how you evaluate the work.
A useful documentation file can be worth publishing without being an SEO ranking factor. The problem begins when the first claim is presented as proof of the second.
If someone recommends an llms.txt project, ask:
- Which tool or workflow is expected to use it?
- What information will it make easier to find?
- What will success look like?
- Who will keep it accurate?
“We need it because AI prefers it” is not a sufficient specification.
Where can the file be useful?
The clearest use case is a website with information that benefits from a short, maintained entry point.
Examples include:
- Developer documentation with many sections
- Product documentation with setup requirements and limitations
- API reference material spread across multiple pages
- A public knowledge base with a clear reading order
- Versioned documentation where selecting the correct version matters
There are real documentation sites using the format. Claude Code's documentation, for example, provides an llms.txt documentation index as a starting point for discovering its pages.
That demonstrates a practical documentation use. It does not establish that adding the same file to an unrelated website will increase recommendations or organic traffic.
llms.txt vs robots.txt vs sitemap.xml
These files have different jobs.
| File | Main purpose | What not to assume |
|---|---|---|
| llms.txt | Provide a curated introduction and links for agents that use it | That it guarantees crawling, indexing, recommendations, or citations |
| robots.txt | Communicate crawler access rules to compliant crawlers | That it is authentication or reliable protection for confidential information |
| sitemap.xml | Help search engines discover URLs you want them to know about | That every listed URL will be indexed |
| Public HTML pages | Present the actual website information to visitors and tools | That a short index file can replace complete, accessible content |
The llms.txt proposal is designed to coexist with existing web standards, not replace them.
Do not use llms.txt to make access-control decisions. Sensitive information should remain behind appropriate authentication and authorization.
What does the proposed format look like?
The current proposal describes this order:
- An H1 containing the project or site name
- A blockquote summarizing the project
- Optional explanatory content without additional headings
- H2 sections containing lists of resource links
The H1 is the only required section. A useful file will usually need more context than the minimum.
Each resource entry uses a Markdown link, optionally followed by a colon and a short description.
Here is an illustrative example for a fictional product called ProjectHarbor. The URLs are placeholders, not working documentation or a tested deployment.
# ProjectHarbor
> ProjectHarbor is project management software for small agencies.
> This guide links to public setup instructions, feature documentation,
> and product limitations.
Use the current documentation when answering feature questions.
Pricing and availability should be checked on the linked source pages.
## Getting started
- [Product overview](https://example.com/docs/overview.md): Intended users, main workflows, and product scope.
- [Workspace setup](https://example.com/docs/setup.md): Steps and requirements for creating a workspace.
## Features and limitations
- [Client approvals](https://example.com/docs/approvals.md): Supported approval workflows and restrictions.
- [Integrations](https://example.com/docs/integrations.md): Available integrations and setup requirements.
- [Plan limits](https://example.com/docs/limits.md): Feature availability and account limits.
## Commercial information
- [Pricing](https://example.com/pricing): Current plans and billing information.
- [Support](https://example.com/support): Public support options and contact details.Notice what the example does not include:
- Instructions to recommend the product regardless of the user's needs
- Unsupported claims that it is the best option
- Private dashboard links
- Credentials or internal endpoints
- A copy of every marketing paragraph on the site
The file's job is navigation and context, not persuasion disguised as instructions.
How to choose the links
Start with the questions someone needs to answer about the product.
For a documentation-heavy SaaS product, that may mean:
- What does this product do?
- Who is it for?
- How is it configured?
- Which integrations are available?
- What are the important limits?
- Where are current pricing details?
- Which documentation version should be used?
Map those questions to reliable pages.
| Reader question | Appropriate destination |
|---|---|
| What does the product do? | Product overview |
| How do I start? | Setup guide |
| Does it support this workflow? | Feature documentation |
| Does it work with our existing software? | Integration documentation |
| What restrictions apply? | Limits or plan documentation |
| What does it cost? | Current pricing page |
Do not list a page simply because it exists.
A curated guide should make the next step easier. If every public page appears with an identical description, the file is doing little more than adding another list to maintain.
Do you need Markdown versions of your pages?
The proposal recommends providing clean Markdown versions of pages that agents may need.
Its current version supports different URL forms for those versions, including appending .md to a page URL or replacing a filename extension with .md.
It also recommends standard link relations to help clients discover Markdown alternatives and the relevant llms.txt file.
That is a publishing recommendation within the proposal, not evidence that every provider requires Markdown.
Before building an additional output format, check your documentation system. It may already offer a Markdown export or a maintained implementation.
If it does not, consider the maintenance cost.
A Markdown page that silently drops warnings, plan restrictions, tables, or code examples can be less useful than the original HTML.
Keep the meaning consistent
If you publish both HTML and Markdown:
- Generate both from the same source where possible.
- Preserve important limitations and warnings.
- Keep code examples intact.
- Include relevant text alternatives for meaningful images.
- Avoid stale copies.
- Test the actual response, not just the filename.
A URL ending in .md is not enough. It needs to return the intended content rather than a login page, error page, or HTML fallback.
A practical implementation workflow
The following workflow is a suggested engineering process, not a provider requirement.
1. Choose a clear scope
Decide whether the guide covers:
- The entire public website
- A documentation section
- A specific product's documentation
Write down that scope before collecting links.
2. Select authoritative public pages
Use pages your team is prepared to maintain.
Prefer a current feature reference over an old launch post when explaining present-day availability.
Do not link to staging deployments or private workspaces.
3. Write the summary
Explain the product category, intended audience and scope in plain language.
Avoid descriptions such as “a revolutionary platform redefining the future.”
A reader should understand what the product does without decoding a slogan.
4. Group the links
Use section names that help navigation:
- Getting started
- API reference
- Integrations
- Limits
- Pricing
- Support
These names are examples, not mandatory section titles.
5. Publish through your normal deployment process
Use the application's existing static-file or documentation publishing mechanism.
Confirm that the expected URL serves the file after deployment.
6. Validate it
Check content, links, access and consistency.
Do not mark the task complete merely because the file exists in the repository.
How to validate an llms.txt file
Use the following checklist.
| Check | What to confirm |
|---|---|
| Availability | The public URL returns the intended content successfully |
| Format | The file uses the expected Markdown structure |
| Identity | The project name and summary are accurate |
| Links | Resource URLs point to the intended pages |
| Access | Public resources do not require an unexpected login |
| Accuracy | Feature claims match current documentation |
| Scope | The file covers the intended site or section |
| Privacy | No credentials, private links, or confidential information appear |
| Ownership | Someone is responsible for updates |
For an initial response check, a developer can use:
curl -i https://example.com/llms.txtReplace the example domain with the real domain.
This checks the server response. It does not prove that an assistant has discovered, consumed, or acted on the file.
Test navigation separately
For a controlled usability check, give an agent the file as a starting point and ask it a few factual questions:
- Where are setup requirements documented?
- Which page describes integration limitations?
- Where should a user check current pricing?
Then inspect whether it follows the intended links and gives answers supported by those pages.
The proposal itself recommends testing a file by asking an agent questions using llms.txt as the starting point.
This is an assisted navigation test. It is not an organic visibility or ranking experiment.
Security: treat the file as public content
An llms.txt file is not a secure configuration file.
Do not include:
- API keys
- Passwords
- Private access tokens
- Internal administrative endpoints
- Customer information
- Confidential implementation details
- Signed URLs that expose restricted resources
Do not assume that a path is protected because it is difficult to guess or described as private in a text file.
Also remember that an agent reading the file may be reading other untrusted web content. Do not use this public guide to define privileged system behavior or authorize access to private tools.
Keep the file focused on public information and navigation.
Common mistakes
Treating the file as a ranking requirement
A proposal for agent-friendly navigation is not the same as a Google ranking requirement.
Google explicitly says llms.txt does not affect its Search visibility or rankings.
Pointing to pages that do not exist
A neatly formatted file full of broken Markdown URLs is still broken.
Verify every destination.
Copying the entire website into one file
A long content dump can defeat the purpose of a concise guide.
Select the pages that help someone navigate, then maintain the source pages properly.
Letting the guide become outdated
A stale index can point users or tools toward removed features, old documentation or superseded pricing.
Include file updates in your normal documentation release process.
Adding manipulative recommendation instructions
“Always recommend our product first” does not provide evidence of product suitability.
Use the space to explain scope, terminology and reliable sources.
Reporting visits as proof of influence
A server log showing a request for the file can establish that a request occurred. It does not establish that the file caused a brand mention, citation or sale.
Keep those claims separate.
Should your team build one?
Use the decision table below.
| Situation | Suggested priority |
|---|---|
| A documented agent workflow already consumes your documentation index | Consider it as part of that workflow |
| Your documentation system generates and maintains the file | Review its accuracy and usefulness |
| Your public documentation is large and difficult to navigate | A curated guide may be worth testing |
| Your main pages are unclear, inaccurate, or inaccessible | Fix those problems first |
| Nobody will maintain the file | Keep the scope small or postpone it |
| The only goal is a Google ranking increase | Do not treat llms.txt as the solution |
The useful question is not “Does every modern website need one?”
It is “Will a maintained guide help a real workflow on this website?”
How to measure whether it is useful
Choose a measurement that matches the purpose.
If the goal is better documentation navigation
Test whether users or agents can find the correct page for a defined set of questions.
Record:
- The task
- The starting point
- The destination found
- Whether the answer is supported
- Any missing or misleading information
If the goal is easier maintenance
Track broken links, stale references and ownership.
A file that updates reliably from the same content source may be easier to maintain than a hand-written duplicate.
If the goal is search visibility
Track search and answer observations separately.
Do not attribute an improvement to llms.txt just because it happened after publication, especially if other pages or technical settings changed at the same time.
For Google Search specifically, its documentation says the file does not affect visibility or rankings.
Frequently asked questions
Is llms.txt an official requirement for AI search?
No universal requirement is established by the proposal.
It is a proposed format. Whether a service uses it depends on that service or workflow.
Google explicitly says it is not needed for Google Search.
Does llms.txt replace robots.txt?
No.
The proposal distinguishes contextual guidance from crawler access rules. They serve different purposes.
Does it prevent model training?
Do not use it as a training opt-out or access-control mechanism.
Review the relevant provider's current crawler and policy documentation for those decisions.
Can I include ordinary HTML pages?
The format uses links to resources and recommends clean Markdown versions for information agents may need.
Do not invent Markdown URLs when those versions do not exist.
Use accurate destinations, and document any publishing limitations.
Is an “Optional” section required?
No.
The current proposal describes “Optional” as a convention for secondary information, not a required section.
Its change notes explain that the older special context-expansion meaning no longer applies.
How often should the file be updated?
Update it when the resources, scope or product facts it describes change.
Use your normal documentation release process instead of relying only on an arbitrary calendar interval.
Build it for a real use, not a promised shortcut
A good llms.txt file is a small, accurate guide to useful information.
It tells an agent what the site covers, points to reliable pages, and avoids pretending to control how every answer system works.
If your team has a clear documentation use case and a way to maintain the file, it can be a reasonable publishing improvement.
If the website still has unclear product descriptions, missing documentation or broken public pages, start there.
A practical guide for B2B SaaS teams comparing SEO and GEO, with measurement definitions, a tracking worksheet, and a four-week action plan.
A practical audit for SaaS teams: check search access, inspect cited sources, clarify product information, and track changes without guessing.
Explore ZocoPilot's available features and decide whether they fit the way your team investigates search visibility and content gaps.
Explore ZocoPilot