llms.txt Generator
Build a valid llms.txt from your own pages. The H1, the blockquote summary and the H2 link sections come out in the order the proposal specifies, with relative URLs, duplicates and missing notes flagged before you publish.
Your file
One link per line. A bare URL works, Title | URL | note works, and an existing - [Title](URL): note line is passed through unchanged. Name a section Optional and it is emitted last.
llms.txt · 0 links
# Your project name
5 checks to look at
Section "Docs" has no links. An empty H2 tells a model nothing
Section "Guides" has no links. An empty H2 tells a model nothing
No project name. The H1 is the only line the spec requires
No summary. The blockquote is what a model reads before deciding to fetch anything
No links yet. A file with no links is a title and nothing else
Short answer
Llms.txt is a markdown file at the root of a domain that lists the pages you want a large language model to read, each with a one-line note. It was proposed by Jeremy Howard in September 2024 and has a fixed structure: an H1 project name, an optional blockquote summary, optional prose, then H2 sections containing only link lists. It is a recommendation, not a permission file: nothing is obliged to read it, no major AI provider has documented support, and blocking crawlers is still a robots.txt job. Publish it at yourdomain.com/llms.txt and keep it to the pages you would hand someone who asked what you do.
What goes in an llms.txt file, in order
The structure is fixed, and that is the point of it. A parser can read the file without guessing, so every part has one job and one position.
| Part | Markdown | Required | What it is for |
|---|---|---|---|
| Project name | # CrawlRaven | Yes | The only required line. Use the name people search for |
| Summary | > One sentence | Recommended | Read before anything is fetched, so it decides whether the rest matters |
| Details | Plain paragraphs | Optional | Context with no page of its own: caveats, what is behind a login |
| Sections | ## Docs | Recommended | The only grouping signal in the file. Name them by what a reader wants |
| Links | - [Title](url): note | Yes in practice | The index itself. The note is what lets a model choose between two pages |
| Optional section | ## Optional | Optional | Reserved: marks links a model may skip when context runs short |
The note is the whole value
Strip the notes and you have a sitemap in markdown. The sentence after each colon is what lets a model pick your pricing page over your pricing blog post without fetching both, which is the only thing this file does that an XML sitemap cannot.
How llms.txt differs from robots.txt and your sitemap
All three are root-level files about crawlers, which is why they get confused. They answer different questions, and only two of them are enforced by anything.
| robots.txt | XML sitemap | llms.txt | |
|---|---|---|---|
| Answers | What may I fetch? | What exists? | What is worth reading? |
| Format | Directives | XML | Markdown |
| Audience | All crawlers | Search engines | Language models |
| Enforced | Yes, by convention | Read by Google and Bing | No confirmed support |
| Typical size | A few dozen lines | Every indexable URL | Ten to sixty links |
| Blocks anything | Yes | No | No |
The distinction that matters most: llms.txt carries no permission semantics. If your goal is to keep a training crawler out, that is a robots.txt rule against the user-agent names each provider publishes, and the robots.txt generator has those presets built in.
- Publish all three. They cost nothing to keep and each one covers a reader the others do not.
- Never let llms.txt replace the sitemap. Search engines use the sitemap for discovery, and dropping it to publish a markdown index is a trade nobody should make.
- Keep the link sets different. The sitemap wants every indexable URL. This file wants the short list you would hand a new hire.
Is llms.txt worth publishing yet?
Honest answer: the evidence is thin. No major provider has documented reading the file, and Google's John Mueller has compared it publicly to the keywords meta tag, which is not a flattering comparison.
- The cost is close to zero. One file, a few dozen lines, regenerated alongside your sitemap. That is the entire argument for shipping it now.
- The upside is real but unproven. If a model does fetch it, a curated index with notes is strictly better than letting it guess from your navigation.
- The risk is treating it as the work. A site that is slow, blocked or thin does not get cited because it published a markdown file.
- The maintenance is the real cost. A stale file that points at deleted pages actively misleads whatever reads it.
The things that demonstrably drive AI citations are the same ones that drive organic traffic: pages a crawler can reach and render, answers stated directly rather than buried, and structured data that matches the copy. Our guide on ranking in ChatGPT and AI Overviews covers what the evidence actually supports.
How to create an llms.txt file
Seven steps. The fourth and fifth are where a useful file separates from a link dump.
- 01
Name the project in the H1
This is the only line the proposal requires. Use the name a person would search for rather than a legal entity, because the file is read by a model that is trying to match your name to a question.
- 02
Write the blockquote summary
One sentence saying what the project is and who it is for. A model reads this before deciding whether any of the links below are worth fetching, so it is the highest-leverage line in the file.
- 03
Add details only if a link cannot carry them
The free prose block is for context that has no page of its own: what the product does not do, what is behind a login, which docs are current. Everything else belongs in a link with a note.
- 04
Group your pages into H2 sections
Docs, Guides, API, Policies. The heading is the only grouping signal in the file, so a section named Docs that contains blog posts costs you more than having no sections at all.
- 05
Give every link a one-sentence note
The note after the colon is what lets a model choose between two plausible pages without fetching both. A link list with no notes is a sitemap with extra steps.
- 06
Push the low-value pages into an Optional section
Optional is reserved vocabulary in the proposal: it marks links a model may skip when context runs short. Changelogs, archives and legal pages belong there.
- 07
Publish it at the root and keep it current
The file goes at https://yourdomain.com/llms.txt, served as text or markdown. A file that lists three pages you deleted last quarter is worse than no file, so it belongs in the same release step as your sitemap.
Before you list a page here, confirm a crawler can actually reach it. The page indexability checker catches the noindex tags and robots rules that make a listed URL a dead end, and the broken link checker catches the ones that have already gone.
Generate it from the URLs you already have
Pasting a bare URL is enough to get a line: the title is derived from the last path segment so you can paste a column straight out of a spreadsheet or out of the sitemap URL extractor, then fix the titles that read badly.
What this generator does
Spec-ordered output
H1, blockquote, prose, then H2 link sections, every time
Three input shapes
A bare URL, Title | URL | note, or an existing markdown line
Titles from URLs
Paste a column of URLs and get readable labels to correct
Optional handled
A section named Optional is always emitted last
Relative URL check
Flags any link a model could not resolve from the file alone
Duplicate detection
Across every section, not one list at a time
Download and copy
A ready-to-upload llms.txt, or the raw markdown
Runs in your browser
An unpublished site map never leaves your machine
This is a convention, not a standard
Frequently asked questions
What is llms.txt?
A markdown file at the root of a domain that lists the pages you want a large language model to read, with a one-line note on each. It was proposed by Jeremy Howard in September 2024 as a curated index for models working with limited context, not as a permission file.
Is llms.txt the same as robots.txt?
No, and they solve opposite problems. Robots.txt is a permission file that tells a crawler what it may not fetch, and it is honoured by the major crawlers. Llms.txt is a recommendation file that says what is worth reading, and nothing is obliged to read it.
Does Google or OpenAI actually use llms.txt?
There is no confirmed support. Google's John Mueller has publicly compared it to the keywords meta tag, and no major AI provider has documented reading it. Treat it as a low-cost bet on an emerging convention rather than as a channel that is already working.
Where do I put the llms.txt file?
At the root of the domain, at https://yourdomain.com/llms.txt, the same place robots.txt lives. Serve it as text/plain or text/markdown. A file in a subdirectory is not discoverable, because the whole point of the convention is a predictable path.
What is the difference between llms.txt and llms-full.txt?
Llms.txt is an index of links. Llms-full.txt is a companion file holding the full text of those pages concatenated into one document, so a model can read everything in a single fetch. The full file is optional and is usually generated by a build step rather than by hand.
What format does llms.txt use?
Markdown, in a fixed order: an H1 with the project name, an optional blockquote summary, optional free prose, then H2 sections containing nothing but a markdown list of links. Each link may carry a note after a colon.
What does the Optional section mean?
It is the one piece of reserved vocabulary in the proposal. Links under a heading named Optional are the ones a model may skip when it is short of context. Use it for changelogs, archives and legal pages rather than deleting them from the file.
How many links should llms.txt contain?
Enough to answer the questions people actually ask about you, which for most sites is between ten and sixty. The value of the file is the curation, so once it grows past a couple of hundred links it has become a sitemap and stopped doing its job.
Do I still need an XML sitemap if I have llms.txt?
Yes. They serve different readers. Search engines use the XML sitemap to discover every URL you want indexed, and it is a documented, supported format. Llms.txt is a short curated subset for models, and no search engine uses it for discovery.
Will llms.txt get my site cited in ChatGPT or AI Overviews?
Not on its own. Citations follow from being crawlable, being clearly written, and being the best answer to the question. The file may help a model pick the right page once it is already reading your site, which is worth having but is not a ranking mechanism.
Should the notes in llms.txt repeat my meta descriptions?
No. A meta description is written to earn a click from a search result. The note here is written to help a model decide whether to spend context on the page, so it should say what the page contains rather than why it is worth reading.
Can llms.txt stop AI crawlers from training on my content?
No. It carries no permission semantics at all. Blocking training crawlers is a robots.txt job, using the user-agent names each provider publishes, such as GPTBot and ClaudeBot, and a robots.txt file is the only one of the two that is actually enforced.
How often should I update llms.txt?
Whenever the pages it points at change, which in practice means adding it to the same release step that regenerates your sitemap. A file listing URLs that now 404 sends a model to dead pages and is worse than not publishing one.
Other crawl and indexing tools
Website URL Extractor
Crawl any website and download every URL it links to as CSV, TXT, or JSON. Works straight from the live site, no sitemap required.
Use toolRedirect Chain Checker
Trace every redirect hop with status codes and response times. Detect loops, long chains, protocol downgrades, and 302 misuse.
Use toolPage Indexability Checker
Find out why a page is not indexed. Checks status codes, noindex tags and X-Robots-Tag headers, robots.txt rules, canonicals, sitemap membership, and rendered content.
Use toolBroken Link Checker
Find broken internal and external links on any page. Checks up to 50 links for 404 errors, redirects, and timeouts.
Use toolRobots.txt Tester
Analyze robots.txt to see which AI crawlers and search engines are blocked, find sitemaps, and identify access issues.
Use toolRobots.txt Generator
Build a correct robots.txt from presets for WordPress, ecommerce, and AI crawlers. Decide which bots to allow, add your sitemap, copy the file.
Use toolA curated index of pages nothing can reach
A model can only read what a crawler can fetch. CrawlRaven crawls the site, joins it to Search Console and GA4 in one report, and ranks every blocked, redirected and uncrawlable page by the traffic it is costing you.