llms.txt and GEO housekeeping with Claude Code

Claude Code can generate an llms.txt file from your site's own pages, rebuild it on every deploy so it doesn't drift out of date, and check that every link in it works. It can also handle the GEO housekeeping that matters more: reviewing which AI crawlers your robots.txt allows, finding facts that disagree across pages and structured data, and flagging markup that doesn't match what visitors see. GEO, or generative engine optimization, is the work of getting your business found and cited in AI-generated answers.

Be clear about what llms.txt does and doesn't do before you spend time on it. It's a proposal, not a standard. Google says Google Search ignores these files, and that creating one "will neither harm nor help your site's visibility or rankings in Google Search". The crawler documentation from Anthropic, OpenAI and Google linked in step 4 doesn't mention llms.txt at all, so don't expect the file to make your site appear or be cited more often in AI answers. Treat llms.txt as a cheap, optional extra, generated automatically, and put most of your effort into the housekeeping below. For background on the file itself, read our explainer what is llms.txt.

Step 1: decide whether you want an llms.txt

Here is what's known as of October 2026.

If you decide yes, make it cheap: generated, checked and forgotten. If you decide no, skip to step 4.

Step 2: generate the file from your content

The llms.txt proposal describes a Markdown file with an H1 naming the site ("the only required section"), a blockquote summary, optional detail, and H2 sections of links. A section called "Optional" holds secondary links. Version 2 adds that the file "can be placed at the site root, or at any path within it."

Example prompt: Write a build step that generates /llms.txt from our content. Use the site name as the H1 and the one-sentence description in site-config as the summary. Add H2 sections for Services, Guides and Company, each listing the pages in that section of the sitemap with their title and meta description. Put older posts under an H2 called Optional. Only include pages that return 200, are indexable and are their own canonical. Show me the output before wiring it into the build.

Two rules keep it honest. First, the summary and page notes come from text you've already approved on the site, so the file never says something the site doesn't. Second, the file is output, regenerated on every build. When we rebuilt eseospace.com, a publishing step once regenerated every post from source files and deleted edits made only in the built copy. Edits to generated files don't survive, so fix the source instead.

A person reviews: the first generated file, line by line, for accuracy and for anything you don't want promoted.

Step 3: check the file on every build

1. Every URL in the file returns a 200 status, with no redirects.
2. No URL is noindexed or canonicalized to another page.
3. Every indexable page in the main sections is listed, or deliberately left out.
4. The file is valid Markdown in the proposal's order: H1, summary, then H2 lists.

Example prompt: Add a check to the build that fails if any URL in llms.txt redirects, returns an error, is noindexed or points its canonical elsewhere. Print the offending lines.

Check: break one link on purpose in a test branch and confirm the build fails.

Step 4: review AI crawler rules in robots.txt

llms.txt doesn't control access. robots.txt does. The main AI companies publish separate tokens for different purposes, so you can allow some and block others.

Sources: Anthropic's crawler article, OpenAI's crawler overview and Google's list of common crawlers. OpenAI notes changes can take "~24 hours" to take effect for search.

Example prompt: Read our robots.txt and any robots rules set by the host or CMS. Report, for each AI crawler token in this table, whether it is allowed, blocked or not mentioned, and which rule decides it. Don't change anything.

Whether to block training crawlers while allowing search crawlers is a business decision. Claude Code can implement it once the owner decides. Then fetch the live robots.txt and confirm it matches, since hosts and CDNs sometimes serve their own.

Step 5: make your facts agree everywhere

AI answers draw on what your pages say. If your phone number, address, hours, prices or service area differ between pages, or between a page and its structured data, you're giving every system two answers. In our view, fixing that is basic hygiene for visitors and machines alike, not a guaranteed visibility gain. This is where Claude Code is most useful, because it can read every page at once. It's one of the jobs in our list of 25 practical jobs Claude Code can do on a business website.

Example prompt: List every place the site states our business name, address, phone, hours, prices and service areas, including structured data, meta descriptions, the footer and llms.txt. Group identical values and show me every value that disagrees with the contact page.

A person decides which value is correct. Then fix them together, using our routine for changing a fact everywhere on your site.

Step 6: keep structured data accurate, not elaborate

Google says "Structured data isn't required for generative AI search" and that there's "no special schema.org structured data that you need to add" for AI Overviews or AI Mode. Markup still helps describe your pages, and it has to be honest: Google's policies say "Don't mark up content that is not visible to readers of the page" and don't mark up "fake reviews".

Ask Claude Code to compare each page's markup with its visible content and report mismatches, such as a rating with no reviews on the page or an address that differs from the footer. Our guide to fixing structured data with Claude Code covers this at scale.

Step 7: avoid the GEO tactics that backfire

• Pages for every query variation. Google says creating separate content for every variation "primarily to manipulate rankings or generative AI responses in Google Search violates Google's scaled content abuse spam policy".
• City-page networks. Google's doorway examples include "multiple domain names or pages targeted at specific regions or cities that funnel users to one page".
• Fake authority. Google calls "AI-generated headshots, made-up names, or false credentials" a form of deception. Keep author and About pages true.
• Seeding fake sources. Anthropic's Usage Policy update, which takes effect November 12, 2026, prohibits manipulating "the sources from which search engines or AI systems draw answers" with content that misrepresents its origin, authorship or independence.
• Fake freshness. Don't change dates on pages that haven't substantially changed.

Step 8: make it a recurring check

Housekeeping drifts. Package steps 3 to 6 as a reusable routine that you, or someone you trust, runs before each release and once a month: llms.txt links, crawler rules, fact consistency and markup mismatches. Claude Code reports, and a person decides what to fix. Our guide to Claude Code for technical SEO covers the wider audit this fits into, and SEO practitioners will find more in Claude Code for SEO specialists.

Conclusion

llms.txt is optional and unproven for search visibility, so if you publish one, let Claude Code generate and check it automatically. The bigger wins are ordinary: crawler rules the owner has chosen, facts that agree everywhere, and markup that matches the page. No one can guarantee citations in AI answers, so be wary of anyone who does. For more on how AI search works, see our GEO and AEO Guide, and if you want an audit or ongoing help, see our technical SEO services. More guides are in our Claude Code Guide.

Frequently asked questions