Publishing is 4 changes, not 1: article, JSON-LD, llms.txt, sitemap
On this page
Quick Answer
An AI-ready publish ships four changes together: the article, its Article and FAQPage JSON-LD, its llms.txt entry, the sitemap. Push the body alone, and the structured signals vanish.
The llms.txt line is the humblest of the four. In June 2025 an r/seopub poster cited NerdyData's crawl of over 350 million websites, which found only 488 using the file, and fretted that answers might cite a .md file instead of the real URL. Fair. So the page, its markup and the sitemap carry the weight...
Key Points
- A complete AEO publish ships four changes in one release: the article page, its Article and FAQPage JSON-LD, its llms.txt entry, and the sitemap that lists it.
- A server-log study of 137,210 domains, reported by Deepak Gupta, found 28% publish llms.txt and 97% of those files got zero requests in May 2026.
- Google's generative AI search guidance says llms.txt files won't harm or help visibility, while the AI Visibility specification v1.8.0 wants an H1 exactly matching the canonical business name.
One release, four files: the article, its JSON-LD, its llms.txt line and the sitemap.
Picture the release. The article goes live, the champagne is poured, and nobody touches the markup, the llms.txt line or the sitemap. Charming. Also dreadfully incomplete...
Answer engines do not admire a page the way a visitor does. A commenter in an r/neocities thread on blocking AI crawlers describes the path plainly: when ChatGPT is asked to open a site, it calls a fetch tool that retrieves the page and reduces it to its content, so the model never reads the raw HTML at all. That same tool can cache pages, and the thread adds that ChatGPT may keep reaching a site for a few hours after robots.txt changes.
So timing matters. A file corrected tomorrow is a file read late.
Which leaves every publisher with four artifacts and one quiet obligation: make them agree, on the same day, in the same release, whether the page is a pricing explainer or a modest glossary entry that nobody at the agency remembers commissioning. Rather a tall order for a tool that only pushes the body.
There is a vanity problem too. Some teams pour every page they own into a single llms-full.txt, and practitioners already wonder aloud whether such swollen files fit inside a model's context window at all. A curated list travels lighter.
And robots.txt, that stern doorman, turns out to be advisory. It enforces nothing. It simply asks, very nicely, and hopes the guests are well brought up...
A publish is rather more than a page going live. In an answer engine pipeline it is four coordinated changes: the article, its Article and FAQPage JSON-LD, its llms.txt entry, and the sitemap update that lists it. Push the body alone and the other three quietly go stale...
Parallel draws the lines rather neatly: "While robots.txt tells crawlers what they can't access and sitemap.xml tells them what exists, llms.txt tells AI systems what matters." Three files. Three jobs. And not one of them updates itself when an article ships.
Too like moving house and telling the bank, but never the post office. The letters keep arriving at the old door.
Our knowledge-base pilot offers the cleaner habit. Every product claim was checked line by line against the code, and every one came back correct. I would hold a publish to the same test: the page, its markup, its llms.txt line and its sitemap entry, read against one another before anyone pops the champagne.
Google, for its part, asks for no special AI files at all. So ask any pipeline vendor what its publish button changes besides the page: the markup, the llms.txt line, the sitemap... and watch for the pause.
What does a complete publish step change on your site?
A complete AEO publish changes four artifacts in one release: the article page, its Article and FAQPage JSON-LD, its llms.txt entry, and the sitemap that lists it.
Before anything is called published, check four things in this order:
- The article is live at its final URL, with the answer and the FAQ visible to a human reader.
- The JSON-LD for that page (Article, plus FAQPage for the questions) sits in the same HTML and says exactly what the page says.
- The llms.txt file gains a line for the new URL if it belongs among the pages that matter, and loses any line the new page retires.
- The sitemap lists the new URL, so crawlers learn it exists at all.
Four changes. One release.
Oh, it sounds fussy. It is rather the opposite. My co-founder is a full-stack engineer and content infrastructure architect with 20 years of building enterprise systems, and in that world a release that ships half its files is not a release at all. It is an incident wearing a charming hat...
The common assumption is that llms.txt is simply the new robots.txt, a velvet rope for AI crawlers. Quite wrong. The Grounding Page Project describes llms.txt as "not a formal IETF or W3C standard, not an access control mechanism, and not a training opt-out." Search Engine Land, in its comparison of robots.txt, sitemap.xml and llms.txt, frames the newer file as something that "acts more like a menu, a curated map" guiding ChatGPT and Perplexity toward a site's most valuable pages and away from outdated ones.
Combining 2 sources points to a tidy division of labour. Each file answers a different machine question:
| Change | Where it lives | Question it answers for a machine | Changes on every publish? |
|---|---|---|---|
| Article page | The new URL's HTML | What is the answer? | Yes |
| Article and FAQPage JSON-LD | A script block inside that HTML | What is this page, who wrote it, which questions does it settle? | Yes |
| llms.txt entry | /llms.txt at the site root | Which pages matter most? | Yes, when the page earns a place |
| Sitemap entry | /sitemap.xml | Which URLs exist? | Yes |
| robots.txt | /robots.txt | Who may enter? | No, access rules only |
In practice, robots.txt sits out the dance. Access is a policy decision, not a per-article one.
The article is the part everyone admires, the frock at the ball. Writing it well is its own discipline, and our research-driven content process treats it as step one, never the whole evening. Skip the other three and the page exists for readers while staying perfectly vague to the machines that decide whether to quote it.
Which brings us to the dreadfully common case: a tool that pushes the body, waves goodbye, and leaves...
Do AI crawlers actually read llms.txt, and is it worth maintaining?
Rarely, by the current evidence. Retrieval crawlers barely request it and Google Search ignores it, yet a fresh, accurate llms.txt still costs little and keeps a site's machine layer consistent.
Oh, the supply side is perfectly thrilling. Deepak Gupta, drawing on Originality.ai's tracking of 3 million sites, reports llms.txt files rising from 4,088 in June 2025 to 36,120 in May 2026, "an 8.8x rise." Demand is another matter entirely. A server-log study of 137,210 domains found that 28% publish the file, and that 97% of those files received zero requests in May 2026, from bots and humans alike.
Zero. Quite.
And the requests that did arrive? Gupta's breakdown is rather deflating: SEO audit tools made up 21.7% of them, while the retrieval crawlers that decide live citations account for "roughly one request in ninety." The most devoted reader of llms.txt, it seems, is the software that checks whether you have one...
Google has been equally cool. On June 15, 2026 it added a note to its AI optimization guidance stating that llms.txt files are not required for Google Search, and its help documentation now says Search simply ignores them.
Practitioners got there first. In a 2025 r/SEO thread, no commenter reported a measurable improvement from llms.txt, and one wrote that proper structured data was what they had "seen working for sure" for Google AI Overviews. Another practitioner, the same year, was blunter still: "LLMs.txt files fix a problem that does not exist."
So why keep it in the publish at all?
I keep it for two unglamorous reasons. First, the tools that audit sites look for it, and a missing or stale file shows up in their reports long before any answer engine notices. Second, a client's question in a review meeting is far easier to answer with a file in place than with an apology.
Duty, darling. Housekeeping.
Ask, then, a smaller question: is the file still true this week? The research section below walks through what those few fetches are actually worth. For the questions people ask most about these files, the answers in our AEO FAQ go a little further into the weeds.
What would settle it is a count nobody has published. We have scored 11,000+ domains across 15 sectors and 28 categories, and the tidy figure would be how many of them carry an llms.txt that disagrees with their own sitemap. That number is not on record yet. Once collected, it would show whether a stale file is a rare embarrassment or simply the usual state of things...
Until then, I treat the file like a guest list at the door. Rarely read. Dreadfully awkward when it still names someone who left at nine.
Our promise to clients runs the other way round: named in AI answers in 90 days, or we work free until you are. A promise like that rests on the page, the markup and the sitemap doing their jobs, every release. The llms.txt line simply has to stay honest beside them, quietly, without fuss.
Which leaves the practical matter of catching a tool that quietly skips it, along with the JSON-LD, the sitemap and, frankly, half the evening...
If answer engines skip llms.txt, why ship it with every publish?
Look at which requests actually reach llms.txt files and you find an odd audience. The most frequent visitors are tools that check whether a site has one.
A server-log study of 137,210 domains by an SEO software company, reported in June 2026, found that 28% published an llms.txt and 97% of those files got zero requests in May 2026, from bots or people. For the few files that were requested, here is who asked.
Audit tools top the list. "That is close to a self-referential market," wrote Deepak Gupta, who analyzed the log data in an August 2026 explainer on the file. Our own audits check for llms.txt, so we are part of that market. At the bottom sit the retrieval bots, the crawlers that supply sources for live AI answers, at what Gupta calls "roughly one request in ninety." A June 2026 search digest added one more detail from the study: "zero AI bots ever went looking for an llms.txt that didn't exist."
Google sends a mixed message too. Chrome added an llms.txt audit to Lighthouse, its site-checking tool, while Google's generative AI search guidance says such files "won't harm (nor help) your visibility or rankings in Google Search, as Google Search ignores them." Pressed by the SEO Lily Ray, Google Search Advocate John Mueller called the file a "temporary crutch" for AI coding agents reading developer docs. OpenAI's and Anthropic's crawler documentation both point site owners to robots.txt.
Gupta's own test in early 2026 put llms.txt on four sites and saw an 11% rise in citations over 90 days, mostly from Claude and Perplexity. He no longer thinks the file caused it, and his account of what works is blunt.
"Structuring pages for extraction moves citations. Schema moves citations. Being reachable by the retrieval crawlers moves citations. An llms.txt does not."
Deepak Gupta, llms.txt explainer, 2026
| File | Its job | Status | Documented support |
|---|---|---|---|
| robots.txt | Tells crawlers what they may access | De facto standard since 1994 | Major crawlers honor it. OpenAI and Anthropic direct site owners to it. |
| sitemap.xml | Lists what exists on the site | Standard documented by Google and Bing | Major crawlers honor it |
| llms.txt | Points AI systems to what matters | Proposal published by Jeremy Howard, September 2024 | Google Search ignores it. Gupta says no major AI company parsed it as of mid-2026. |
| JSON-LD | Describes the page as structured data | Not covered by our sources | Gupta credits schema with moving citations |
If the file does so little, the best reason to keep it in the publish step is to prevent harm. Gupta says a hand-kept index "goes stale the moment your docs change." He argues a stale copy is worse than none, because the few systems that read it "get a confidently wrong picture of your product." GitBook, a documentation platform that generates the file for its customers, is more forgiving: "A stale llms.txt file is still better than nothing, but a fresh one is far more useful."
Both camps land on the same fix. Parallel, which sells web search tools for AI agents, recommends a script that builds llms.txt from the site's own titles and descriptions, so the file "deploys with each content update." Hostinger's free WordPress plugin updates the file whenever the site changes. Gupta objects to keeping the file current by hand, which he says "drifts fast."
One specification asks for even closer agreement. Version 1.8.0 of AI Visibility's llms.txt specification wants an H1 "exactly matching the canonical business name" and an identity that matches the publisher's identity.json file. Each file speaks for a single host, so the apex and www copies should be identical, or one should redirect to the other. Validators report gaps "as conformance findings rather than as syntax errors."
Seen as a whole, the evidence suggests llms.txt works more like a receipt than a signal. AI engines rarely open it. Auditors are its largest single group of readers, and the strictest specification grades it on agreement with the site's other files. A copy written once and left alone drifts away from those files as content changes. A copy regenerated in the same release stays in step at almost no cost. Effort meant to win citations is better spent on the page and its schema, according to the people who have tested the file.
- Have the same release that writes the page, its JSON-LD and the sitemap regenerate llms.txt, and stop editing it by hand.
- After each publish, compare the llms.txt H1 and summary with the business name your other files use, and confirm the apex and www hosts serve the same copy or that one redirects to the other.
- Before anyone spends hours tuning the file, check your server logs: has OAI-SearchBot, PerplexityBot or Claude's search crawler requested it at all?
- When an audit flags llms.txt, ours included, ask whether it checks only that the file exists or also that it agrees with the rest of the site.
- Report a passing llms.txt check as housekeeping, and judge citation work by page structure, schema and whether retrieval crawlers can reach the page.
How we checked this
Our sources were a server-log study of 137,210 domains as reported by Deepak Gupta and by a June 2026 search digest, Google's generative AI search guidance as quoted in that digest, the AI Visibility llms.txt specification, and guides from GitBook, Parallel and Hostinger. We did not read the original log study, and it covers only May 2026. Gupta's 11% figure comes from four of his own sites, and he no longer thinks llms.txt caused it. GitBook and Hostinger offer tools that generate llms.txt, and Parallel sells tools for AI agents. None of the figures here are ours. We run AI-visibility audits and argue in this article for shipping llms.txt with every publish, so we have a stake in the answer. Still unknown: whether a stale file measurably changes how any AI engine describes a business. None of our sources tested that.
- Deepak Gupta, llms.txt spec and working example, August 2026.
- r/DoSEO, AI search digest on the llms.txt myth, June 2026.
- AI Visibility, llms.txt specification v1.8.0, January 2026.
- GitBook, guide to llms.txt, August 2025, updated April 2026.
- Parallel, article on llms.txt, October 2025.
- Hostinger Academy, video on generating llms.txt, June 2025.
How do you check that a publish left all four artifacts in agreement?
Check that every publish leaves all four artifacts in agreement, line by line, the way our knowledge-base pilot checked each product claim against the code.
After each publish, I run five checks, in this order:
- Sitemap. The new URL is listed in the sitemap and loads for a logged-out visitor at that exact address.
- JSON-LD against the body. The Article markup carries the headline, author and date the page shows, the FAQPage markup carries the same questions and answers word for word, and the publisher name is the canonical business name.
- llms.txt identity. The H1 matches the canonical business name, a blockquote summary follows it, and the file is served from the site root as plain text.
- llms.txt curation. The new URL is listed with a short description if it earns a place, and any page it replaces is removed.
- robots.txt. Untouched, unless crawler policy changed on purpose.
Five checks. No heroics.
The identity checks are not my invention. One third-party llms.txt specification, version 1.8.0, asks for an H1 "exactly matching the canonical business name," a blockquote summary immediately after it, and consistency with the publisher's identity.json where that file exists. Files that miss these remain valid under the original convention, and the spec is firm: "Validators MUST report these gaps as conformance findings rather than as syntax errors." So the file passes. The audit still flags it.
On curation, Parallel's guide describes llms.txt as a curated overview typically linking 10 to 15 key pages, and advises: "Review the file on the same schedule you publish content." Its preferred arrangement is generation inside the deployment workflow, so the file "deploys with each content update, so it stays in sync."
Why fuss so over agreement? Our own repo-based knowledge-base pilot is the cautionary tale. Reading the product end to end, the pilot found the admin page describing two-factor sign-in as using an authenticator app, while the actual sign-in screen sends a one-time code by text and email.
Two surfaces. Two stories.
An answer engine quoting the wrong one would be perfectly within its rights... JSON-LD that drifts from the body is the same absurdity in a different frock, and so is a publisher name in the markup that differs from the H1 in llms.txt. Whatever you want AI engines to repeat about you, from articles to sponsored listings AI engines cite, needs one name and one description everywhere it appears.
And robots.txt? Leave it alone on publish day. One crawler operator matches user-agent strings literally, so disallowing GPTbot/1.0 is read as permission for GPTbot/1.4 and its siblings. That belongs to a policy review with its own calendar, not to the line items of each article, which is rather where most tools quietly...
What should the next publish you approve actually ship?
It should ship the article, its Article and FAQPage JSON-LD, a refreshed llms.txt line and an updated sitemap, all in one release, all telling the same story.
A Start & Grow Your Business video, leaning on Search Engine Land, calls llms.txt "not a perfect way to direct them, but it's a way, better than nothing." Quite. Its one real talent is steering AI models away from outdated pages and toward newer content. That talent works only if the file changes on the very day the article does.
My wager for the next two years is rather unglamorous. Index files will keep multiplying while almost none of them are ever fetched, so the advantage drifts to the release that keeps page, markup, index and sitemap in step, every single time, without anyone remembering to ask.
Dull pipelines win. Charming, really.
And somewhere, tonight, a retired page is still on the guest list...
Summarize This Article With AI
Open this article in your preferred AI engine for an instant summary.
Frequently Asked Questions
What else do people ask about publishing for AI search?
Most questions circle the same few files: how large llms.txt may be, whether it touches indexing, who controls crawler access, and how to keep everything current.
How large should an llms.txt file be?
The Webflow tutorial advises keeping it "under 100kb and UTF-8 encoded," saved as plain text. Small. Clean. Easy to parse, rather like a well-engraved invitation.
Does llms.txt change how search engines index my site?
No. An llms.txt file is a plain-text map that points language models toward a site's best pages when they compose answers. The same video says it "doesn't affect how your site is indexed by search engines."
Can llms.txt block AI crawlers?
Not at all. That job stays with robots.txt, and crawler operators may match user-agent strings literally, so each new bot version can need its own line.
How do I get help setting up a complete publish?
AEO Content offers Schema.org structured data implementation, llms.txt creation and its content pipeline (create, score, refine, publish). Start a conversation through the contact page.
Should I write llms.txt by hand or generate it?
Generate it, then edit. One tutorial puts it plainly: "You can edit it for sure, but don't start writing from scratch." Check every URL before the file goes up.
How often should llms.txt change?
Every time a page ships or retires. Generated in the deploy workflow, it travels with each release; kept by hand, it drifts, and quite soon it is recommending a page you deleted in spring...
Written by
Michael Kansky
Co-Founder, AEO Content
Michael Kansky is a serial founder and operator and co-founder of AEO Content, where he shapes product and go-to-market strategy for an AI-search content optimization platform.
Connect on LinkedIn
