Fix these 3 files before you pay any AEO company
On this page
Quick Answer
Before hiring an AEO company, ship three files yourself: AI-crawler rules in robots.txt, an llms.txt and Organization JSON-LD. Then judge each retainer only on the content, prompt tracking and authority work that follows.
File work is mechanical. In our first repo-based pilot, a full index of each repository took 24 and 31 seconds, and a two-topic study about two minutes. No index decides which questions a business should answer. That judgment is what a retainer buys.
Key Points
- A July 2026 post by SOCi cited a study of 137,000 domains with llms.txt files and found that 97% of those files received zero requests.
- OpenAI uses separate OAI-SearchBot and GPTBot robots.txt tags, so a site can appear in OpenAI search results while keeping its content out of model training.
- AEO Content prices the paid work from $99 a month on its Growth plan, with 5 prompts tracked weekly, up to $1,000 for Business Plus with Local AEO.
Three files an owner can check alone come before any AEO retainer.
Three files sit at the gate of every website, and an owner can read each of them in the time it takes to finish a cup of coffee: robots.txt, which tells crawlers what they may visit; llms.txt, a plain summary written for language models; and Organization JSON-LD, the structured description of who the business is. They are fixed in scope. They pass or they fail. Beyond them lies the long country that a retainer should actually pay to cross.
The same division showed itself in an unlikely place. In our first pilot of a repo-based knowledge base, run in October 2026 for a live-chat software company, AEO Content indexed 1,368 text files across two Bitbucket repositories, and a re-index after a day of commits re-read only the 14 files that changed. Full-text search across both repositories answered support questions in 5 to 16 milliseconds. The machinery was the small part.
What took judgment was choosing what to say. The study step proposed topics tied to one to four specific files and line ranges, and it dropped a topic outright when the controls it would describe were not yet wired up in the code. There is something instructive in that refusal. A file can be checked by the person who owns it; deciding which questions deserve an answer, and which must wait for the ground to be ready, is the work of a guide rather than a gatekeeper.
So I would ship the three files first, at the cost of an afternoon, and read every proposal afterward for the work that remains, asking each AEO company to price that judgment line by line rather than fold the files into its retainer.
Every traveller learns, sooner or later, that the hardest part of a journey is not the road but deciding which guide to trust before setting out. Owners asking ChatGPT or Perplexity for the best AEO companies meet the same difficulty. The answers rank vendors. None of them says what to finish first.
Answer engine optimization (AEO) is the practice of making a business the one AI assistants name and cite when a buyer asks a question. Part of that practice is fixed and public. Part of it is open-ended and slow.
The fixed part lives in three files on your own server. A 2025 guide on Medium showed how plainly the first of them can be read: Cloudflare's robots.txt carried separate rules for GPTBot and ClaudeBot, and a Sitemap line, all visible to anyone who typed the address. No agency is needed to read such a file. Nor to correct it.
I would ask any owner to do this work alone, before a single proposal is opened, because what remains afterward is the real question: which company will build the content and authority that make an engine choose you, and how will it prove it?
Which three files can an owner open, edit and verify without an agency?
The three files are robots.txt, llms.txt and Organization JSON-LD. Each lives at a known address on your own site, and each can be read and corrected in an afternoon.
- robots.txt, at
yoursite.com/robots.txt: confirm thatGPTBot,ClaudeBotandGoogle-Extendedare not disallowed from the pages you want AI engines to read. - llms.txt, at
yoursite.com/llms.txt: a markdown file listing your key sections and most important pages, each with a brief description and a link. - Organization JSON-LD, in the head of your home page: a block of structured data, written in the schema.org vocabulary, that states who the business is.
Picture a small pool-replastering firm, the example I will carry through this piece. Its owner has never opened any of these files. There is something clarifying in that ignorance, because every one of them can be inspected in a browser, read line by line, and judged against a public specification that no vendor controls.
Start with the gate. A robots.txt file is a plain-text file that tells crawlers which URLs they may access on a site, and Google Search Central describes its purpose as managing crawl load rather than visibility. The common assumption is that the file hides pages; Google states that it is "not a mechanism for keeping a web page out of Google," and that obedience rests with the crawler, since respectable bots follow it while others might not. OpenAI's GPTBot honors these rules, and Google steers Gemini's AI crawling through a separate user-agent, Google-Extended. Correct, for our pool firm, means no Disallow line under those names that covers its service pages.
The second file is newer and less settled. The third is the oldest in spirit: JSON-LD is a format for structured data that machines read as facts, and the Organization type names the business, its address, its logo and its profiles. Schema.org itself notes that it "does not define any notion of 'mandatory' property," so correctness means accurate and consistent, not exhaustive. Its vocabulary moves through named releases every few weeks; Version 30.1 was published on 2026-09-16.
A comparison of 5 sources shows one pattern: each file is small, public and fixed in scope. AEO Content has run 26,577 AI-visibility audits, and I have come to separate these files from everything else such an audit weighs. A file is present and correct, or it is not.
That matters for the buyer. Work with a yes or no answer should not sit inside an open-ended retainer. Our AEO knowledge base guides walk through each check in more detail, and in the end the owner who has looked at these files with their own eyes reads every proposal differently, as a traveller who has seen the map reads a guide's promises with a quieter, more exacting attention.
Do llms.txt and structured data improve AI visibility on their own?
On their own, rarely. The files remove barriers to being read; they do not make a business worth recommending, and the evidence for llms.txt in particular is thin.
The most sobering figure comes from a July 2026 post by the marketing company SOCi, which cited a study of 137,000 domains that had published an llms.txt file. It found that 97% of those files received zero requests. Of the traffic that did arrive, only about 1% came from AI retrieval bots; the rest was SEO audit tools, technical profilers and ordinary web crawlers. "The bots that actually matter aren't reading these files in any meaningful volume," the post concluded.
The providers say little to contradict it. An Ahrefs review, summarized in a November 2025 guide on Medium, reported that as of 2025 no major LLM provider admitted to supporting llms.txt for automated crawling. Anthropic publishes its own llms.txt, yet did not state that its crawlers use the standard. Google's Gemini took its instructions from robots.txt, with no mention of llms.txt at all.
Structured data fares a little better in argument, but not always in practice. One pool-company marketer described adding structured data and FAQ pages to client sites before hiring anyone, and reported that the change barely moved anything. Files describe a business. They do not vouch for it.
So why fix them at all? Because the cost is an afternoon, and the downside of leaving them broken is real. A stray Disallow in robots.txt shuts the door on every crawler that obeys it, and no amount of writing beyond that door will be read. An llms.txt costs almost nothing to publish and serves, at the least, as plain documentation for any human who opens it. My co-founder at AEO Content is a full-stack engineer and content infrastructure architect with 20 years of building enterprise systems, and the discipline that background brings is to treat each file as a switch to set correctly, not as a strategy.
The gap in the evidence deserves saying plainly. There is no benchmark on record for how much an llms.txt alone lifts a site's AEO Rank; a fair figure would need before-and-after audits of the same sites with nothing else changed. Until that exists, any vendor quoting a lift is guessing. Our AEO FAQ on AI search optimization keeps the current answers to these file questions in one place.
There is something of the desert in this, where a well dug in the right place matters more than any amount of grand design above it, yet a well alone feeds no one. The forecast would change only if OpenAI or Google formally adopted llms.txt, or if retrieval bots began fetching it in volume. Neither has happened. What does move a recommendation is harder to build, and it is exactly the work a buyer should be asking an AEO company to price, line by line, before signing.
What do the AI engines themselves say the three files can do?
OpenAI's crawler documentation points site owners to one file for managing its bots. Google's guide to AI Overviews says appearing there takes no additional requirements. So what are the three files for?
We read the pages where OpenAI and Google explain how their systems treat a website, then compared them with what people trying to get businesses named in AI answers report. The files hold up better as a checklist than as a strategy.
Robots.txt is the one file with a documented AI reader. OpenAI says it "uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI." The settings are independent: a site can allow OAI-SearchBot "in order to appear in search results" and disallow GPTBot, keeping its content out of OpenAI's model training. OpenAI's search can take "~24 hours" to reflect an edit. A 2025 guide by AI engineer Arshveer Gahir reports that Google manages Gemini's AI crawling through a robots.txt line for the Google-Extended user agent.
Google's own robots.txt page sets the limits. The file is used "mainly to avoid overloading your site with requests," and "it's up to the crawler to obey them." Robots.txt can close the door to a compliant bot. Neither company's documentation says that leaving the door open earns a mention.
The other two files have weaker support. The same 2025 guide reports that "no major LLM provider currently admits to supporting llms.txt for automated crawling." Google does read structured data, and warns that it "does not guarantee that your structured data will show up in search results, even if your page is marked up correctly." Schema.org, which maintains the vocabulary, "does not define any notion of 'mandatory' property." An Organization block can therefore pass every validator and still say very little.
| File | Who is documented as reading it | What it can do | What the documentation does not promise |
|---|---|---|---|
| robots.txt | OpenAI's OAI-SearchBot and GPTBot. Google-Extended for Gemini, per a 2025 guide | Tell compliant crawlers what they may fetch, and keep OpenAI search separate from training | That every crawler obeys, or that crawling leads to a mention |
| llms.txt | No major LLM provider admits support, per the same 2025 guide | Serve as human-readable documentation, in that guide author's view | Any use by the engines |
| Organization JSON-LD | Google, for search features | Make a feature eligible to appear | That the feature appears, or that any particular field is required |
"There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary."
Google Search Central, AI features documentation
Google adds that ordinary SEO best practices "remain relevant." A marketer for Florida pool companies learned something similar the expensive way. In a July 2026 Reddit post, the marketer described leads "drying up even though rankings held steady" and traced it to prospects asking ChatGPT and Perplexity who should replaster their pool in Tampa and getting other names. Structured data and FAQ pages, added before hiring anyone, hardly moved anything. An agency that said it did AEO was hired in February and paid $4,200; four months later, its work turned out to be "backlinks and meta descriptions." One commenter called standalone AEO "mostly snake oil, it's dependent on SEO and PR fundamentals." One person's experience cannot show what typically happens. It does show how easily file fixes and an old SEO checklist can stand in for the work that matters.
What practitioners say does work sits outside all three files. The summary of a 2025 Lenny's Podcast episode with Ethan Smith, CEO of an SEO growth agency with 18 years in traditional SEO, lists "landing pages, YouTube videos, and Reddit comments" as the tactics that work. It says help-center content can become "your highest-ROI investment" and that AI-generated content does not work. In October 2025, an Early Adoptr podcast host called ranking an entry ticket into the pool an engine draws from; inside it, what counts is being clear, factual and easy to quote. The hosts told businesses to be "authentically everywhere."
Read side by side, these sources split a typical AEO bill in two. One part is permission: robots.txt, llms.txt and an Organization block, all checkable in an afternoon and none promised to change an answer. The other is reputation: pages worth quoting, an honest presence where the engines look, and evidence that the engines now say your name. Our AEO Rank audits score sites on 53 criteria across 5 pillars, and we track separately whether ChatGPT, Claude, Perplexity and Google AI Overviews name our clients, because the two results can differ.
Five checks before the next proposal
- Open your robots.txt and read the OAI-SearchBot and GPTBot lines separately. Blocking model training does not require blocking search.
- Ask what your Organization markup actually tells a reader. Passing a validator guarantees nothing, and schema.org requires no particular property.
- Ask any agency for its deliverables month by month. If the list matches the pool marketer's, backlinks and meta descriptions, ask which items are aimed at AI answers.
- Ask how naming will be measured: which engines, which customer questions, and which metric over time, such as share of voice or citation frequency.
- Ask what off-site and help-center work is planned, and how Reddit or YouTube activity will stay authentic.
How we checked this
We read OpenAI's crawler documentation, three Google Search Central pages, schema.org's project documentation, a 2025 Medium guide, a 2026 Reddit thread, and two 2025 podcast episodes. The 53 criteria, 5 pillars and the list of engines we track come from our own audits.
Google's pages describe Google Search only. They cannot show how ChatGPT, Claude or Perplexity treat structured data. The llms.txt finding comes from a 2025 guide summarizing other people's research, so it may be out of date. The pool account is one person's report. Smith's tactics come from an episode summary, and he runs an agency. We sell AEO services, so we have a stake in where this line falls.
Still unknown: whether any major engine has started reading llms.txt since 2025, and whether Organization markup changes naming outside Google. None of our sources measured either.
- OpenAI, overview of OpenAI crawlers, retrieved November 19, 2026.
- Google Search Central, AI features and your website, retrieved November 19, 2026.
- Google Search Central, introduction to robots.txt, updated December 10, 2025.
- Google Search Central, general structured data guidelines, retrieved November 19, 2026.
- Schema.org, how the project works.
- Arshveer Gahir, guide to agentic search optimization, Medium, November 2025.
- r/SEO, thread on an AEO agency for a pool business, July 2, 2026.
- Lenny's Podcast with Ethan Smith, episode on answer engine optimization, September 2025.
- Early Adoptr podcast, episode on getting ChatGPT to recommend a business, October 2025.
- AEO Content, audit standard and tracking scope, November 2026.
What should you actually pay an AEO company for, and what does it cost?
Pay for the work that gets a business named: content built around real buyer questions, prompt tracking across the AI engines, and authority signals, with plans starting at $99 a month.
Return to the pool firm. Its robots.txt now admits the AI crawlers, its llms.txt lists its services, its Organization markup states its name and address. So the next step is plain: ask each company for its line items in writing, and strike any that only touch those three files. What remains on the list is what the fee should buy.
The paid scope, in my judgment, has three parts. Each is open-ended, which is precisely why it deserves a monthly fee:
- Content built on buyer questions: articles that answer what customers actually ask the engines, written to be quoted, refreshed as the answers shift.
- Prompt tracking: a fixed set of buyer prompts checked weekly or daily across ChatGPT, Claude, Perplexity and Google AI Overviews, so progress is measured rather than asserted.
- Authority signals: mentions, listings and links from sources the engines already trust, the slow ground no file can substitute for.
Here is how AEO Content prices that work, so a buyer has real numbers to set beside any proposal:
| Plan | Monthly price | Articles | Tracking and extras |
|---|---|---|---|
| Growth | $99 | 10 | 5 prompts tracked weekly |
| Premium | $250 | 20 | 20 prompts tracked daily |
| Business | $500 | 30 | Dedicated strategist |
| Business Plus | $1,000 | 44 | Local AEO, 2 backlinks DR 30+ |
Notice what is absent from every row. No line is spent on robots.txt or llms.txt. If a proposal prices them as monthly work, ask what the retainer covers once they are done.
The results worth paying for look like this. At Imagine Dental Arts, AI-referred new patients rose 900%, from about 5 a month to 50. At Understood Care, Google AI Overview citations rose 500%, from 22 to 132, in about five months. Both are measured in who the engines send and cite, and both describe outcomes no file can produce on its own. We stand behind the same standard with a plain promise: named in AI answers in 100 days, or we work free until you are.
Does that mean a separate agency from your SEO firm? Not necessarily. Some practitioners argue that good SEO earns AI mentions by default, and the fundamentals do overlap. The distinction lies in what is measured: an AEO engagement should report which prompts name you, not which keywords rank. For an early startup with no domain authority yet, the logic is the same, only sharper, since the files cost nothing and authority is the one thing that must be built from bare ground. Our approach to research-driven content built on evidence starts from that ground, with the buyer's own questions as the first stones laid, and the next step is learning to read a proposal against them.
What should an owner do before reading the next AEO proposal?
Fix the three files this week, then judge every proposal only on the content, tracking and authority work that remains, because that work alone decides who the engines name.
Our AEO Rank standard scores every site on 53 criteria across 5 pillars, and the files are only a handful of them. The rest is slower ground, and the proposal worth signing is the one that names, before any money changes hands, the buyer questions it will write for and the engines on which it will check the result.
I will admit the trade-off. Fixing files first delays the conversation with an agency by a few days, and some owners will feel that as lost time. It is not. A buyer who arrives with the gate already open asks sharper questions, and sharper questions are, in the end, the only defense against paying for work that was never the hard part.
So the pool firm opens its proposals with a single test. Which prompts will you track, and how often?
Summarize This Article With AI
Open this article in your preferred AI engine for an instant summary.
Frequently Asked Questions
What else do owners ask about AI-crawler files and AEO retainers?
Owners tend to ask about version numbers, Gemini, plan tiers, help-center content and timing, and each question has a plain answer an owner can confirm alone.
Do I need to name the schema.org version in my JSON-LD?
No. Schema.org says its release numbers "are not generally included when you use schema.org," and the formal dataset behind each release "will not change." What matters is that your Organization markup uses terms the current vocabulary still holds, so a periodic look at the latest release is worth more than any version label carried in the code.
Does Google's Gemini read llms.txt?
Nothing on record says it does. Gemini's crawl behavior is governed through robots.txt under Google-Extended, the user-agent name an owner can allow or refuse to control how Google's AI crawls a site, and llms.txt is not mentioned in that arrangement. The door, as it were, is still the old one.
What separates the smaller plans from the larger ones?
Mostly how closely prompts are watched and how much gets written. Our Growth plan tracks prompts weekly, Premium tracks them daily, and the Business tiers add a dedicated strategist, then Local AEO and backlinks. A pool replasterer serving one county will usually learn more from steady weekly tracking than from a larger plan it cannot yet use.
Can help-center articles be written from the product itself?
They can, and the difference shows. In a pilot of our repo-based knowledge base, the same two questions were written twice; once the writer worked from the code alone, the drafts carried zero web sources and zero first-person opinions. Every product claim checked line by line against the code was correct. Nothing borrowed, nothing guessed.
How soon can new content show results, and how do I start?
Sometimes within weeks. For Modern Wealth, organic search sessions rose 220% and Google clicks 61% within three weeks of the first articles. To begin, reach AEO Content through the contact page at aeocontent.ai/contact, and bring your three files already checked, so the first conversation can open on prompts rather than plumbing.
Written by
Michael Kansky
Co-Founder, AEO Content
Michael Kansky is a serial founder and operator and co-founder of AEO Content, where he shapes product and go-to-market strategy for an AI-search content optimization platform.
Connect on LinkedIn


