Ask an AEO specialist how it checks ChatGPT: API vs logged-in answers

Same question, two capture methods. The answer a buyer reads depends on which one asked.

In a 2025 test, one prompt went through OpenAI's API and the ChatGPT app. The API's answer came back "much different," with a 50+% chance its links were broken.

The developer who ran it posted the details on r/ChatGPTCoding. The API answered in about 7 seconds. The app took "a minute+." The API also cited third-party comparison blogs instead of the retailer pages the app had found. Two machines. One name on the door.

AI visibility tracking is the practice of measuring how often ChatGPT and other answer engines name your brand. Here is the trouble with it. Your search rankings cannot stand in for it: a December 2025 practitioner video reported that 40% of Google AI Overview citations rank beyond position 10, so an AEO specialist that checks ChatGPT has to read the answers themselves. Before your next renewal, take one tracked prompt from the report, type it into your own logged-in ChatGPT account, and see whether your brand turns up.

Our own pilot hit the same split in a different trade. Reading a product from the admin setting to the screen users actually see, it found the admin page describing two-factor sign-in through an authenticator app, while the real sign-in screen sent a one-time code by text and email. The documentation was confident. The screen was right.

Even the honest door wobbles. So ask a specialist what it counts besides mentions: one of our clients went from a quiet site to 10 booked calls in 3 weeks, and a booked call stays put whichever door the sample came through.

What does the logged-in ChatGPT app add that the API leaves out?

The logged-in ChatGPT app wraps each prompt in a system prompt, history, memory and custom instructions before the model sees it. A raw API call adds none by default.

Before you trust a single ChatGPT figure in a vendor report, get these answers in writing:

1. Which access point produced each answer: the logged-in app or a raw API call.
2. Whether memory, custom instructions and chat history were on, off or wiped for the run.
3. Whether web search was running when the answer was captured.
4. Which model and which temperature the API calls used, if any were made.

A thread on r/ChatGPTPro laid the machinery out cold. One commenter wrote that the web app "takes a string from you and adds assistant history, user history, a system prompt, and other context to that string before it sends it to the LLM on the server." Another was blunter: "ChatGPT has a system prompt, the API does not." The person who started the thread had paid someone to script a very detailed prompt through the API, got weaker output than the app gave, and still wasn't sure which model the script was calling. Nobody fixed it. The thread just stopped.

A separate developer thread put the split in plainer terms: the API "doesn't include the system prompt by default," while the app "will include the memory, custom instructions etc with your prompt." Read 4 practitioner sources side by side and they agree on the anatomy, even where they argue about the cause.

The common assumption is that one model name means one answer. It doesn't. Even in 2023, early API users noted that the API exposed a "system" message role the regular chat interface did not offer. Search is a separate layer again. Regular ChatGPT can answer from what the model already holds, without touching the web, and search gets layered on top when it runs. So a brand can turn up in one mode and be nowhere in the other.

The threads say nothing about location, so I won't either. What matters is simpler. History and memory differ from one account to the next. So run a small test of your own: ask the same buying question in your account and in a colleague's, and write down whether the brands named change. Keep those dated screenshots, because they are the cheapest baseline you will ever own, and any vendor's report can be held up against them.

At AEO Content we promise a client will be named in AI answers in 90 days, or we work free until they are. That promise depends on "named" meaning named in the answer a buyer reads, inside the app, with every layer switched on. If you want the ground covered first, our FAQ on AI search optimization takes the basics one question at a time. The harder question is how the people selling dashboards actually collect what they show you.

How do AI visibility tools and AEO agencies actually sample ChatGPT?

Our team has run 26,577 real AI-visibility audits. Trackers mostly sample ChatGPT two ways: automated prompt panels at scale, or manual checks typed into a logged-in session.

The first way is scale. In 2025, tracking platforms were selling themselves on the size of their prompt databases: one claimed 405M+ prompts, another more than 1.5 billion real user prompts. Yet a ranking scored with that second vendor's own model conceded that eight of the nine platforms it ranked published no dataset size at all. None of those figures says which door the answers came through.

Size is not method. A count tells you how many questions were asked. Ask how many of those prompts concern your category at all, because a vast database can still hold only a thin slice of the questions your buyers would type.

The second way is the manual check. John Ozuysal, who founded the agency House of Code, describes it without dressing: take your prompts, "put them to ChatGPT, and then search them one by one," then look at whether the company is recommended and whether it turns up "on the source side." Slow work. Honest about the door, too, because it happens inside the app a buyer uses. But one marketer's session carries that marketer's own history and memory, not the buyer's, and a handful of prompts typed in one sitting is a snapshot of one sitting and nothing more.

Both methods hit the same wall, and Ethan Smith of Graphite named it on Lenny's Podcast in 2025. Ask ChatGPT the same question twice and you get different answers, because the model is effectively drawing a weighted random sample from the answers it could give, and the results shift again by question variant and by surface. His remedy was to ask each question multiple times, and ask its variants, since nothing else shows how often you really appear. In the same conversation he said his company kept a page listing 60 answer-tracking tools in 2025, and that they were probably all pretty similar.

So the useful question is never just how many prompts. Ask also when each answer was captured, because an answer that shifts between two runs on one afternoon will not hold still for a whole quarter. When a vendor quotes a keyword count instead, ask how many of those keywords were ever typed into ChatGPT as full questions.

Measurement in this trade has always been sold by the pound. Nobody weighs what's actually in the sack. Hold every vendor to the same standard, ours included; our AI visibility tracking page is as fair a place as any to start asking. Which leaves you, the buyer, with one question to put across the table, and the job of telling a straight answer from a dressed-up one.

What should you ask an AEO agency before you hire it?

Ask one thing before you sign: do your ChatGPT checks come from the API, from logged-in sessions, or both, and how do you reconcile them? Then listen closely.

The demand is plain enough. Buyers keep asking AI engines which companies specialize in answer engine optimization, and they get back lists of names. Names, never methods. So put the question to the agency yourself, in writing, in words it cannot slide past:

When you check whether ChatGPT names my brand, do those answers come from the API, from logged-in sessions, or both, and how do you reconcile the two?

Then add four follow-ups, because the first answer only tells you which door they used:

1. How many times do you run each prompt, and do you run its variants?
2. Whose account runs the logged-in checks, and what history does it carry?
3. Can I see the raw answer and its sources for any result in my report?
4. When the API and the app disagree, which answer goes in my report?

A credible specialist answers the first question in a sentence. It names the door. It tells you what the door costs. Something like: we run an API panel for volume and trend, we run logged-in checks to see what a real session shows, and where the two split, we report the session and flag the split. A weak answer talks about database size, or a method it can't describe, or turns the question back on you.

The API can belong in a good answer. ClaimMaster, a patent-drafting add-in for Microsoft Word, sends its users' prompts to GPT through what it calls an "Enterprise-level API," and its documentation notes that OpenAI does not use data submitted through the API to train models unless the customer opts in, keeping it for abuse monitoring for a maximum of 30 days. If your buyers read GPT inside tools like that, the API is the door they actually walk through. The point is not to ban it. The point is to make the vendor say which door, and why.

I don't ask other vendors anything I wouldn't answer myself. Alex and I bring a combined 40+ years of SEO and content infrastructure experience to AEO, and our team includes a full-stack engineer and content infrastructure architect with 20 years of building enterprise systems. That matters for one blunt reason: capturing what a logged-in session really shows, at any volume, is infrastructure work, not a dashboard setting. If you want to know which questions your own buyers are putting to AI before you hire anyone, our research into the questions AI answers about your market is a sane place to begin.

Most of this trade would rather sell you a number than tell you where it came from. So bring three questions your own buyers type into ChatGPT to the first meeting, and ask the agency to run them in a logged-in session while you watch. Watch what the specialist does with the fourth follow-up, because that's where the soft ones start to sweat.

What should a buyer demand from the next ChatGPT visibility report?

Demand the raw transcript behind every mention, with its sources. Our own pilot found an admin page describing an authenticator app while the real sign-in screen sent a one-time code by text and email.

The label said one thing. The screen did another. An API-only report is the label: it tells you what the model can say, not what a logged-in buyer with history, memory and search switched on was shown.

HubSpot draws that line too. In 2025, HubSpot's Asia told Marketing Against the Grain that the team tracks Google AI Overviews with a separate tool, because those answers come with no login. The same guest admitted the tracked prompts were the ones "we think our personas are asking," and called it "super imperfect."

If a team that size calls its own prompt list a guess, an API panel is a guess run through a different machine. I expect buyers to stop paying for that before long. In 2025, one developer's API run came back with more than half its links broken, on a job the web app handled reliably. Ask the vendor to flag any mention it found only through the API and never confirmed in a logged-in session. It still gets billed.