ChatGPT API vs UI: do they return the same citations?

Across 555 prompts run 47 times on each surface, average brand visibility differed by 41% between the ChatGPT interface and the API Landwehr · Peec AI · 2026C. The most visible brand in the interface did not reach the API top three. The figure comes from Peec AI's Malte Landwehr. It is a single vendor study with no disclosed configuration; treat it as provisional.

The divergence itself is not in doubt, and the sample is an unusually large one for this question. What the study cannot establish is how much of the gap belongs to the surface and how much to the settings. That distinction decides whether the number means anything to a brand team.

The 41% gap is real, and the sampling behind it is unusually strong

The design runs 555 prompts 47 times on each surface, which is 52,170 chats Landwehr · Peec AI · 2026C. That is well past the point where sampling noise explains a gap this size. Reliable citation-share estimates need roughly 40 to 50 queries on Gemini and 150 or more on SearchGPT Sielinski · IQRush · 2026A. Typical 95% confidence intervals in that work spanned 3 to 6 percentage points.

The reported differences are large rather than marginal. Competitor domains rose from 38% to 52% of citations between the two surfaces Landwehr · Peec AI · 2026C. Listicles fell from 25% to 4%, and product pages and how-to guides grew from 34% to 63%. Movements of that size do not come from a sampling artefact at 47 repeats.

Forty-seven runs per prompt already answers the instability objection

The most common objection to the study is that AI answers are unstable against themselves. As a general matter the objection is well founded. Within-prompt resampling explained 34.8% of outcome variance across 12,933 responses, while brand identity explained just 0.7% Żatuchin · Estonian Entrepreneurship University of Applied Sciences (EUAS); Rankfor.AI · 2026A. A single run on either surface is one draw.

The objection does not land on this particular design. The same decomposition found that repeats past the fifth added almost nothing Żatuchin · Estonian Entrepreneurship University of Applied Sciences (EUAS); Rankfor.AI · 2026A. A separate framework recommends a minimum of 5 to 10 repeated runs per evaluation Mustahsan · arXiv · 2025A. At 47 runs per prompt per surface, the study clears both thresholds several times over. Repetition is the one thing it did thoroughly, and the argument around it has largely gone unmade.

Repetition removes random error, not a setting

A setting confound is systematic, so averaging more runs does not remove it. Suppose the two surfaces ran different model variants, or one had web search enabled and the other did not. Every one of the 47 repeats then inherits that difference identically. The study discloses no temperature, no web-search setting, no interface account state and no model variant Landwehr · Peec AI · 2026C.

This is the distinction the surrounding argument keeps collapsing. Run-to-run variance is a sampling problem that repetition fixes. Configuration divergence is an identification problem that repetition cannot touch. Treating generative engine optimisation as a stochastic, partially observable pipeline captures the first problem and not the second Martinez · arXiv (cs.IR) · 2026A. A partially observable system still has to be observed under stated settings.

GPT-5.6 is three models, so "both are GPT-5.6" does not name the comparison

The study states that both surfaces run GPT-5.6. OpenAI documents three models under that name, with separate API identifiers and three price tiers OpenAI · 2026A. Sol runs at five and thirty dollars per million input and output tokens, Terra at 2.50 and fifteen, and Luna at one and six. Sol is the flagship, Terra a lower-cost option, and Luna the fastest and most cost-efficient.

Which one the interface serves depends on the account. GPT-5.6 Luna became the default model for Free and Go users OpenAI · 2026A. OpenAI states separately that the Chat build of Sol is distinct from the build powering Work and Codex. A comparison that does not name the variant on each side may be measuring model against model in part. Naming the variant is a documented requirement rather than a pedantic one.

A web-search setting can manufacture a citation collapse

Several of the study's reported differences are citation-composition findings Landwehr · Peec AI · 2026C. Those are vulnerable to a single unstated parameter. An ungrounded API call retrieves nothing, so it returns no citations at all. A near-total fall in user-generated and editorial sources on the API side is consistent with a genuine retrieval difference. It is equally consistent with web search being switched off on one side.

The study does not say which. Rankfor.AI's Dmitrij Żatuchin reported a small check in the comment thread, running six prompts six times on gpt-5.6-terra with and without the web-search tool. That is an unverified test by a competitor of the study's author, so it belongs in the question rather than the answer. The question stands on its own. A citation-composition finding is interpretable only once the grounding configuration is stated for both arms.

No published API fan-out count should anchor a decision yet

The study reports a much lower fan-out ceiling in the API than in the interface Landwehr · Peec AI · 2026C. A separate analysis ran about 4,000 prompts through the GPT-5.6 Sol API Long · Nectiv · 2026C. It reports an average of 7.61 fan-out queries per prompt, with the longest single chain reaching 29 searches. That study is a single vendor analysis; treat its figures as provisional. The two accounts of API fan-out cannot both describe the same instrument, and neither is reconciled.

OpenAI's own documentation supplies the likely reason. The web_search_call output item will usually, but not always, include the search queries that were searched OpenAI · 2025A. Fan-out observed through the API is therefore partial by the platform's own account. An API-side count may be a floor rather than a full count.

On that reading, a low API fan-out figure measures what the API reveals rather than what the model did. The disagreement is a counting problem before it is a behavioural finding.

The model names brands before it searches, so neither surface isolates retrieval

The argument treats the API as the model and the interface as the model plus retrieval. Network-traffic analysis complicates that split. In 21 of 27 conversations, the model's first search query already contained brand names the user never typed Mohanadasan · Snippet Digital; Keyword Insights · 2026C. No page had been fetched at that point, and 11 of 13 unrelated product categories showed the same pattern. That analysis is a single practitioner study; treat it as provisional.

What the model already holds therefore shapes what it goes looking for. Brands named in ChatGPT's own query reached the final answer 68.9% of the time, against 2.1% for brands fetched but never named Mohanadasan · Snippet Digital; Keyword Insights · 2026C. The author states those percentages are directional rather than measurements. He narrowed the headline in August 2026, after further testing found unnamed brands can still reach the answer. The direction still holds, and a two-surface comparison has more than two variables in it.

Neither surface is ground truth, and the divergence is engine-wide

Interface results are not a stable reference point either. ChatGPT's switch to GPT-5.3 Instant on 4 March 2026 cut cited URLs per response by about 20% de Segonzac · Resoneo · 2026B. That study tracked 27,000 responses to 400 prompts over 14 weeks, and unique domains per response fell from roughly 19.6 to 15.5. Those figures belong to that switch and do not transfer to later models. The surface a tracker calls real changes underneath the tracker.

Cross-engine divergence sets the outer bound on what any single-surface number describes. Of 11,647 domains cited across five engines, 69.6% were cited by only one engine and 2.7% by all five Khallad · SurfacedBy · 2026B. A critical survey of 45 GEO studies reaches the general form of the same conclusion Martinez · arXiv (cs.IR) · 2026A. It describes generative engine optimisation as a stochastic, partially observable pipeline rather than one ranking task.

What a defensible AI visibility number carries

The practical output of this argument is a disclosure list, not a verdict on either surface. A visibility figure becomes comparable when it states five things. Access path, model variant, grounding configuration, run count, and spread. Without those, two numbers that look like the same metric may describe different systems.

Single-snapshot measurement understates true brand presence, and stochasticity is a measurement parameter rather than noise to discard Schulte et al. · University of St. Gallen · 2026A. The addition this episode makes is narrow. The access path belongs in the same disclosure block as the run count. The study that started the argument is large enough to show the gap exists, and incomplete enough to show which fields the category leaves out.

Related reading:

• You can't measure AI search with one number for the four instruments and their blind spots.
• What we can actually know about AI chatbot prompts for the epistemics of prompt data.
• How to read the evidence in AI visibility for the source-tier system that governs the Tier C caveats above.