Getting cited by ChatGPT, Perplexity and Google's AI Overviews: what they actually read, and what they will never see

Why many modern websites are invisible to AI crawlers, which bots to let in through robots.txt, how to write a page a model is happy to cite — and which parts of all this are still a gamble.

by Nadia Maar · 19 September 2026

A growing share of searches ends without a click: the answer comes from ChatGPT, from Perplexity, or from the overview Google places above the results. If you sell services, the question isn't whether this changes SEO — it does — but whether that answer contains your name, with a link, or a competitor's.

Here is what is known for certain about how these systems read websites, what you can do right away, and what is still a hypothesis. The distinction matters: this topic attracts plenty of promises and very few measurements.

1 · The problem you can't see from the browser

Many modern sites are built as applications: the server sends an almost empty page, and JavaScript draws the content in the browser. For a visitor nothing changes. For a crawler everything does, because it depends on who runs that JavaScript and who doesn't.

Googlebot runs it: it renders the page much like a browser, with some delay. AI crawlers, for the most part, don't. An analysis published by Vercel and MERJ in late 2024, based on real crawler traffic, found that GPTBot, ClaudeBot and PerplexityBot sometimes download JavaScript files but don't execute them: they read the HTML the server sends, and that's it.

The practical consequence

If your site is a client-rendered application, to these crawlers your pages are a title and little else. You can have the best article in your field and never be cited, simply because the text never reaches them.

2 · Who to let in

The second filter is robots.txt. Many sites have blocked AI crawlers out of caution, or found them blocked by a hosting provider's setting. That's a legitimate choice, but make it knowing what you block: some of these crawlers collect data to train models, others answer users in real time, with a link to the source.

robots.txt

# Bots that answer users and cite the source.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
Disallow: /private-area/

# Bots that collect for training: your call.
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /private-area/

# Everyone else.
User-agent: *
Disallow: /private-area/

The detail almost everyone misses: a crawler follows only the most specific group that names it, and ignores the rules written under the asterisk. That's why every group repeats the restrictions — otherwise, by naming a crawler to let it in, you would also open your private area to it.

3 · A page a model is happy to cite

Once the text gets through, the question is what makes it citable. There are fewer certainties here, but some patterns fit how these systems work: they look for a passage that answers the question, and quote it with its source.

• Answers that stand on their own. A paragraph that opens with the answer can be extracted; one that reaches it after three lines of context, much less so.
• Explicit questions and answers. An FAQ section is already in the shape of what a model has to produce, and often gets quoted almost verbatim.
• Tables and comparisons. A comparison in a table is structure that can be read without interpretation.
• Numbers with their source, and dates. A verifiable claim beats an adjective, for a reader and for a system choosing whom to cite.
• A named author. Who wrote it, with what expertise, and a link to who they are: on the page, not just in the metadata.

Then there's structured data: the markup that tells an engine what a page is — an article, a service, a business with a name and an address. It doesn't guarantee a citation, but it removes ambiguity about who you are, which is the first thing a system has to settle before it names you.

4 · What is still a gamble

This topic is full of certainties sold as such. Some things, though, simply aren't known today:

• The llms.txt file. It's a 2024 proposal to give models a readable index of a site. It takes half an hour and does no harm, but none of the major systems has confirmed using it: treat it as cheap insurance, not a strategy.
• Measurement. There is no Search Console for generated answers: the tools that count how often you're cited do so by querying models with sample questions. They give indications, not data.
• Stability. These systems change every few months. What lasts rests on principles that don't depend on the model of the moment: text readable without JavaScript, clear answers, a declared identity.

A rule for deciding whom to listen to

Be wary of anyone promising to "rank you on ChatGPT" with a guaranteed result. Nobody controls those answers, not even the people who build the systems. You can make your site the easiest source to read and cite; the rest isn't up to any consultant.

5 · Where to start

• Check what a crawler sees: request one of your key pages with GPTBot's user agent — curl from a terminal is enough — and look for the text in the response. If it isn't there, that's your first job.
• Check robots.txt and your hosting provider's firewall: decide who gets in, instead of finding out.
• Pick the five pages that sell and rewrite each opening so it answers straight away the question that brings people there.
• Add genuine FAQs, and structured data consistent with what the page shows.
• Only then think about llms.txt and measurement tools.

This site is built that way: service pages and articles reach crawlers as complete HTML even without JavaScript, with a structured-data graph and an llms.txt file. To see it for yourself, request this page the way a crawler does and read what comes back.

Your site, as a crawler sees it

The question isn't whether AI will cite you, but whether it could today: whether the text gets through, whether the crawlers are let in, whether the pages answer. Those are three technical checks, and they don't take long.

The free SEO analysis starts right there: what a crawler sees of your pages, compared with your competitors, and in what order to fix things.

The service behind this article

Technical SEO and AI search optimization

If this was useful, carry on from here

• A multilingual site for selling abroad: hreflang, URL architecture and the mistakes that cost rankings — Translating isn't enough: you have to tell search engines which version is for whom. The four possible architectures, how languages are declared, what to translate beyond the words, and how to check it works.
• The architecture decision a startup founder makes in week one, and rewrites in month eight — Why a client-only React app is the fastest thing to build and the most expensive thing to fix, how Next.js's App Router separates what must be findable from what only needs to be usable, and the code at the boundary between the two.

FAQ

Do ChatGPT and Perplexity read JavaScript websites?

Mostly not. Their crawlers read the HTML the server sends and don't execute JavaScript: if the content is drawn in the browser, the page is almost empty to them. Googlebot, on the other hand, does run JavaScript.

Should I block AI crawlers in robots.txt?

It depends on what you want. Some collect data to train models, others answer users with a link to the source. Blocking the latter means giving up on being cited.

Does an llms.txt file actually help?

It's a 2024 proposal that none of the major systems has confirmed using. It's cheap and harmless, but it doesn't replace readable text, clear answers and structured data.

Can anyone guarantee you'll be cited by ChatGPT?

No, and be wary of anyone who promises it. You can make your site the easiest source to read and cite; what a model chooses to quote isn't controlled by any consultant.

Nadia Maar