Your website may have been invisible to ChatGPT, Claude and Perplexity for months without you realising it. Google Search Console won't tell you, because Search Console only measures Google. Your analytics won't tell you either, because when an LLM recommends a competitor instead of you there's no click to track. You only notice when leads come in slower than usual, when a prospect tells you «before calling you I asked ChatGPT for companies like yours», or when a business owner checks their own brand on an LLM and finds that the models simply don't know the company exists.
The most common cause in 2026 isn't your SEO, your robots.txt, or a forgotten noindex. It's a line that Cloudflare has been injecting on top of your configuration —silently, by default, for months— on hundreds of thousands of sites. A line you never wrote. A line that tells GPTBot, ClaudeBot and Google-Extended: don't come in.
This article explains exactly what's happening, how to check it yourself in 30 seconds with three commands, which decision makes sense depending on your type of business, and how to find out afterwards whether LLMs actually cite you when someone asks about your industry.
The toggle that locks you out of ChatGPT without your knowing
Cloudflare has been activating by default, on millions of accounts, a switch called «AI Scraper & Crawler protection» (you may also see it as «Block AI Bots» or «AI Labyrinth» depending on your dashboard version). The stated purpose is to protect content owners from AI scraping at scale. The actual effect on many small and mid-size websites is different: Cloudflare injects a block into your robots.txt that blocks GPTBot, ClaudeBot, Google-Extended and friends, regardless of what you've written in the file.
If you open your own robots.txt from the browser you see what you uploaded. If you request it from the actual server, you discover there's another robots.txt sitting on top of yours. One that Cloudflare serves before the original even has a chance.
# what the owner wrote User-agent: * Allow: / # what Cloudflare injects above it # BEGIN Cloudflare Managed content User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: PerplexityBot Disallow: / # END Cloudflare Managed content
For GPTBot that line is the equivalent of a «do not enter» sign. And GPTBot respects it. So does Claude. And Perplexity. And Google-Extended (the crawler feeding AI Overviews). Your site disappears from the pool of answers these models give to the people searching for what you offer.
The issue isn't Cloudflare's decision —there are legitimate reasons to protect sites from commercial scraping—. The issue is that the decision is on by default, silent, and overrides what you have declared at the root of your own site. Many owners have been invisible to LLMs for months without realising.
How to check in 30 seconds whether it's happening to you
The test is simple and requires no installation. Three commands you can run from the terminal on your own computer (Mac, Linux or Windows with WSL). Replace yourdomain.com with yours:
# 1. Request the robots.txt as GPTBot would see it curl -A "GPTBot" https://yourdomain.com/robots.txt # 2. Request the homepage as ClaudeBot would see it curl -I -A "ClaudeBot" https://yourdomain.com/ # 3. Check whether Cloudflare has injected managed content curl https://yourdomain.com/robots.txt | grep -i "Cloudflare Managed"
If step 1 returns Disallow: / under GPTBot or ClaudeBot, your site is not being indexed either for training or for conversational search by those models. If step 2 returns a 403 or a 429 when you identify as an AI bot, the AI Labyrinth is active. If step 3 shows the «Cloudflare Managed» block, the toggle is on.
Not comfortable with the terminal? Any online tool that lets you simulate a User-Agent will do (httpie.io, webhook.site, or any decent HTTP client). The principle is the same: request your own robots.txt identifying as GPTBot and look at what the actual server returns, not what your browser shows you.
What to do if the toggle is on
Deciding whether to leave it on or turn it off isn't a purely technical decision. It depends on where your business lives. This is where you actually have to think, not apply dogma.
If your case is the first one —most of the SMBs we work with—, what you need to do is turn the toggle off:
- Log into your Cloudflare dashboard and select the domain.
- Go to Security → Settings (or Scrape Shield on older dashboards).
- Look for «AI Scraper & Crawler protection», «Block AI Bots» or similar.
- Set it to Off, or adjust the rule to the specific bots you actually want to block.
- Re-run the three commands above and verify that your
robots.txtno longer injectsDisallow: /for GPTBot and ClaudeBot.
The change is reversible at any time and propagates within minutes. But do it with awareness: if your business model depends on not being scraped, that's a different conversation.
Beyond Cloudflare: checking whether LLMs actually know you
Removing the block is only the first step. The fact that GPTBot can enter your site doesn't mean it's reading you, nor that it cites you when someone asks a question. The next diagnosis is more revealing: ask the models directly about your brand and your industry, and measure what they return.
The exercise is simple to describe and eye-opening to do. Take the 10 or 15 queries an ideal customer of yours would run before buying (not classic SEO keywords —real human questions). Run them in ChatGPT, Claude, Perplexity and Gemini. Record whether they mention you, recommend you, cite the source, and benchmark yourself against the three competitors that do appear.
The results tend to be uncomfortable. Many brands that think they're well positioned discover that the LLMs never mention them, not even for very industry-specific questions. And that instead the models recommend competitors who may rank worse than them on Google but have more external citations, better entity structure, or simply let AI bots read their site.
This systematic diagnosis —repeated every few months, with reproducible prompts, frequency metrics and share of voice— is what we call an AI visibility audit or GEO diagnosis (Generative Engine Optimization). It's a new discipline that complements classic SEO rather than replacing it. At Gecko Studio we already run it as part of every audit, and we're setting up continuous monitoring with tools such as Ahrefs Brand Radar for clients who want monthly tracking of their share of voice inside the models.
What happens if you do nothing
The difference with classic SEO is that the pie of AI answers is much smaller. A Google SERP shows ten results, plus an AI Overview block, plus the local pack, plus the sidebar. A ChatGPT answer to a commercial question typically cites between three and five sources. Seven at most. That number doesn't expand or reshuffle with each search the way Google does: it tends to consolidate around the same handful of brands week after week once the model has formed a preference.
Translated to your business: if the competitors who are already inside the LLMs' index consolidate their citations this quarter, regaining presence later costs a lot more than getting in now while there's still room. It's not a linear ladder like Google, where moving from position 12 to 8 is a matter of content and backlinks. When a model has decided that the five relevant names in your industry are someone else, earning a sixth seat at that table means months of building public entity signals.
If you want the full map of how to manage AI crawling on your site —which bots to allow, which ones to block selectively and how to monitor changes— we covered it in detail in the article Your website and AI bots: what they crawl, what they ignore, and what you should be controlling right now.
How to know whether your website is visible to ChatGPT today
Five checks you can run this afternoon with no paid tools:
- Check your real
robots.txtwith an AI bot User-Agent (thecurlcommand above). If you seeDisallow: /under GPTBot or ClaudeBot, the Cloudflare toggle is on. - Decide consciously which bots to allow. Most commercial SMBs want to allow them. That's not an inherited default, it's a business decision.
- Ask ChatGPT, Claude and Perplexity about the ten most relevant queries in your industry. If they don't mention you, you have an AEO gap even when Cloudflare is clean.
- Review your content structure. LLMs cite clear paragraphs, lists and short definitions. If your homepage is just a slider of generic claims, they won't cite you even if they read you.
- Build external citations. There's no AI visibility without mentions on industry reference sites. The entity graph matters as much as your own domain.
If after running the three curl commands you find that Cloudflare is blocking you and you'd like us to confirm afterwards that the fix is in place, get in touch through the contact page and send us the URL. That check is free because we have it automated and it takes minutes.
If you want to go further and actually measure how LLMs cite you compared to your competitors —with prompts tailored to your industry, count of mentions and citations per model, share of voice comparison and a 90-day action plan to gain AI presence—, that work is done to measure inside our GEO audit. Tell us where you stand and we'll scope it and price it for you.
