Website Technical Checklist: Which AI Crawlers to Allow in robots.txt, and What to Do About Structured Data and llms.txt

For ChatGPT, Perplexity, Claude and Google's AI Overviews to read your website, you have to clear three technical hurdles. Your robots.txt can't block the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot). Your CDN or firewall can't quietly stop them. And your basic business details should be spelled out in structured data. Whether to block the training crawlers, GPTBot and ClaudeBot, is a separate choice, and each company manages it apart from search. Google-Extended is a special case. Besides Gemini training, it also controls whether Gemini Apps can use your content as a reference when they answer. llms.txt is fine to add, but for now it's only a supporting extra.

Every technical claim below links to the official source. We opened each one and checked it on October 16, 2026. Platform rules change, so click through again before you make changes.

AI crawlers come in three kinds, and blocking the wrong one can take you out of AI search

AI companies split their crawlers by job. One kind builds the index for search features. One collects content for model training. And one only opens a page when a user asks a question. OpenAI says it most plainly: the OAI-SearchBot and GPTBot settings are independent. You can allow OAI-SearchBot so your site appears in search results while disallowing GPTBot to signal that your content shouldn't be used for training (OpenAI crawler documentation). The table below follows each company's official docs:

The row people misread most is Google-Extended. Many articles file it under “training,” which is only half right. Google's own wording also covers grounding in Gemini Apps and Vertex AI, which means handing content from the Google Search index to the model at the moment it answers. AI Overviews and AI Mode are part of Search. Google's page on AI features and your website says the control for them is your robots.txt rules for Googlebot. So blocking Google-Extended won't remove you from Google Search or AI Overviews, but it may stop Gemini from using your content when it answers.

The user-triggered agents need a separate look. For ChatGPT-User and Perplexity-User, the companies say robots.txt may not apply or is generally ignored, because the user asked the AI to open the page. Claude-User is different. Anthropic says site owners can use robots.txt to control whether it can access their site.

A robots.txt that allows search and refuses training looks like this

robots.txt lives at the root of your site, for example https://www.example.com/robots.txt, and each subdomain needs its own file. Here's what the lines below do. GPTBot and ClaudeBot can't use your content for training. The Google-Extended line refuses Gemini training, and it may also stop Gemini Apps from using your content as a reference when they answer. Every other crawler, including OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot, falls under the “User-agent: *” group and is allowed. Replace example.com with your domain and copy the lines one by one:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

The first three lines together form one group of rules. Google's robots.txt spec says you can repeat user-agent lines so one group of rules applies to several crawlers (Google's robots.txt specification). If you don't want to block anything, delete the first four lines. If you want Gemini to be able to use your site when it answers, delete only the Google-Extended line. Keep any rules you already have under “User-agent: *”, such as a Disallow for your admin paths.

The most common mistake is adding a separate User-agent: Googlebot group with an Allow, just to be safe. The same spec says a crawler follows only the group with the most specific matching user agent and ignores the rest. Groups for specific user agents are also not combined with the global “*” group. The result: admin paths you blocked under “*” are suddenly wide open to Googlebot. Either don't add that group, or copy your “*” rules into it as well.

After you change the file, each company needs some time to pick up the new version. OpenAI says it can take about 24 hours for its search results to reflect a robots.txt update. Perplexity says changes can take up to 24 hours.

Your CDN or firewall can block AI crawlers before robots.txt even comes into play

Getting robots.txt right isn't enough. On Google's page about AI features and your website, the first basic practice listed is making sure crawling is allowed “in robots.txt, and by any CDN or hosting infrastructure.” OpenAI's help page on ChatGPT search also asks site owners to confirm that their host or CDN allows traffic from OpenAI's published searchbot IP addresses (OpenAI Help Center). Perplexity goes a step further and gives setup steps for Cloudflare WAF and AWS WAF.

If your site runs on Cloudflare, look closely. Cloudflare splits AI traffic into three behaviors, each with its own setting. Search covers crawlers that index your content to answer questions about it later. Agent covers automated activity acting in real time for a person, such as chat fetch bots. Training covers crawlers that take your content to train or fine-tune a model (Cloudflare docs). Two things changed on September 15, 2026 (Cloudflare blog). First, choosing Block or Block on pages with ads for Training now also applies to mixed-use crawlers that do both search and training. Cloudflare names Applebot, Bingbot and Googlebot, so search gets blocked too. To block training and keep search, choose Disallow AI Training instead. Second, new domains are now offered one of two recommended setups, depending on whether the site earns money from ads. With the setup for ad-supported sites, Agent traffic is blocked on pages with ads.

One more detail that's easy to miss. Cloudflare's Bot Preference Sync adds its own managed rules to the front of your existing robots.txt, based on your settings, and serves the combined file to crawlers (Cloudflare docs). So the file on your server isn't always what crawlers get. Always check by opening your live robots.txt URL in a browser.

1. If you use Cloudflare, go to Security Settings and open Configure AI bot policies. Check the setting for each of Search, Agent and Training. Don't set Search to Block.
2. If you want to block training, choose Disallow AI Training, not Block. Otherwise Googlebot and Bingbot get blocked too. Cloudflare also notes that Bing doesn't yet read a no-training preference in robots.txt, so until Bing adds support, this option can't pass that preference on to Bing.
3. Open the crawler list in AI Crawl Control and check that search crawlers like OAI-SearchBot, Claude-SearchBot and PerplexityBot have their action set to Allow.
4. Open your robots.txt in a browser and make sure the section Cloudflare added (marked “BEGIN Cloudflare Managed content”) says what you agreed to.
5. If you don't use Cloudflare, ask your host or developer to search the server logs for requests from OAI-SearchBot, PerplexityBot and Claude-SearchBot, and check whether they're getting refusals like 403.

Structured data helps machines identify you but doesn't guarantee citations

Structured data is information in a standard format, added to your page's code for machines to read. It's usually written in JSON-LD. Two types matter most for AI search. The first is Organization on your homepage. Google says it helps Google understand your organization's administrative details and tell you apart from other organizations in search results (Google Organization documentation). The second is LocalBusiness, which tells Google things like your business hours and departments. Its only required properties are name and address (Google local business documentation).

Be clear about its limits too. Google's official guide to generative AI search says structured data isn't required for generative AI search, and there's no special schema.org markup you need to add. Its value is helping you qualify for rich results (Google's guide to generative AI search). Google's structured data guidelines also say that even correct markup isn't guaranteed to show up (Google's guidelines). Other AI platforms haven't published how they use structured data. So think of it as a name tag that helps machines know who you are, with your name, address and phone number written the same way everywhere. It's not a switch that makes AI recommend you more.

1. Pick the most specific type. Google recommends the closest subtype, such as Restaurant under LocalBusiness for a restaurant, or OnlineStore for an online shop.
2. Put Organization on your homepage or on a single page that describes your company, like your About page. Google says you don't need it on every page.
3. Fill in name, address, telephone and url, and add openingHoursSpecification if you have business hours. Google asks you to include the country code and area code in the phone number. In the US, that means a number starting with +1.
4. Use sameAs to list your official pages on other sites, such as your business pages on social media or review sites.
5. Make sure the markup matches what's visible on the page. Google's guidelines say not to mark up content that readers can't see.
6. Validate it with the Rich Results Test and fix any critical errors. Then use the URL Inspection tool in Search Console to see what Google actually read.

Here's a minimal working sample for web developers to put in the page code. The values below are placeholders for a made-up restaurant in Austin, so swap each one for your own: the business name, street address, city, two-letter country code, phone number with the +1 country code, and your Facebook page. Whatever you put in has to match what the page says:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Restaurant",
  "name": "Your Business Name",
  "url": "https://www.example.com/",
  "telephone": "+1-512-555-0100",
  "address": {
    "@type": "PostalAddress",
    "streetAddress": "123 Example St",
    "addressLocality": "Austin",
    "addressRegion": "TX",
    "addressCountry": "US"
  },
  "sameAs": [
    "https://www.facebook.com/yourpage"
  ]
}
</script>

llms.txt is fine to add, but for now it's only a supporting extra

llms.txt is a proposal Jeremy Howard published in September 2024. The idea is to put a Markdown file at the root of your website, with short descriptions and links that help AI agents quickly find the important information on your site (llmstxt.org). The format is simple. The first line is an H1 with the name of the site or project, and that's the only required part. Next comes a blockquote with a short summary, then H2 sections listing links to your important pages. The proposal also says it serves a different purpose from robots.txt. robots.txt tells automated tools what they're allowed to access. llms.txt is read on demand, when an AI agent needs information while helping a user.

But as of October 16, 2026, we haven't found any public commitment from OpenAI, Anthropic or Perplexity to read site owners' llms.txt files when they pick search results or sources to cite. Google says it plainly in its official guide: Google Search itself doesn't use LLMS.txt or similar files, they neither help nor hurt your visibility in Google Search, and it's completely fine to create them for other services. So when you set priorities, put llms.txt after robots.txt, your CDN and structured data. It's cheap, so add it if you have time, but don't expect it to change who AI recommends.

After every change, use official tools to confirm the crawlers can really get in

Each item above has an official tool you can check it with. Google Search Console's Generative AI performance report has been available to all websites since August 31, 2026. It shows which of your URLs appeared in AI Overviews and AI Mode (Google's announcement). The AI Performance report in Bing Webmaster Tools shows how often your pages were cited in Copilot and in AI-generated summaries in Bing (Bing's announcement).

There's also a switch that rolled out worldwide on the same date, August 31, 2026: the Search generative AI control in Search Console settings. The default is to include your site. If someone changed it to exclude, your site won't appear in AI Overviews, AI Mode or generative AI features in Discover (Search Console Help). If you take over a site someone else built, always check this setting.

Common questions

Once these technical checks are done, the next step is to see how AI actually answers the questions your customers ask. You can start with our free quick scan, which looks at one platform, ChatGPT. It needs no sign-up, takes about 1 to 2 minutes, checks whether AI can read your website and runs 2 live ChatGPT questions.