Which AI Crawlers Visit Your Website, and Should You Block Them? I Checked 50 Sites

Your website is being read by AI. Some of these visitors collect pages to train models, some index them for AI search, and some fetch a page because a person just asked a chatbot about it. They behave differently, and you can treat them differently.

I wanted to see how established sites handle this, so on 29 October 2026 I read the robots.txt of 50 well-known Indian and global websites and checked how each one treats thirteen AI crawlers.

The crawlers, and what each one is for

The important split is between training crawlers, which take content to build models, and search and user agents, which bring your content to a person who’s asking about it right now and may cite or link to you.

What 50 sites do

I grouped the sites into five sectors of ten. Seven of them, including NDTV, Moneycontrol and Stack Overflow, refused my request for robots.txt itself with a bot-protection error. That’s a choice too, but it means I couldn’t read their rules, so the figures below cover the 43 I could.

And by crawler, across all 43:

What stands out

Global news publishers block almost everything. All ten block at least one AI crawler, and the New York Times, Bloomberg and Reuters block nearly the whole list. Their content is their product, and several are negotiating or litigating over its use.

But they don’t treat every company the same. Every global news site blocks Anthropic’s ClaudeBot, but only half block OpenAI’s GPTBot. Several publishers have licensing agreements with some AI companies, and their robots.txt files reflect who they’ve made deals with.

Indian news sites are far more open. Most only block Meta’s crawler, and three of them (Times of India, Hindustan Times and Mint) publish an llms.txt file to help AI tools read their content.

Developer sites want to be read. None of the readable tech sites block GPTBot or ClaudeBot outright. Six of the ten publish llms.txt. GitHub names the AI crawlers but only limits certain paths. When your audience asks AI for help, being in the answer is the point.

Some sites separate training from search. Reuters, eBay, Medium, Tripadvisor and Airbnb block OpenAI’s training crawler but allow its search crawler. Tripadvisor is the cleanest example: it blocks every training crawler I checked and allows every search and user agent.

How to set it up

On WordPress, edit robots.txt with an SEO plugin, or add rules with the robots_txt filter. In Next.js, app/robots.ts generates the file:

// app/robots.ts
import type { MetadataRoute } from "next";

const TRAINING = ["GPTBot", "ClaudeBot", "Google-Extended", "CCBot", "Applebot-Extended", "Bytespider", "meta-externalagent"];

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: TRAINING, disallow: "/" },
      { userAgent: "*", allow: "/" },
    ],
    sitemap: "https://example.com/sitemap.xml",
  };
}

Three things to know:

• robots.txt is a request, not a lock. Reputable companies follow it, but nothing forces a crawler to. To actually stop one, you need a firewall rule. Cloudflare and some hosts offer a one-click setting for AI bots.
• Name each crawler you mean. A crawler follows the most specific group that names it and ignores the * rules. If you name a crawler to allow something, also repeat any Disallow lines you want it to obey.
• User agents act for people. Blocking ChatGPT-User or Claude-User stops a person’s chatbot from reading your page when they ask about it, and OpenAI says its user agent may not follow robots.txt anyway.

What I’d recommend

This site currently allows every AI crawler. Someone asking an AI assistant for a WordPress or Next.js developer in India is exactly who I want to reach. Whatever you decide, decide it on purpose. The default is that everything is allowed.