How robots.txt works for AI crawlersHow robots.txt works for AI crawlers

A rising number of webmasters are asking a version of the same question: with AI crawlers now reading the web to build search indexes and train models, does my old robots.txt file still do what I think it does — and does it even cover the new crawlers showing up in my logs? The honest answer is: mostly yes, with some details worth getting right.

What robots.txt actually is

robots.txt is a plain text file, sitting at the root of your domain (yoursite.com/robots.txt), that states which parts of your site automated crawlers are and aren't welcome to read. It's not a technical lock — nothing forces a crawler to obey it — it's a convention, honoured by every crawler that wants to be considered well-behaved. That convention has held up remarkably well for decades precisely because ignoring it is treated as a serious breach of etiquette, and reputable crawlers, AI ones included, are built to check it.

A typical entry looks like this:

User-agent: askFinz-Crawler
Disallow: /admin/
Crawl-delay: 2

User-agent names which crawler the rule applies to (or * for all of them). Disallow marks paths that crawler shouldn't read. Crawl-delay sets a minimum number of seconds between requests from that crawler — a floor, not a target; a well-behaved crawler will only ever wait longer than that, never shorter.

What's different with AI crawlers, and what isn't

The mechanism hasn't changed. What has changed is the number of distinct crawlers now operating, and the reasons they're reading your site. A single organisation's crawling activity might actually be several differently named crawlers, each doing a different job — general reading, news-specific reading, research-focused reading — and a robots.txt file written years ago, before those existed, may only name the one or two crawlers a webmaster knew about at the time. Naming four crawlers explicitly while ten are actually active means you're only controlling access for the four you thought to list — the rest fall through to whatever your wildcard (*) rule says, which may not be what you intended either way.

How to actually check what's crawling you

Rather than guessing, check your server logs for the User-Agent string on incoming requests. Every legitimate crawler identifies itself by name in that string, and a reputable one publishes documentation stating exactly what that string looks like, what it does, and how to control it — askFinz-Crawler's documentation lists the current identity family this way specifically so a webmaster checking their own logs against it isn't left guessing which of several similarly named crawlers is actually which. If you see a name in your logs that isn't covered by any rule in your file, it's currently operating under your wildcard rule by default, whatever that says.

Getting the rules right

A few practical points worth checking on your own file:

• Name crawlers explicitly if you want different rules for different ones. A general wildcard rule is a reasonable default, but if you want to allow general reading while blocking something more specific, that needs its own named entry.
• Set Crawl-delay if your server is sensitive to load. A well-behaved crawler treats it as a strict minimum, so it's a genuinely effective lever, not just a suggestion. That said, a properly built crawler already paces and globally rate-limits its own requests regardless of what your file specifies, so a missing Crawl-delay isn't an invitation to be hammered — it just means you haven't set your own floor.
• Disallow doesn't mean "invisible," it means "not read." Blocking a path stops a well-behaved crawler from reading it; it doesn't remove references to that URL that might exist elsewhere on the open web.
• Revisit the file periodically. New crawlers appear. A file that hasn't been touched in years is very likely missing entries for crawlers that didn't exist when it was written.

What being crawled well actually gets you

Blocking indiscriminately has a real cost of its own: a site that's overly cautious about crawler access can end up simply absent from AI search and AI-generated answers, even though it would happily be found through ordinary search. If you want the opposite — reachable and considered, rather than either scraped carelessly or shut out entirely — robots.txt plus a direct domain submission gives you both the control and the visibility, on your terms.

Further reading

• The full crawler identity family and how to allow or block each one: askFinz-Crawler documentation
• What a well-behaved crawler is supposed to do beyond just reading your file: What is a web crawler?
• Put your site directly in the queue instead of waiting to be found: How to get your site indexed by AI