robots.txt for AI Crawlers: We Counted Our Log Before Meta Starts on 20 November

On 30 October, Google and Microsoft announced on the same day that they will support a shared robots.txt signal against AI training. On 20 November, Meta's new business terms take effect, and they allow Meta to crawl websites tied to a business account. Our robots.txt names ten AI user agents and none of them is Meta's. Before changing anything, we wanted to know who actually visits us.

This post follows up GEO: Why AI Search Engines Overlooked Us. That one was about letting crawlers in. This one is about which ones you want to let in.

In short

• The new signal is called __usage_ai-training. It is an IETF draft by Kevin Kelley (Anthropic) and no provider honours it yet. Google says 2027, Microsoft says 1 February 2027.
• Meta matters from 20 November, but only if your site belongs to a Meta business account. Whether that applies to us is open.
• In our log (16 October to 5 November 2026), 50,365 requests carry one of the user agents we searched for, and 18,108 of them (36 percent) come from scanner addresses. Anyone counting unfiltered overestimates AI crawlers by more than a third.
• In those 20 days, Meta's addresses did not fetch robots.txt once.

What an AI crawler is

An AI crawler is a program that fetches web pages so its operator can use the content for model training, a search index or the answer to a single user question. The purpose is given in the user agent and in the operator's documentation. robots.txt can allow or block each user agent. Whether the operator complies is stated only in its own documentation.

What the providers say themselves

The purposes are separate, and that is what makes a decision possible:

Sources read on 5 November 2026, see the links below. Note that meta-externalagent names two purposes. Blocking it excludes training and also the direct indexing for products.

The shared signal

The IETF draft draft-kelley-aipref-training-preference-signal-00 is an individual draft dated 30 October. It expires on 3 May 2027 and has no formal status. The token is written as a User-Agent line in robots.txt. A disallowed path means: do not use for training.

Google writes that a site using the token without naming Google-Extended will be treated as if it had disallowed Google-Extended. Support is planned for 2027. Microsoft names 1 February 2027, 15 December 2026 for transition settings in Bing Webmaster Tools, and 1 March 2027, after which NoArchive and NoCache no longer count as a training or grounding preference. For grounding, Bing has no site-wide setting. It is set per page with a meta tag or header. Anthropic's help page says that if robots.txt names ClaudeBot, that rule takes priority over a general no-AI-training signal for the affected paths.

For us this means the token does not replace the individual rules. It comes on top of them later. Today it changes nothing.

What our Caddy log shows

The Caddy edge on .181.1 writes JSON logs for the host senn-tech.com. Static files and our monitoring addresses are excluded. The oldest surviving files start on 16 October 2026 at 01:26, which makes the window 20 days. The analysis was read-only. There are 318,289 requests in total, of which 50,365 carry one of the 23 user agents we searched for. 18,108 of those come from 70 addresses that use at least four different AI user agents in the period.

Those 70 addresses request paths such as /prisma/.env and /@fs/home/ubuntu/.aws/credentials. That is a scanner posing as a bot. We count it separately. The four-user-agent threshold is our own rule of thumb.

Our own measurement, Caddy on .181.1, 16 October to 5 November 2026, unit: requests to senn-tech.com. Each request counts once; when a user agent string contains several names, the first match wins. Google-Extended does not appear because Google says it has no user agent of its own.

What stands out about Meta

Of the 796 remaining meta-externalagent requests, 606 came on 22 October and 180 on 29 October. The rest are scattered hits. Nearly all addresses sit in 57.141.x.x. 203 requests went to /_next/image, the rest to pages such as /de/blog and /de/leistungen.

From that address range, there was not one fetch of /robots.txt in 20 days, under any user agent. Meta states a cache of up to 24 hours, which does not explain 20 days without a fetch. We do not know why. We also did not check that the range belongs to Meta against a list from Meta.

Meta's documentation says facebookexternalhit (109 requests here, 34 of them for robots.txt) and meta-externalfetcher may bypass robots.txt. A block is therefore at best half a statement.

Reading robots.ts turned up a second point. Every AI user agent has its own group with Allow: /. Under RFC 9309 a crawler follows only the group for its own token, so the * group with Disallow: /admin does not apply to it. In the log, no genuine AI user agent requested /admin, so the gap is theoretical.

How we approach the decision

Three questions, in this order:

1. Does the site belong to a Meta business account? If not, the clause does not reach your site, though robots.txt entries still make sense.
2. Do you want to appear in Meta AI answers? That decides meta-webindexer.
3. Do you want your content to flow into training? That decides meta-externalagent, bearing in mind that it also indexes for products.

Our decision is still open, and robots.ts is unchanged as of 5 November 2026. Meta itself says webmasters may indicate their preferred crawling behaviour through robots.txt. According to Search Engine Journal, neither preview of the terms says whether a robots.txt setting limits what a business grants by accepting them. A block is a technical signal and has no legal force of its own.

Count the other side too. Blocking OAI-SearchBot removes you from ChatGPT search. For us that is the largest item at 8,548 requests. Whether any of it turns into visitors, we have not measured. If you need this weighed for your own site, our offer is GEO and SEO review, and the hub Making sense of AI collects further context.

Still to measure

Three steps we have not done yet:

• Check addresses against vendor lists. Done for Anthropic: of 1,324 requests with a ClaudeBot or Claude-User user agent, 1,110 fall inside the list at claude.com/crawling/bots.json. Most of the other 214 come from 160.79.104.0/21, which ARIN assigns to Anthropic but which is not in the list. Still open for Meta, OpenAI and Perplexity. Method: download the provider's address list and check every source address of that user agent against it.
• Clarify the Meta account. Method: look in Meta Business Manager to see whether senn-tech.com is attached to an account as a domain or link. Result open; someone with access to the account has to look.
• Effect of a change. Method: after changing robots.txt, run the same count for 14 days and use the 22 October peak (606 requests in one day) as a comparison.

What is documented and what is open

Documented, each read from the primary source: the content of the IETF draft, the announcements from Google and Microsoft, and the crawler purposes at OpenAI, Anthropic and Meta. Documented by our own measurement: the numbers in the table for 16 October to 5 November 2026.

Open: when OpenAI supports the signal (the OpenAI announcement returned a 403 for us, so we give no date), whether Meta adopts it, whether the clause affects our site, and why Meta's addresses do not fetch robots.txt. The 20 November date and the clause itself come from press reports, because Meta's terms page would not open for us. This is not legal advice.

Sources last checked: 2026-11-05.

Your 10-minute check

1. Run curl -s https://your-domain/robots.txt and look for meta-. If nothing is there, everything is allowed today.
2. On the reverse proxy, count the log by user agent, for example with zcat -f access*.log* | grep -c meta-externalagent.
3. Group the same user agent by source address. A source that uses four or more different AI names is not a crawler.
4. Check whether the domain is attached to a Meta business account.
5. Decide what training and search are worth to you, and only then touch robots.txt.

Further Reading

• IETF: Machine-Readable Training Preference Signal
• Google Search Central: AI training control
• Bing: Clearer preferences for training and grounding
• Google: Common crawlers, Google-Extended
• OpenAI: Crawler documentation
• Anthropic: Crawling and blocking
• Meta: Web Crawlers
• RFC 9309: Robots Exclusion Protocol
• Search Engine Journal: Meta delays business terms