90 of 100 Privacy Companies Have Never Named an AI Crawler in robots.txt (August 2026)

A site's robots.txt is the one place where a company states, in public and in writing, what it wants automated clients to do with its pages. So for an industry whose entire product is control over data, it is a fair question: has it said anything about AI crawlers on its own websites?

We read the robots.txt of 106 VPN providers, password managers, encrypted messengers, self-hosted tools and privacy-focused hosts on 17 August 2026 - the same list of domains we used for the HTTP security headers ranking.

robots.txt asks. It does not prevent.

Before any number means anything, this has to be said plainly: robots.txt is a convention. It is a text file that requests certain clients stay away, and a crawler that ignores it meets no obstacle at all - the pages are served to whoever asks for them. Compliant operators honour it; that is the whole mechanism.

The gate in the photograph is about waist high and the deck carries on behind it. That is the accurate mental model: the sign works on anyone who reads signs.

So a BLOCK in the table below does not mean a crawler was stopped. It means the operator wrote down what they wanted. That is exactly what makes the silence interesting - saying nothing costs nothing, and 90 sites still said nothing.

The method, in full

One unauthenticated GET of /robots.txt per domain, following the robots specification for group selection:

1. a group whose User-agent: names the crawler wins over the wildcard * group;
2. inside the winning group, Disallow: / is read as a block; anything else is read as access;
3. no applicable group at all is recorded as -.

Four values appear in the data, and the difference between the first pair and the second is the entire point:

Our first version of the scanner collapsed those two pairs, and reported that Proton, Mullvad, DuckDuckGo and Nextcloud all allowed every AI crawler. They do not: they have simply never written the word GPTBot. Reading inherited permission as consent would have produced the exact opposite headline from the truth, so the distinction is preserved in every row.

The ten sites that took a position

posteo.de is the only site in 106 that addressed all eight agents, and it blocks every one of them.

Two things in the data worth noticing

Nobody welcomes Apple. Applebot-Extended - Apple's opt-out token for training - is named by three sites and blocked by all three. It is the only one of the eight with zero explicit allows. Whether that reflects a judgement about Apple or simply that the operators who bother to name it are the ones who block everything, this dataset cannot say.

Password managers split down the middle on the same question. Five of the ten deciders are password managers, and they went opposite ways: Bitwarden, LastPass and Dashlane name AI crawlers and let them in; KeePassXC and Standard Notes name them and shut them out. Same product category, same year, opposite conclusions. We are not going to invent a reason for that - we are recording that the disagreement exists inside one category, which is more informative than an industry average.

Download the raw data

privacy-industry-ai-crawlers-2026.csv - one row per site, one column per agent, 106 rows.

• 100 measured, 6 with empty fields because the request failed. A failed measurement is not a finding.
• A homepage_redirects_to column flags the six domains whose homepage redirects elsewhere (openziti.io→netfoundry.io, vaultwarden.dev→vaultwarden.com, threema.ch→threema.com, tutanota.com→tuta.com, www.veracrypt.fr→veracrypt.io, uptimekuma.org→uptimekuma.co). Note that /robots.txt and the homepage do not always behave the same way: openziti.io answers on its own /robots.txt (it has none) while redirecting its homepage to netfoundry.io, which does have one. Both rows are real measurements of different things, and that is why they differ.
• Released into the public domain under PDDL 1.0. Reuse it, republish it, correct it. If you cite it, cite the date - robots.txt changes the day someone edits it.
• The scanner is MIT-licensed and dependency-free. Point it at your own list and you will get the same kind of table.

We would rather be corrected than believed. If your site is in the data and the row is wrong - or if you have changed your robots.txt since 17 August - tell us and we will re-measure it.