

Welcome to OpenRobotsTXT
OpenRobotsTXT is an open archive of the world’s robots.txt files. By visiting domains and caching these files over time, we track how they change and which user agents access them. Our goal is to provide valuable insights, tools, and reports for webmasters, researchers and the wider internet community for open, public study.
The site, as you see it now, is a placeholder while we work behind the scenes to build a searchable robots.txt archive. The next time you return, you'll hopefully be able to browse the robots.txt archives, and check out more in-depth stats on the biggest and newest crawlers.
Stats
Hostnames Scanned
622.4m
So far
User-Agents
49,922
Found in files
Last Updated
14 mins ago
15 Jun 2025, 21:47 (UTC)
Top Explicitly Allow-All Bots
User-Agent | Mentions | Allow All | |
1 | Googlebot | 16.3m | 1.0m |
2 | facebookexternalhit | 0.9m | 0.4m |
3 | Mediapartners-Google | 14.4m | 0.4m |