Ascii Art of 'OPEN ROBOTS TXT'

Welcome to OpenRobotsTXT

OpenRobotsTXT is an open archive of the world’s robots.txt files. By visiting domains and caching these files over time, we track how they change and which user agents access them. Our goal is to provide valuable insights, tools, and reports for webmasters, researchers and the wider internet community for open, public study.

The site, as you see it now, is a placeholder while we work behind the scenes to build a searchable robots.txt archive. The next time you return, you'll hopefully be able to browse the robots.txt archives, and check out more in-depth stats on the biggest and newest crawlers.

Find out more »

Stats

Hostnames Scanned
622.4m
So far
User-Agents
49,922
Found in files
Last Updated
14 mins ago
15 Jun 2025, 21:47 (UTC)

Top Explicitly Allow-All Bots

User-Agent Mentions Allow All
1 Googlebot 16.3m 1.0m
2 facebookexternalhit 0.9m 0.4m
3 Mediapartners-Google 14.4m 0.4m

More Allowed User-Agents »