OpenRobotsTXT is an open archive of the world’s robots.txt files. By visiting domains and caching these files over time, we track how they change and which user agents access them. Our goal is to provide valuable insights, tools, and reports for webmasters, researchers and the wider internet community for open, public study.

The site, as you see it now, is a placeholder while we work behind the scenes to build a searchable robots.txt archive. The next time you return, you'll hopefully be able to browse the robots.txt archives, and check out more in-depth stats on the biggest and newest crawlers.

Find out more »