OpenRobotsTXT is an open archive of the world’s robots.txt files. By visiting domains and caching these files over time, we track how they change and which user agents access them. Our goal is to provide valuable insights, tools, and reports for webmasters, researchers and the wider internet community for open, public study.

Here is what you can do.

• View Stats for the user-agents (bots) we’ve detected in our analysis.
• Lookup data on individual bots via the bots search. Validate your user-agent declarations against web consensus.
• Look at your robots.txt as bots do. Great for making sure your server setup delivers the robots.txt file as you intend.

If you want to learn more about the OpenRobotsTXT project, visit the about page.