Crawler is designed to only examine robots.txt. Redirects will only be respected if they resolve to robots.txt. Any other content will be disregarded.

The bot only wants to access robots.txt itself. It does not crawl site content, so no directives are triggered or violated. So while it plays by the rules and tries to obey robots.txt instructions, it shouldn't find itself in a situation where it has to.

Data is archived, analysed and made available freely available for verified users on this site, openrobotstxt.org.

Of course.

OpenRobotsTXT adheres to the robots.txt standard. As the bot is only interested in robots.txt, it should not visit any other page on your site unless your host's server configuration orders it to do so. However, for your peace of mind, you can request that it does not look at any page other than robots.txt by adding this line.

If you have reason to believe that OpenRobotsTXT did NOT obey your robots.txt commands, then please let us know. Please provide URL to your website and log entries showing bot trying to retrieve pages other than robots.txt.

Please note that OpenRobotsTXT accepts data from a few trusted partners. They may use their own crawl methods, with a different User Agent and ruleset.

Yes.

As well as offering aggregated reports on webmaster sentiment regarding blocking and allowing certain bots, OpenRobotsTXT aims to act as a back-up for those times when a site's robots.txt is not available. If OpenRobotsTXT is blocked, you remove the ability for other crawlers to check our secondary records to see if and how they should respectfully crawl your site.

Additionally, we will not be able to provide you with an audit ability to see if a named crawler should have been able to visit parts of your site on a certain day.

Robots.txt informs crawling behavior for site content, which our crawler does not visit. Robots.txt instructions are not intended to apply to scanning of robots.txt itself.

However, OpenRobotsTXT will obey the following optional instruction to control republishing of your robots file ONLY WHEN placed in OpenRobotsTXT directives:

Options are [yes/no]. Change Publish-Robots to 'no' to request that we do not publish this robots.txt

Sometimes a server may send a 30x redirect in response to a request for a robots.txt instead of the file anticipated.

The OpenRobotsTXT crawler will follow up to 5 redirects. The crawler will only follow redirects where the path resolves to “/robots.txt”. Redirect chains falling outside of these parameters will cause the crawler to log an error and move on.

This is a stricter interpretation than some other crawlers, which may accommodate internal ( same host ) redirects to any path.

We are keen to see any reports of potential violations of robots.txt by OpenRobotsTXT.

There are a number of false positives raised - this can be a useful checklist when configuring a web server:

• Off site redirects when requesting robots.txt - OpenRobotsTXT follows redirects, but only on the same domain. The ideal is for robots.txt to be available at "/robots.txt" as specified in the standard.
• Multiple domains running on the same server. Modern webservers such as Apache can log accesses to a number of domains to one file - this can cause confusion when attempting to see what webserver was accessed at which point. You may wish to consider adding domain information to the access log, or splitting access logs on a per domain basis
• Robots.txt out of sync with developer copy.

The current operating versions of OpenRobotsTXT is:

• v1.0.0 (Current - May 2025)

We intend to support Reverse DNS. Details coming soon.