OpenRobotsTXT

Yeti

Avg Crawl Delay (when set) 20s

Mentions

1,287,786

0.09% of hostnames

Disallow All

1,034,386

80% of mentions

Allow

73,079

5% of mentions

Disallow All
+ Allow Some

1,362

0% of mentions

Disallow Some

41,937

3% of mentions

Disallow Some
+ Allow Some

72,075

5% of mentions

No Directives

64,920

5% of mentions

Average Crawl Delay

20

(when set)