OpenRobotsTXT

Yeti

Avg Crawl Delay (when set) 25s

Mentions

1,222,413

0.11% of hostnames

Disallow All

1,002,024

81% of mentions

Allow

63,026

5% of mentions

Disallow All
+ Allow Some

1,305

0% of mentions

Disallow Some

41,159

3% of mentions

Disallow Some
+ Allow Some

62,414

5% of mentions

No Directives

52,461

4% of mentions

Average Crawl Delay

25

(when set)