OpenRobotsTXT

Diffbot

Avg Crawl Delay (when set) 14s

Mentions

1,360,099

0.09% of hostnames

Disallow All

1,225,162

90% of mentions

Allow

57,562

4% of mentions

Disallow All
+ Allow Some

4,464

0% of mentions

Disallow Some

41,215

3% of mentions

Disallow Some
+ Allow Some

9,763

0% of mentions

No Directives

21,885

1% of mentions

Average Crawl Delay

14

(when set)