Diffbot
Avg Crawl Delay (when set) 11s
Mentions
1,034,332
0.08% of hostnames
Disallow All
931,185
90% of mentions
Allow
41,134
3% of mentions
Disallow All
+ Allow Some
+ Allow Some
2,584
0% of mentions
Disallow Some
2,499
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
2,019
0% of mentions
No Directives
54,875
5% of mentions
Average Crawl Delay
11
(when set)