Diffbot
Avg Crawl Delay (when set) 11s
Mentions
967,685
0.08% of hostnames
Disallow All
872,785
90% of mentions
Allow
37,132
3% of mentions
Disallow All
+ Allow Some
+ Allow Some
2,089
0% of mentions
Disallow Some
2,332
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
796
0% of mentions
No Directives
52,523
5% of mentions
Average Crawl Delay
11
(when set)