Diffbot
Avg Crawl Delay (when set) 15s
Mentions
1,383,595
0.1% of hostnames
Disallow All
1,246,231
90% of mentions
Allow
58,667
4% of mentions
Disallow All
+ Allow Some
+ Allow Some
4,466
0% of mentions
Disallow Some
42,237
3% of mentions
Disallow Some
+ Allow Some
+ Allow Some
10,096
0% of mentions
No Directives
21,847
1% of mentions
Average Crawl Delay
15
(when set)