Diffbot
Avg Crawl Delay (when set) 11s
Mentions
881,275
0.08% of hostnames
Disallow All
792,293
89% of mentions
Allow
32,304
3% of mentions
Disallow All
+ Allow Some
+ Allow Some
1,992
0% of mentions
Disallow Some
2,238
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
633
0% of mentions
No Directives
51,795
5% of mentions
Average Crawl Delay
11
(when set)