Twitterbot
Avg Crawl Delay (when set) 19s
Mentions
948,786
0.08% of hostnames
Disallow All
54,552
5% of mentions
Allow
346,812
36% of mentions
Disallow All
+ Allow Some
+ Allow Some
35,817
3% of mentions
Disallow Some
67,194
7% of mentions
Disallow Some
+ Allow Some
+ Allow Some
191,627
20% of mentions
No Directives
252,766
26% of mentions
Average Crawl Delay
19
(when set)