Twitterbot
Avg Crawl Delay (when set) 16s
Mentions
1,135,747
0.09% of hostnames
Disallow All
91,137
8% of mentions
Allow
459,300
40% of mentions
Disallow All
+ Allow Some
+ Allow Some
38,180
3% of mentions
Disallow Some
73,882
6% of mentions
Disallow Some
+ Allow Some
+ Allow Some
206,058
18% of mentions
No Directives
267,166
23% of mentions
Average Crawl Delay
16
(when set)