Twitterbot
Avg Crawl Delay (when set) 16s
Mentions
1,171,241
0.09% of hostnames
Disallow All
91,287
7% of mentions
Allow
493,780
42% of mentions
Disallow All
+ Allow Some
+ Allow Some
37,390
3% of mentions
Disallow Some
69,299
5% of mentions
Disallow Some
+ Allow Some
+ Allow Some
207,839
17% of mentions
No Directives
271,623
23% of mentions
Average Crawl Delay
16
(when set)