Twitterbot
Avg Crawl Delay (when set) 16s
Mentions
1,135,666
0.09% of hostnames
Disallow All
91,138
8% of mentions
Allow
459,250
40% of mentions
Disallow All
+ Allow Some
+ Allow Some
38,181
3% of mentions
Disallow Some
73,898
6% of mentions
Disallow Some
+ Allow Some
+ Allow Some
206,052
18% of mentions
No Directives
267,123
23% of mentions
Average Crawl Delay
16
(when set)