Twitterbot
Avg Crawl Delay (when set) 44s
Mentions
1,559,649
0.11% of hostnames
Disallow All
111,801
7% of mentions
Allow
809,298
51% of mentions
Disallow All
+ Allow Some
+ Allow Some
39,292
2% of mentions
Disallow Some
73,468
4% of mentions
Disallow Some
+ Allow Some
+ Allow Some
234,790
15% of mentions
No Directives
290,972
18% of mentions
Average Crawl Delay
44
(when set)