Twitterbot
Avg Crawl Delay (when set) 45s
Mentions
1,517,362
0.11% of hostnames
Disallow All
110,699
7% of mentions
Allow
780,316
51% of mentions
Disallow All
+ Allow Some
+ Allow Some
39,147
2% of mentions
Disallow Some
72,440
4% of mentions
Disallow Some
+ Allow Some
+ Allow Some
228,880
15% of mentions
No Directives
285,855
18% of mentions
Average Crawl Delay
45
(when set)