Twitterbot
Avg Crawl Delay (when set) 38s
Mentions
1,404,242
0.1% of hostnames
Disallow All
105,041
7% of mentions
Allow
682,343
48% of mentions
Disallow All
+ Allow Some
+ Allow Some
38,596
2% of mentions
Disallow Some
72,319
5% of mentions
Disallow Some
+ Allow Some
+ Allow Some
224,062
15% of mentions
No Directives
281,855
20% of mentions
Average Crawl Delay
38
(when set)