Twitterbot
Avg Crawl Delay (when set) 17s
Mentions
1,024,939
0.09% of hostnames
Disallow All
70,683
6% of mentions
Allow
392,046
38% of mentions
Disallow All
+ Allow Some
+ Allow Some
37,324
3% of mentions
Disallow Some
65,895
6% of mentions
Disallow Some
+ Allow Some
+ Allow Some
201,109
19% of mentions
No Directives
257,864
25% of mentions
Average Crawl Delay
17
(when set)