Twitterbot
Avg Crawl Delay (when set) 45s
Mentions
1,554,303
0.11% of hostnames
Disallow All
111,676
7% of mentions
Allow
805,134
51% of mentions
Disallow All
+ Allow Some
+ Allow Some
39,349
2% of mentions
Disallow Some
73,491
4% of mentions
Disallow Some
+ Allow Some
+ Allow Some
236,199
15% of mentions
No Directives
288,426
18% of mentions
Average Crawl Delay
45
(when set)