HaosouSpider
Avg Crawl Delay (when set) 19s
Mentions
404,881
0.03% of hostnames
Disallow All
198,092
48% of mentions
Allow
42,728
10% of mentions
Disallow All
+ Allow Some
+ Allow Some
185
0% of mentions
Disallow Some
1,817
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
965
0% of mentions
No Directives
161,092
39% of mentions
Average Crawl Delay
19
(when set)