HaosouSpider
Avg Crawl Delay (when set) 22s
Mentions
400,611
0.03% of hostnames
Disallow All
200,933
50% of mentions
Allow
42,474
10% of mentions
Disallow All
+ Allow Some
+ Allow Some
86
0% of mentions
Disallow Some
2,437
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
645
0% of mentions
No Directives
154,034
38% of mentions
Average Crawl Delay
22
(when set)