Nutch
Avg Crawl Delay (when set) 24s
Mentions
5,266,339
0.44% of hostnames
Disallow All
5,188,405
98% of mentions
Allow
7,012
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
951
0% of mentions
Disallow Some
4,802
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
509
0% of mentions
No Directives
64,614
1% of mentions
Average Crawl Delay
24
(when set)