HTTrack
Avg Crawl Delay (when set) 220s
Mentions
1,387,510
0.1% of hostnames
Disallow All
1,379,797
99% of mentions
Allow
585
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
425
0% of mentions
Disallow Some
1,578
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
854
0% of mentions
No Directives
4,263
0% of mentions
Average Crawl Delay
220
(when set)