HTTrack
Avg Crawl Delay (when set) 212s
Mentions
1,371,075
0.1% of hostnames
Disallow All
1,363,271
99% of mentions
Allow
577
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
432
0% of mentions
Disallow Some
1,554
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
854
0% of mentions
No Directives
4,378
0% of mentions
Average Crawl Delay
212
(when set)