HTTrack
Avg Crawl Delay (when set) 220s
Mentions
1,362,080
0.1% of hostnames
Disallow All
1,354,239
99% of mentions
Allow
540
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
434
0% of mentions
Disallow Some
1,552
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
872
0% of mentions
No Directives
4,434
0% of mentions
Average Crawl Delay
220
(when set)