HTTrack
Avg Crawl Delay (when set) 211s
Mentions
1,330,377
0.11% of hostnames
Disallow All
1,323,033
99% of mentions
Allow
504
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
425
0% of mentions
Disallow Some
1,546
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
786
0% of mentions
No Directives
4,075
0% of mentions
Average Crawl Delay
211
(when set)