Xenu
Avg Crawl Delay (when set) 793s
Mentions
1,488,954
0.11% of hostnames
Disallow All
1,484,561
99% of mentions
Allow
262
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
411
0% of mentions
Disallow Some
746
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
514
0% of mentions
No Directives
2,457
0% of mentions
Average Crawl Delay
793
(when set)