WebStripper
Avg Crawl Delay (when set) 843s
Mentions
1,524,299
0.12% of hostnames
Disallow All
1,519,600
99% of mentions
Allow
250
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
591
0% of mentions
Disallow Some
735
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
613
0% of mentions
No Directives
2,506
0% of mentions
Average Crawl Delay
843
(when set)