WebStripper
Avg Crawl Delay (when set) 868s
Mentions
1,552,017
0.11% of hostnames
Disallow All
1,547,422
99% of mentions
Allow
269
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
613
0% of mentions
Disallow Some
750
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
618
0% of mentions
No Directives
2,341
0% of mentions
Average Crawl Delay
868
(when set)