WebStripper
Avg Crawl Delay (when set) 855s
Mentions
1,493,480
0.13% of hostnames
Disallow All
1,489,371
99% of mentions
Allow
226
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
578
0% of mentions
Disallow Some
746
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
581
0% of mentions
No Directives
1,974
0% of mentions
Average Crawl Delay
855
(when set)