Slurp
Avg Crawl Delay (when set) 16s
Mentions
4,472,077
0.32% of hostnames
Disallow All
517,516
11% of mentions
Allow
237,253
5% of mentions
Disallow All
+ Allow Some
+ Allow Some
24,841
0% of mentions
Disallow Some
180,250
4% of mentions
Disallow Some
+ Allow Some
+ Allow Some
167,406
3% of mentions
No Directives
3,344,777
74% of mentions
Average Crawl Delay
16
(when set)