Slurp
Avg Crawl Delay (when set) 16s
Mentions
2,752,214
0.24% of hostnames
Disallow All
362,986
13% of mentions
Allow
154,932
5% of mentions
Disallow All
+ Allow Some
+ Allow Some
22,099
0% of mentions
Disallow Some
180,697
6% of mentions
Disallow Some
+ Allow Some
+ Allow Some
155,727
5% of mentions
No Directives
1,875,747
68% of mentions
Average Crawl Delay
16
(when set)