Slurp
Avg Crawl Delay (when set) 16s
Mentions
3,987,446
0.34% of hostnames
Disallow All
430,083
10% of mentions
Allow
175,747
4% of mentions
Disallow All
+ Allow Some
+ Allow Some
23,205
0% of mentions
Disallow Some
183,069
4% of mentions
Disallow Some
+ Allow Some
+ Allow Some
159,061
3% of mentions
No Directives
3,016,253
75% of mentions
Average Crawl Delay
16
(when set)