Slurp
Avg Crawl Delay (when set) 16s
Mentions
4,009,692
0.34% of hostnames
Disallow All
435,208
10% of mentions
Allow
177,194
4% of mentions
Disallow All
+ Allow Some
+ Allow Some
23,559
0% of mentions
Disallow Some
183,264
4% of mentions
Disallow Some
+ Allow Some
+ Allow Some
159,141
3% of mentions
No Directives
3,031,298
75% of mentions
Average Crawl Delay
16
(when set)