Slurp
Avg Crawl Delay (when set) 16s
Mentions
4,636,446
0.32% of hostnames
Disallow All
532,885
11% of mentions
Allow
248,631
5% of mentions
Disallow All
+ Allow Some
+ Allow Some
25,222
0% of mentions
Disallow Some
173,526
3% of mentions
Disallow Some
+ Allow Some
+ Allow Some
169,826
3% of mentions
No Directives
3,486,323
75% of mentions
Average Crawl Delay
16
(when set)