Slurp
Avg Crawl Delay (when set) 16s
Mentions
4,151,123
0.32% of hostnames
Disallow All
472,351
11% of mentions
Allow
192,155
4% of mentions
Disallow All
+ Allow Some
+ Allow Some
24,378
0% of mentions
Disallow Some
198,505
4% of mentions
Disallow Some
+ Allow Some
+ Allow Some
162,560
3% of mentions
No Directives
3,101,142
74% of mentions
Average Crawl Delay
16
(when set)