Slurp
Avg Crawl Delay (when set) 16s
Mentions
4,696,330
0.33% of hostnames
Disallow All
560,679
11% of mentions
Allow
250,325
5% of mentions
Disallow All
+ Allow Some
+ Allow Some
25,375
0% of mentions
Disallow Some
173,257
3% of mentions
Disallow Some
+ Allow Some
+ Allow Some
170,869
3% of mentions
No Directives
3,515,792
74% of mentions
Average Crawl Delay
16
(when set)