Slurp
Avg Crawl Delay (when set) 16s
Mentions
4,710,311
0.33% of hostnames
Disallow All
563,455
11% of mentions
Allow
250,943
5% of mentions
Disallow All
+ Allow Some
+ Allow Some
25,307
0% of mentions
Disallow Some
173,209
3% of mentions
Disallow Some
+ Allow Some
+ Allow Some
170,917
3% of mentions
No Directives
3,526,446
74% of mentions
Average Crawl Delay
16
(when set)