Slurp
Avg Crawl Delay (when set) 16s
Mentions
4,226,870
0.32% of hostnames
Disallow All
489,736
11% of mentions
Allow
197,780
4% of mentions
Disallow All
+ Allow Some
+ Allow Some
23,678
0% of mentions
Disallow Some
194,689
4% of mentions
Disallow Some
+ Allow Some
+ Allow Some
164,434
3% of mentions
No Directives
3,156,523
74% of mentions
Average Crawl Delay
16
(when set)