Avg Crawl Delay (when set) 8s
Mentions
217,201
0.02% of hostnames
Disallow All
9,117
4% of mentions
Allow
65,208
30% of mentions
Disallow All
+ Allow Some
+ Allow Some
4,778
2% of mentions
Disallow Some
37,654
17% of mentions
Disallow Some
+ Allow Some
+ Allow Some
56,357
25% of mentions
No Directives
44,080
20% of mentions
Average Crawl Delay
8
(when set)