Avg Crawl Delay (when set) 9s
Mentions
220,606
0.02% of hostnames
Disallow All
7,705
3% of mentions
Allow
65,728
29% of mentions
Disallow All
+ Allow Some
+ Allow Some
4,164
1% of mentions
Disallow Some
37,692
17% of mentions
Disallow Some
+ Allow Some
+ Allow Some
57,016
25% of mentions
No Directives
48,294
21% of mentions
Average Crawl Delay
9
(when set)