Avg Crawl Delay (when set) 9s
Mentions
219,795
0.02% of hostnames
Disallow All
7,821
3% of mentions
Allow
65,496
29% of mentions
Disallow All
+ Allow Some
+ Allow Some
5,985
2% of mentions
Disallow Some
38,128
17% of mentions
Disallow Some
+ Allow Some
+ Allow Some
56,429
25% of mentions
No Directives
45,929
20% of mentions
Average Crawl Delay
9
(when set)