Avg Crawl Delay (when set) 9s
Mentions
220,720
0.02% of hostnames
Disallow All
7,844
3% of mentions
Allow
65,726
29% of mentions
Disallow All
+ Allow Some
+ Allow Some
4,084
1% of mentions
Disallow Some
37,711
17% of mentions
Disallow Some
+ Allow Some
+ Allow Some
56,987
25% of mentions
No Directives
48,361
21% of mentions
Average Crawl Delay
9
(when set)