Avg Crawl Delay (when set) 70s
Mentions
55,387
0.0% of hostnames
Disallow All
5,863
10% of mentions
Allow
33,147
59% of mentions
Disallow All
+ Allow Some
+ Allow Some
287
0% of mentions
Disallow Some
3,662
6% of mentions
Disallow Some
+ Allow Some
+ Allow Some
10,094
18% of mentions
No Directives
2,329
4% of mentions
Average Crawl Delay
70
(when set)