Meta
Avg Crawl Delay (when set) 1028s
Mentions
76,679
0.01% of hostnames
Disallow All
3,194
4% of mentions
Allow
73,372
95% of mentions
Disallow All
+ Allow Some
+ Allow Some
6
0% of mentions
Disallow Some
26
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
5
0% of mentions
No Directives
76
0% of mentions
Average Crawl Delay
1028
(when set)