Avg Crawl Delay (when set) 1s
Mentions
4,055,946
0.35% of hostnames
Disallow All
17,554
0% of mentions
Allow
6,628
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
42
0% of mentions
Disallow Some
37,230
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
59,235
1% of mentions
No Directives
3,935,255
97% of mentions
Average Crawl Delay
1
(when set)