Googlebot
Avg Crawl Delay (when set) 36s
Mentions
22,930,416
1.63% of hostnames
Disallow All
739,626
3% of mentions
Allow
2,597,946
11% of mentions
Disallow All
+ Allow Some
+ Allow Some
83,888
0% of mentions
Disallow Some
13,939,323
60% of mentions
Disallow Some
+ Allow Some
+ Allow Some
1,329,930
5% of mentions
No Directives
4,236,707
18% of mentions
Average Crawl Delay
36
(when set)