Googlebot
Avg Crawl Delay (when set) 35s
Mentions
23,011,205
1.99% of hostnames
Disallow All
489,225
2% of mentions
Allow
1,320,654
5% of mentions
Disallow All
+ Allow Some
+ Allow Some
73,823
0% of mentions
Disallow Some
17,313,816
75% of mentions
Disallow Some
+ Allow Some
+ Allow Some
1,108,855
4% of mentions
No Directives
2,702,120
11% of mentions
Average Crawl Delay
35
(when set)