Googlebot
Avg Crawl Delay (when set) 35s
Mentions
23,402,844
2.03% of hostnames
Disallow All
481,755
2% of mentions
Allow
1,316,164
5% of mentions
Disallow All
+ Allow Some
+ Allow Some
73,749
0% of mentions
Disallow Some
17,898,123
76% of mentions
Disallow Some
+ Allow Some
+ Allow Some
1,104,322
4% of mentions
No Directives
2,526,017
10% of mentions
Average Crawl Delay
35
(when set)