Googlebot
Avg Crawl Delay (when set) 38s
Mentions
23,612,583
1.63% of hostnames
Disallow All
807,384
3% of mentions
Allow
2,931,365
12% of mentions
Disallow All
+ Allow Some
+ Allow Some
87,552
0% of mentions
Disallow Some
14,070,011
59% of mentions
Disallow Some
+ Allow Some
+ Allow Some
1,379,116
5% of mentions
No Directives
4,334,303
18% of mentions
Average Crawl Delay
38
(when set)