Googlebot
Avg Crawl Delay (when set) 37s
Mentions
23,052,427
1.63% of hostnames
Disallow All
745,174
3% of mentions
Allow
2,668,216
11% of mentions
Disallow All
+ Allow Some
+ Allow Some
84,456
0% of mentions
Disallow Some
13,955,500
60% of mentions
Disallow Some
+ Allow Some
+ Allow Some
1,337,491
5% of mentions
No Directives
4,258,556
18% of mentions
Average Crawl Delay
37
(when set)