Googlebot
Avg Crawl Delay (when set) 35s
Mentions
20,960,772
1.61% of hostnames
Disallow All
631,771
3% of mentions
Allow
1,649,102
7% of mentions
Disallow All
+ Allow Some
+ Allow Some
78,608
0% of mentions
Disallow Some
13,597,651
64% of mentions
Disallow Some
+ Allow Some
+ Allow Some
1,190,637
5% of mentions
No Directives
3,810,117
18% of mentions
Average Crawl Delay
35
(when set)