Googlebot-News
Avg Crawl Delay (when set) 29s
Mentions
253,394
0.02% of hostnames
Disallow All
19,194
7% of mentions
Allow
100,250
39% of mentions
Disallow All
+ Allow Some
+ Allow Some
19,896
7% of mentions
Disallow Some
11,146
4% of mentions
Disallow Some
+ Allow Some
+ Allow Some
86,684
34% of mentions
No Directives
16,211
6% of mentions
Average Crawl Delay
29
(when set)