Googlebot-News
Avg Crawl Delay (when set) 29s
Mentions
299,277
0.02% of hostnames
Disallow All
29,887
9% of mentions
Allow
126,052
42% of mentions
Disallow All
+ Allow Some
+ Allow Some
21,431
7% of mentions
Disallow Some
12,961
4% of mentions
Disallow Some
+ Allow Some
+ Allow Some
89,674
29% of mentions
No Directives
19,258
6% of mentions
Average Crawl Delay
29
(when set)