Googlebot-News
Avg Crawl Delay (when set) 31s
Mentions
267,835
0.02% of hostnames
Disallow All
21,866
8% of mentions
Allow
108,051
40% of mentions
Disallow All
+ Allow Some
+ Allow Some
20,823
7% of mentions
Disallow Some
11,684
4% of mentions
Disallow Some
+ Allow Some
+ Allow Some
87,842
32% of mentions
No Directives
17,556
6% of mentions
Average Crawl Delay
31
(when set)