Googlebot-News
Avg Crawl Delay (when set) 31s
Mentions
501,434
0.04% of hostnames
Disallow All
30,529
6% of mentions
Allow
323,474
64% of mentions
Disallow All
+ Allow Some
+ Allow Some
21,594
4% of mentions
Disallow Some
13,244
2% of mentions
Disallow Some
+ Allow Some
+ Allow Some
92,724
18% of mentions
No Directives
19,853
3% of mentions
Average Crawl Delay
31
(when set)