Googlebot-News
Avg Crawl Delay (when set) 31s
Mentions
250,881
0.02% of hostnames
Disallow All
18,941
7% of mentions
Allow
98,921
39% of mentions
Disallow All
+ Allow Some
+ Allow Some
19,777
7% of mentions
Disallow Some
11,025
4% of mentions
Disallow Some
+ Allow Some
+ Allow Some
86,247
34% of mentions
No Directives
15,957
6% of mentions
Average Crawl Delay
31
(when set)