Googlebot-News
Avg Crawl Delay (when set) 28s
Mentions
294,638
0.02% of hostnames
Disallow All
29,912
10% of mentions
Allow
122,028
41% of mentions
Disallow All
+ Allow Some
+ Allow Some
22,189
7% of mentions
Disallow Some
12,769
4% of mentions
Disallow Some
+ Allow Some
+ Allow Some
88,808
30% of mentions
No Directives
18,918
6% of mentions
Average Crawl Delay
28
(when set)