Googlebot-News
Avg Crawl Delay (when set) 30s
Mentions
891,103
0.06% of hostnames
Disallow All
25,949
2% of mentions
Allow
707,317
79% of mentions
Disallow All
+ Allow Some
+ Allow Some
23,764
2% of mentions
Disallow Some
14,610
1% of mentions
Disallow Some
+ Allow Some
+ Allow Some
96,172
10% of mentions
No Directives
23,275
2% of mentions
Average Crawl Delay
30
(when set)