Googlebot-News
Avg Crawl Delay (when set) 30s
Mentions
901,564
0.06% of hostnames
Disallow All
25,949
2% of mentions
Allow
717,557
79% of mentions
Disallow All
+ Allow Some
+ Allow Some
23,662
2% of mentions
Disallow Some
14,570
1% of mentions
Disallow Some
+ Allow Some
+ Allow Some
96,154
10% of mentions
No Directives
23,656
2% of mentions
Average Crawl Delay
30
(when set)