Googlebot-News
Avg Crawl Delay (when set) 30s
Mentions
705,151
0.05% of hostnames
Disallow All
26,190
3% of mentions
Allow
525,390
74% of mentions
Disallow All
+ Allow Some
+ Allow Some
24,501
3% of mentions
Disallow Some
14,340
2% of mentions
Disallow Some
+ Allow Some
+ Allow Some
94,445
13% of mentions
No Directives
20,269
2% of mentions
Average Crawl Delay
30
(when set)