Meta-ExternalFetcher
Avg Crawl Delay (when set) 14s
Mentions
521,095
0.04% of hostnames
Disallow All
347,815
66% of mentions
Allow
59,958
11% of mentions
Disallow All
+ Allow Some
+ Allow Some
2,977
0% of mentions
Disallow Some
47,157
9% of mentions
Disallow Some
+ Allow Some
+ Allow Some
16,429
3% of mentions
No Directives
46,646
8% of mentions
Average Crawl Delay
14
(when set)