Meta-ExternalFetcher
Avg Crawl Delay (when set) 12s
Mentions
580,291
0.04% of hostnames
Disallow All
432,028
74% of mentions
Allow
42,891
7% of mentions
Disallow All
+ Allow Some
+ Allow Some
2,386
0% of mentions
Disallow Some
7,873
1% of mentions
Disallow Some
+ Allow Some
+ Allow Some
14,152
2% of mentions
No Directives
80,919
13% of mentions
Average Crawl Delay
12
(when set)