magpie-crawler
Avg Crawl Delay (when set) 229s
Mentions
1,495,905
0.13% of hostnames
Disallow All
1,491,926
99% of mentions
Allow
423
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
494
0% of mentions
Disallow Some
685
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
177
0% of mentions
No Directives
2,197
0% of mentions
Average Crawl Delay
229
(when set)