magpie-crawler
Avg Crawl Delay (when set) 235s
Mentions
1,417,614
0.12% of hostnames
Disallow All
1,414,019
99% of mentions
Allow
326
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
466
0% of mentions
Disallow Some
676
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
102
0% of mentions
No Directives
2,022
0% of mentions
Average Crawl Delay
235
(when set)