magpie-crawler
Avg Crawl Delay (when set) 285s
Mentions
2,123,368
0.15% of hostnames
Disallow All
2,114,052
99% of mentions
Allow
714
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
990
0% of mentions
Disallow Some
805
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
3,911
0% of mentions
No Directives
2,894
0% of mentions
Average Crawl Delay
285
(when set)