magpie-crawler
Avg Crawl Delay (when set) 218s
Mentions
1,839,501
0.13% of hostnames
Disallow All
1,833,872
99% of mentions
Allow
571
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
961
0% of mentions
Disallow Some
811
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
277
0% of mentions
No Directives
3,006
0% of mentions
Average Crawl Delay
218
(when set)