msnbot-media
Avg Crawl Delay (when set) 102s
Mentions
167,735
0.01% of hostnames
Disallow All
68,894
41% of mentions
Allow
23,070
13% of mentions
Disallow All
+ Allow Some
+ Allow Some
23,599
14% of mentions
Disallow Some
14,841
8% of mentions
Disallow Some
+ Allow Some
+ Allow Some
13,655
8% of mentions
No Directives
23,665
14% of mentions
Average Crawl Delay
102
(when set)