SiteSnagger
Avg Crawl Delay (when set) 806s
Mentions
1,510,726
0.11% of hostnames
Disallow All
1,506,118
99% of mentions
Allow
253
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
596
0% of mentions
Disallow Some
720
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
625
0% of mentions
No Directives
2,410
0% of mentions
Average Crawl Delay
806
(when set)