SiteSnagger
Avg Crawl Delay (when set) 847s
Mentions
1,463,844
0.13% of hostnames
Disallow All
1,459,858
99% of mentions
Allow
221
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
576
0% of mentions
Disallow Some
719
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
581
0% of mentions
No Directives
1,885
0% of mentions
Average Crawl Delay
847
(when set)