SiteSnagger
Avg Crawl Delay (when set) 813s
Mentions
1,491,001
0.11% of hostnames
Disallow All
1,486,413
99% of mentions
Allow
259
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
591
0% of mentions
Disallow Some
711
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
608
0% of mentions
No Directives
2,415
0% of mentions
Average Crawl Delay
813
(when set)