WebSauger
Avg Crawl Delay (when set) 400s
Mentions
1,373,742
0.1% of hostnames
Disallow All
1,371,155
99% of mentions
Allow
192
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
593
0% of mentions
Disallow Some
517
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
148
0% of mentions
No Directives
1,134
0% of mentions
Average Crawl Delay
400
(when set)