Wget
Avg Crawl Delay (when set) 877s
Mentions
1,397,563
0.1% of hostnames
Disallow All
1,390,421
99% of mentions
Allow
474
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
708
0% of mentions
Disallow Some
2,222
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
752
0% of mentions
No Directives
2,980
0% of mentions
Average Crawl Delay
877
(when set)