Wget
Avg Crawl Delay (when set) 908s
Mentions
1,334,912
0.12% of hostnames
Disallow All
1,328,258
99% of mentions
Allow
411
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
655
0% of mentions
Disallow Some
2,121
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
715
0% of mentions
No Directives
2,748
0% of mentions
Average Crawl Delay
908
(when set)