WebCopier
Avg Crawl Delay (when set) 228s
Mentions
1,579,111
0.12% of hostnames
Disallow All
1,572,309
99% of mentions
Allow
294
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
679
0% of mentions
Disallow Some
995
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
630
0% of mentions
No Directives
4,199
0% of mentions
Average Crawl Delay
228
(when set)