ExtractorPro
Avg Crawl Delay (when set) 410s
Mentions
1,358,150
0.1% of hostnames
Disallow All
1,354,944
99% of mentions
Allow
190
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
579
0% of mentions
Disallow Some
587
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
147
0% of mentions
No Directives
1,699
0% of mentions
Average Crawl Delay
410
(when set)