OpenRobotsTXT

ia_archiver

Avg Crawl Delay (when set) 11s

Mentions

15,833,573

1.33% of hostnames

Disallow All

1,450,043

9% of mentions

Allow

165,895

1% of mentions

Disallow All
+ Allow Some

2,490

0% of mentions

Disallow Some

4,274,793

26% of mentions

Disallow Some
+ Allow Some

123,608

0% of mentions

No Directives

9,816,681

61% of mentions

Average Crawl Delay

11

(when set)