OpenRobotsTXT

Yeti

Avg Crawl Delay (when set) 24s

Mentions

1,233,067

0.1% of hostnames

Disallow All

1,008,954

81% of mentions

Allow

62,441

5% of mentions

Disallow All
+ Allow Some

1,278

0% of mentions

Disallow Some

41,385

3% of mentions

Disallow Some
+ Allow Some

62,986

5% of mentions

No Directives

55,998

4% of mentions

Average Crawl Delay

24

(when set)