OpenRobotsTXT

GoogleOther

Avg Crawl Delay (when set) 13s

Mentions

2,261,762

0.17% of hostnames

Disallow All

299,142

13% of mentions

Allow

11,299

0% of mentions

Disallow All
+ Allow Some

45,015

1% of mentions

Disallow Some

3,938

0% of mentions

Disallow Some
+ Allow Some

1,818,468

80% of mentions

No Directives

83,879

3% of mentions

Average Crawl Delay

13

(when set)