OpenRobotsTXT

facebookexternalhit

Avg Crawl Delay (when set) 39s

Mentions

2,047,767

0.14% of hostnames

Disallow All

228,392

11% of mentions

Allow

1,137,066

55% of mentions

Disallow All
+ Allow Some

27,127

1% of mentions

Disallow Some

103,907

5% of mentions

Disallow Some
+ Allow Some

382,148

18% of mentions

No Directives

169,004

8% of mentions

Average Crawl Delay

39

(when set)