We have so many plans for the Stats area of the site. For now, here are a few high-level data tables that contain the type of analyses we plan to release with our first full site version.

If you have any incredible ideas for some aggregated information you'd like to see here, please do get in touch.

Note. These tables are reports of what we see in robots.txt files. As these files are rarely validated or checked, there will be misspellings and incorrectly formatted crawler names. So, bear in mind, something that you see labelled as a "bot" may be a common typo, rather than a real crawler.

User-Agents mentioned in robots.txt in all analysed sites

• Most Directed
• Most Obscure

Other interesting stats

• Newest Bots

These files are an early-release sample of the data we will be providing. It is very likely that layout and column headers will change in the future.