Public Data Sets

A corpus of web crawl data composed of over 5 billion web pages. This data set is freely available on Amazon S3 and is released under the Common Crawl Terms of Use.

A data set containing Google Books n-gram corpuses. This data set is freely available on Amazon S3 in a Hadoop friendly file format and is licensed under a Creative Commons Attribution 3.0 Unported License. The original dataset is available from http://books.google.com/ngrams/.

A data dump of the basic identifying facts about every topic in Freebase

A data dump of all the current facts and assertions in Freebase

Contains 16 months of hourly pageview statistics for all articles in Wikipedia

DBpedia is a community effort to extract structured information from Wikipedia and to make this information available on the Web

A complete copy of all Wikimedia wikis, in the form of wikitext source and metadata embedded in XML.

Contains 7 months of hourly pageview statistics for all articles in Wikipedia

Freebase is an open database of the world's information, covering millions of topics in hundreds of categories

A processed dump of the English language Wikipedia