This dataset contains a 320 GB sample of the data used to power trendingtopics.org. It includes 7 months of hourly page traffic statistics for over 2.5 Million wikipedia articles (~ 1 TB uncompressed) along with the associated wikipedia content, linkgraph, & metadata.
Compiled by Peter Skomoroch at Data Wrangling, LLC on May, 31, 2009
To mount the snapshot:
localmachine $ ec2-create-volume --snapshot snap-753dfc1c -z us-east-1a localmachine $ ec2-attach-volume vol-ec123456 -i i-df123456 -d /dev/sdf root@domU-XX-XX-XX-XX-XX-XX:/mnt# mkdir /mnt/wikidata root@domU-XX-XX-XX-XX-XX-XX:/mnt# mount /dev/sdf /mnt/wikidata
Contents of the snapshot:
Like Wikipedia itself, All text content is licensed under the GNU Free Documentation License (GFDL). All statistics and link data is also licensed under the GNU Free Documentation License (GFDL). http://www.gnu.org/copyleft/fdl.html
wikidata/wikistats (260G)Contains hourly wikipedia article traffic statistics dataset covering 7 month period from October 01 2008 to April 30 2009, this data is regularly logged from the wikipedia squid proxy by Domas Mituzas.
Each log file is named with the date and time of collection: pagecounts-20090430-230000.gz
Each line has 4 fields: projectcode, pagename, pageviews, bytes
en Barack_Obama 997 123091092 en Barack_Obama%27s_first_100_days 8 850127 en Barack_Obama,_Jr 1 144103 en Barack_Obama,_Sr. 37 938821 en Barack_Obama_%22HOPE%22_poster 4 81005 en Barack_Obama_%22Hope%22_poster 5 102081
wikidata/wikilinks (1.1G)Contains a wikipedia linkgraph dataset provided by Henry Haselgrove.
These files contain all links between proper english language Wikipedia pages, that is pages in "namespace 0". This includes disambiguation pages and redirect pages.
In links-simple-sorted.txt, there is one line for each page that has links from it. The format of the lines is:
from1: to11 to12 to13 ... from2: to21 to22 to23 ... ...
where from1 is an integer labelling a page that has links from it, and to11 to12 to13 ... are integers labelling all the pages that the page links to. To find the page title that corresponds to integer n, just look up the n-th line in the file titles-sorted.txt.
Contains raw wikipedia dumps along with some processed versions using data from: http://en.wikipedia.org/wiki/Wikipedia_databaseSee README files in the corresponding subdirectories for more details
- Data Wrangling -