AWS Public Sector Blog
Author: Rich Skrenta
Rich Skrenta is an experienced technologist and serial entrepreneur with a background in the search and social spaces. He was founder and CEO of Blekko, a web search engine; the Open Directory Project, an innovative community-edited search platform; Topix, a news aggregator combined with a social forum; and Tobiko, a restaurant recommendation platform. Rich has also run web-scale crawling and machine learning teams for several large organizations, including Meta, IBM Watson, and Netscape/America Online. Rich has a bachelor's degree in computer science from Northwestern University and is the author of many open source software projects.
How Common Crawl and AWS Open Data built the foundation for the AI revolution
The Amazon Web Services (AWS) Open Data Sponsorship Program has hosted the Common Crawl open repository of web data at no cost since January 2012. It has become one of the most important sources of training data for the large language models (LLMs) reshaping industries. This is the story of a 14-year collaboration between a small nonprofit and AWS, and how open data infrastructure became the foundation of the AI era.