
Sold by: Common Crawl
Open data
|
Deployed on AWS
A corpus of web crawl data composed of over 300 billion web pages.
Overview
A corpus of web crawl data composed of over 300 billion web pages.
Features and programs
Open Data Sponsorship Program
This dataset is part of the Open Data Sponsorship Program, an AWS program that covers the cost of storage for publicly available high-value cloud-optimized datasets.
Pricing
This is a publicly available data set. No subscription is required.
How can we make this page better?
Tell us how we can improve this page, or report an issue with this product.
Legal
Content disclaimer
Vendors are responsible for their product descriptions and other product content. AWS does not warrant that vendors' product descriptions or other product content are accurate, complete, reliable, current, or error-free.
Delivery details
AWS Data Exchange (ADX)
AWS Data Exchange is a service that helps AWS easily share and manage data entitlements from other organizations at scale.
Open data resources
Available with or without an AWS account.
- How to use
- To access these resources, reference the Amazon Resource Name (ARN) using the AWS Command Line Interface (CLI). Learn more
- Description
- Crawl data (WARC and ARC format)
- Resource type
- S3 bucket
- Amazon Resource Name (ARN)
- arn:aws:s3:::commoncrawl
- AWS region
- us-east-1
- AWS CLI access (No AWS account required)
- aws s3 ls --no-sign-request s3://commoncrawl/
Resources
Vendor resources
Support
Managed By
How to cite
Common Crawl was accessed on DATE from https://registry.opendata.aws/commoncrawl .
License
This data is available for anyone to use under the Common Crawl Terms of Use
Similar products
Common Room helps teams capture and act on every buying signal in one place. It empowers go-to-market teams with AI-driven insights for effective outreach.
End-to-end implementation and migration of Qualtrics programs, including configuration, integrations, dashboards, localization, testing, and go-live support.

Use Common Crawl (AWS Data Exchange for AWS Lake Formation Test Product) to understand how to interact with data made available via AWS Lake Formation.
To maximize your investment in a new technology solution, your organization needs the skills and know-how to use it effectively. We ensure smooth technology transitions and platform adoption by equipping your team with the right knowledge and experience, setting you up for sustained success.