AWS Big Data Blog

February 2023 Update: Console access to the AWS Data Pipeline service will be removed on April 30, 2023. On this date, you will no longer be able to access AWS Data Pipeline though the console. You will continue to have access to AWS Data Pipeline through the command line interface and API. Please note that […]

Nick Corbett is a Big Data Consultant for AWS Professional Services Amazon Kinesis is a fully managed service for processing real-time data at massive scale.  Whether you are building a system that collects data from remote sensors, aggregating log files from multiple servers, or creating the latest Internet of Things (IoT) solution, Amazon Kinesis lets […]

Matt Yanchyshyn is a Principal Solutions Architect at AWS I now have a collection of napkins from customer dinners with various AWS technology solutions sketched on them.  This particular napkin is an Amazon DynamoDB schema design for a customer interested in using the new JSON document support to import a bunch of JSON files into […]

Matt Yanchyshyn is a Principal Solutions Architect at AWS During the keynote on Wednesday we announced Amazon RDS for Aurora, a new high-performance and cost-effective relational database.  I heard from multiple AWS re:Invent attendees that they’re really excited about how AWS is innovating in the data storage space, and from a big data perspective it […]

Matt Yanchyshyn is a Principal Solutions Architect at AWS Today hundreds of AWS customers participated in bootcamps at re:Invent, including three sessions in the big data space: Store, Manage, and Analyze Big Data in the Cloud, Real Time Data Processing and Analysis with Amazon Redshift and Amazon Kinesis and Building High-Performance Applications on DynamoDB.  Chris […]

Matt Yanchyshyn is a Principal Solutions Architect at AWS The breakout sesisons at AWS re:Invent start on Wednesday, but plenty of people have already arrived and the product announcements have started. In the big data world, the Amazon DynamoDB team announced a sneak preview of an exciting new feature: DynamoDB Streams (https://aws.amazon.com/blogs/aws/dynamodb-streams-preview/).  This will allow […]

AWS re:Invent is just a few days away! Big data is being used to transform businesses, increase efficiency, and drive innovation. The AWS Cloud has a comprehensive portfolio of big data and HPC services to accelerate and automate how you put data to work in your organization. In the Big Data & HPC track at […]

Rahul Bhartia is an AWS Solutions Architect Streams of data are becoming ubiquitous today – clickstreams, log streams, event streams, and more. The need for real-time processing of high-volume data streams is pushing the limits of traditional data processing infrastructures. Building a clickstream monitoring system, for example, where data is in the form of a continuous clickstream rather […]

This blog post was last reviewed July, 2022. This post explores customer options for building multi-region or multi-availability zone (AZ) clusters. By default, Amazon Redshift has excellent tools to back up your cluster via snapshot to Amazon Simple Storage Service (Amazon S3). These snapshots can be restored in any AZ in that region or transferred […]

The advertising space is going through a rapid transformation and many of the companies driving this change are using AWS services like Amazon EMR, Amazon Redshift, Amazon DynamoDB, Amazon Kinesis, and Amazon CloudFront.  If you work for an ad tech company in the San Francisco area, you should consider attending a free, one-day event for […]