Amazon Kinesis Firehose is the easiest way to load streaming data into AWS. It can capture and automatically load streaming data into Amazon Kinesis Analytics, Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service, enabling near real-time analytics with existing business intelligence tools and dashboards you’re already using today. It is a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration. It can also batch, compress, and encrypt the data before loading it, minimizing the amount of storage used at the destination and increasing security.
You can easily create a Firehose delivery stream from the AWS Management Console, configure it with a few clicks, and start sending data to the stream from hundreds of thousands of data sources to be loaded continuously to AWS – all in just a few minutes.
With Amazon Kinesis Firehose, you only pay for the amount of data you transmit through the service. There is no minimum fee or setup cost.
Amazon Kinesis Firehose provides a simple way to capture and load streaming data with just a few clicks in the AWS Management Console. You can simply create a Firehose delivery stream, select the destinations, and you can start sending real-time data from hundreds of thousands of data sources simultaneously. The service takes care of stream management, including all the scaling, sharding, and monitoring, needed to continuously load the data to destinations at the intervals you specify.
Amazon Kinesis Firehose is integrated with Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service. From the AWS Management Console, you can point Amazon Kinesis Firehose to an Amazon S3 bucket, Amazon Redshift table, or Amazon Elasticsearch domain. You can then use your existing analytics applications and tools to analyze streaming data.
Amazon Kinesis Firehose automatically scales to handle gigabytes per second of streaming data from hundreds of thousands of data sources simultaneously. No intervention or maintenance is needed to maintain data throughput.
Amazon Kinesis Firehose captures and loads data in near real-time. It loads new data into Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service within 60 seconds after the data is sent to the service. As a result, you can access new data sooner and react to business and operational events faster.
Amazon Kinesis Firehose is a fully managed service which automatically provisions, manages and scales compute, memory, and network resources required to load your streaming data. Once set up, Amazon Kinesis Firehose loads data continuously as it arrives.
With Amazon Kinesis Firehose, you pay only for the volume of data you transmit through the service. There are no minimum fees or upfront commitments.
With Amazon Kinesis Firehose, you can capture data continuously from connected devices such as consumer appliances, embedded sensors, and TV set-top boxes. Amazon Kinesis Firehose loads the data into Amazon Redshift, enabling you to provide your customers near real-time access to metrics, insights, and dashboards. For example, Hello Inc., a consumer devices company, collects and processes real-time data from embedded sensors in mattresses with Amazon Kinesis Streams, and uses Amazon Kinesis Firehose to load the processed information into Amazon Redshift. Hello’s customers can monitor their sleep patterns, conditions that influence sleep like room temperature and humidity, and benchmark averages, enabling them to sleep better. Read the case study »
"With Firehose and Redshift, going from raw, streaming data to advanced, real-time insights was just a matter of a few clicks."
- Tim Bart, CTO, Hello Inc.
You can ingest different types of data records from audience tracking systems, Ad Exchange listeners/bidders, and Ad servers, and multiplex them into the same stream. You can then perform data transformations and analytics either in micro-batches with Spark Streaming or on an event-by-event basis with AWS Lambda. Using Amazon Kinesis Firehose, you can deliver the transformed, enriched, summarized data to other analytics and storage services including Amazon S3 and Amazon Redshift. Try Sample Project »
You can use Amazon Kinesis Firehose to enable delivery of real-time metrics on digital content, enabling authors and marketers to connect with their customers in the most effective way. You can stream billions of small messages that are compressed, encrypted, and delivered to Amazon Elasticsearch Service and Amazon Redshift. Once in Amazon Elasticsearch Service and Amazon Redshift, you can aggregate, filter, and process the data, and refresh content performance dashboards in near real-time. For example, Hearst Corporation built a clickstream analytics platform using Amazon Kinesis Streams, Amazon Kinesis Firehose, Amazon Redshift, and Amazon Elasticsearch Service to transmit and process 30 terabytes of data a day from 300+ Hearst websites worldwide. With this platform, Hearst is able to make the entire data stream—from website clicks to aggregated data—available to editors in minutes. Read the case study »
"I don't know how we could have made our clickstream data pipeline work without Amazon Kinesis. It would have involved many weeks of engineering. Kinesis Streams and Firehose make the entire process extremely simple and reliable."
- Peter Jaffe, Data Scientist, Hearst Corporation
In this guest post, Anton Slutsky of MeetMe will discuss a solution using Amazon Kinesis Firehose to optimize and streamline large-scale data ingestion at MeetMe, which is a popular social discovery platform that caters to more than a million active daily users. The Data Science team at MeetMe needed to collect and store approximately 0.5 TB per day of various types of data in a way that would expose it to data mining tasks, business-facing reporting and advanced analytics. The team selected Amazon S3 as the target storage facility and faced a challenge of collecting the large volumes of live data in a robust, reliable, scalable and operationally affordable way.
Amazon Kinesis Agent is a stand-alone Java software application that provides an easy and reliable way to send data to Amazon Kinesis Streams and Amazon Kinesis Firehose. The agent monitors a set of files for new data and then sends it to Kinesis Streams or Kinesis Firehose continuously. It handles file rotation, checkpointing, and retrial upon failures. It also supports Amazon CloudWatch so that you can closely monitor and troubleshoot the data flow from the agent.
In this post we use Twitter public streams to analyze the candidates’ performance, both Republican and Democrat, in a near real-time fashion. We show you how to integrate Amazon Kinesis Firehose, AWS Lambda (Python function), and Amazon Elasticsearch Service to create an end-to-end, near real-time discovery platform.
For a full list of blog posts related to Amazon Kinesis Firehose and real-time analytics, see the blog posts page »
For information about all the new features in Amazon Kinesis Firehose, see the what's new page »