Amazon Kinesis provides an easy way to process fast moving data in real-time. You simply provision the level of input and output required for your data stream, in blocks of 1 megabyte per second (MB/sec), using the AWS web management console, API, or SDKs. The size of your stream can be adjusted up or down at any time without restarting the stream and without any impact on the data sources pushing data to Amazon Kinesis. Once your stream has been created, you can immediately start loading your data, with simple HTTP PUTs. Amazon Kinesis automatically manages the infrastructure, storage, networking, and configuration to collect and process your data at the level of throughput your application needs.
Within seconds, data PUT into an Amazon Kinesis stream is available for analysis. Stream data is stored across multiple availability zones in a region for 24 hours. During that window, data is available to be read, re-read, backfilled, and analyzed, or moved to long-term storage (e.g. Amazon S3 or Amazon Redshift). The Amazon Kinesis client library enables developers to focus on creating their business applications while offloading the burden associated with load-balancing streaming data, coordinating distributed services, and fault-tolerant data processing. Low latency connections between Amazon EC2 and Amazon Kinesis enable you to build applications that can respond quickly to your data, whether that is a stock ticker, a social media feed, or a sensor network. Amazon Kinesis also integrates with other AWS services built to manage data at scale, such as Amazon DynamoDB, Amazon Redshift, Amazon S3, and AWS Lambda.
Amazon Kinesis is a managed service, which means that you don't have to manage storage options, configure complex software and file systems, or worry about scaling. Amazon Kinesis will automatically manage the infrastructure resources required to collect and store your data, leaving you free to focus on high-value decisions and functionality. Amazon Kinesis uses proven cryptographic methods to securely authenticate users and helps prevent unauthorized data access. You'll only pay per one million data PUTs, and for the provisioned throughput of your stream.
Amazon Kinesis is a fully managed service for real-time streaming data ingestion and processing – you simply create a stream and let the service handle the rest. Amazon Kinesis reduces your operational burden by offloading the tasks related to hardware or software provisioning, software patching, or operating a reliable, scalable, and durable distributed real-time processing platform.
Amazon Kinesis has built-in fault tolerance, automatically and synchronously replicating your data across multiple Availability Zones in a Region. Amazon Kinesis' high availability and durability help protect your data against failures ranging from individual machines to an entire facility.
When creating a stream, simply specify how much throughput capacity you require in terms of shards. With each shard in Amazon Kinesis, you can capture up to 1 megabyte per second of data at 1,000 PUT records per second. Your Amazon Kinesis applications can read data from each Shard at up to 2 megabytes per second. If your throughput requirements change, simply update your stream's capacity by changing the number of Shards you need using the AWS Management Console or the Amazon Kinesis APIs. Your stream keeps flowing while scaling is underway, with new capacity available within seconds.
Amazon Kinesis integrates with Amazon S3, Amazon Redshift, and Amazon DynamoDB. With Amazon Kinesis, you can reliably ingest, process, and transform all of your data in real-time, before sending it to a business application, or a data store for further analysis. For example, with Amazon Kinesis it is easy for you to write an application that de-duplicates and aggregates your log data, and then archive it to Amazon S3. Another example is a Amazon Kinesis application that extracts key metrics from the data and then, using the Amazon Kinesis-Redshift connector, writes it to your Amazon Redshift data warehouse for querying by business analysts, product managers, or other staff in your organization. Amazon Kinesis also integrates with AWS Lambda to trigger action on specific data as well as AWS Identity and Access Management (IAM) to provide fine-grained access control.
Amazon Kinesis provides developers with client libraries to build and operate real-time streaming data processing applications. The Amazon Kinesis Client Library enables you to focus on your business logic, letting the client library automatically handle complex issues like adapting to changes in stream volume, load-balancing streaming data, coordinating distributed services, and processing data with fault-tolerance.
Amazon Kinesis displays key operational metrics for your streams in the AWS Management Console. The service also integrates with Amazon CloudWatch so you can see your throughput, utilization, and latency for each Amazon Kinesis stream, and easily track your resource consumption.
Amazon Kinesis is designed to be cost-efficient for workloads of any scale. You can get started by provisioning a simple stream comprised of a single shard, enabling up to 1 MB per second of reliably-ingested and durably-stored data. With pay-as-you-go pricing, you pay low hourly rates only for the resources you consume. With easy administration and usage-based pricing, Amazon Kinesis can offer significantly lower total cost of ownership (TCO) compared to custom-built alternatives.