Posted On: Dec 21, 2016
You can now configure Amazon Kinesis Firehose to prepare your streaming data before it is loaded to data stores. With this new feature, you can easily convert raw streaming data from your data sources into formats required by your destination data stores, without having to build your own data processing pipelines.
To use this feature, simply select an AWS Lambda function from the Amazon Kinesis Firehose delivery stream configuration tab in the AWS Management console. Amazon Kinesis Firehose will automatically apply that function to every input data record and load the transformed data to destinations.
Amazon Kinesis Firehose provides pre-built Lambda blueprints for converting common data sources such as Apache logs and system logs to JSON and CSV formats. You can use these pre-built blueprints without any change, or customize them further, or write your own custom functions. You can also configure Amazon Kinesis Firehose to automatically retry failed jobs and back up the raw streaming data.
For more information about this feature, see Data Transformation in the Amazon Kinesis Firehose developer guide.
Amazon Kinesis Firehose is the easiest way to load streaming data into Amazon Elasticsearch Service, Amazon Redshift, Amazon Kinesis Analytics, and Amazon S3. To get started with Amazon Kinesis Firehose, sign into the AWS Management console. To learn more about Amazon Kinesis Firehose, see our website.