AWS Big Data Blog

Stream data to an HTTP endpoint with Amazon Data Firehose

November 2024: This post was reviewed and updated for accuracy.

The value of data is time sensitive. Streaming data services can help you move data quickly from data sources to new destinations for downstream processing. For example, Amazon Data Firehose can reliably load streaming data into data stores like Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon OpenSearch Service, and Splunk.

We’re now expanding the Firehose delivery destinations to include generic HTTP endpoints. This enables you to use a fully managed delivery service to HTTP endpoints without building custom applications or worrying about operating and managing the data delivery infrastructure. Additionally, the HTTP endpoint enhancement opens up a number of key integration opportunities between Firehose and other AWS services such as Amazon DynamoDB or Amazon SNS or Amazon RDS using Amazon API Gateway’s AWS Service integrations.

All the existing Firehose features are fully supported, including AWS Lambda service integration, retry option, data protection on delivery failure, and cross-account and cross-Region data delivery. Traffic between Firehose and the HTTP endpoint is encrypted in transit using HTTPS. Firehose incorporates error handling, automatic scaling, transformation, conversion, aggregation, and compression functionality to help you accelerate the deployment of data streams across your organization. There is no additional cost to use this feature.

In this post, we walk through setting up a Firehose HTTP endpoint. Our example use case ingests data into a Firehose delivery stream and sends it to an Amazon API Gateway REST API endpoint that conditionally writes the data to a SNS topic using the AWS Lambda integration.

Configuring a delivery stream to an HTTP endpoint

To set this up, you provide the URL for your HTTP endpoint application or service, add an optional access key as a header to the HTTP calls made to your HTTP endpoint, and include optional key-value parameters. You can configure the endpoint on the AWS Management Console, the AWS Command Line Interface (AWS CLI), or the AWS SDK.

  1. On the console, choose Amazon Data Firehose.
  2. Choose Create Firehose stream.
  3. For source, select Direct PUT or other sources as the source using the Firehose PutRecord API.
  4. For destination, select HTTP Endpoint.
  5. Use the prefilled Firehose stream name or change the name as required.
  6. Skip the Transform records.
  7. Move to Destination settings.
  8. For HTTP endpoint name, enter a name.
  9. For HTTP endpoint URL, enter your endpoint URL.

The API Gateway REST API endpoint receives incoming data from the Firehose stream. It then triggers a Lambda function via the API Gateway Lambda integration. The Lambda function processes the received data and checks if a specific field, here PRICE, within the data exceeds a predefined threshold value. If the condition is met, the Lambda function publishes a notification message to a SNS topic. a Make sure you follow the HTTP endpoint delivery request and response specifications for Data Firehose to successfully deliver data to a custom HTTP endpoint. More specifically, the request and response from the HTTP endpoint must contain requestId and a timestamp. Importantly, requestId in the response must match the requestId in the original request.

  1. For Access key, enter an access key (optional).
  2. For Content encoding, select Not Enabled
  3. For Retry duration, leave at its default of 300 seconds.

You can optionally choose to encode and compress your request body before posting it to your HTTP endpoint.

  1. For Parameters, you can pass optional key-value parameters as needed.
  2. For S3 backup mode, select Failed data only.
  3. For S3 bucket, enter an S3 bucket as a backup for the delivery stream to store data that failed delivery to the HTTP API endpoint.
  4. Review your settings and choose Create Firehose stream.

When the Firehose stream is active, your source can start streaming data to it.

Testing the delivery stream

For this post, we use the Test with demo data feature available in Firehose to stream sample data to the newly created delivery stream.

  1. On the Firehose console, choose the Firehose stream you just created.
  2. Choose Test with demo data.

The delivery stream delivers the demo data to the API Gateway REST API, that is configured as the HTTP endpoint.

Monitoring the delivery stream

You can view Amazon CloudWatch metrics on the Monitoring tab. Pertinent metrics to observe are Delivery to HTTP Endpoint data freshness and HTTP Endpoint delivery success.

Conclusion

This post demonstrated how to create a delivery stream to a HTTP endpoint, which eliminates the need to develop custom applications or manage the corresponding infrastructure. Firehose provides a fully managed service that helps you reduce complexities, so you can expand and accelerate the use of data streams throughout your organization.

About the Authors

Imtiaz (Taz) Sayed is the World Wide Tech Leader for Data Analytics at AWS. He is an ardent data engineer and relishes connecting with the data-analytics community. He likes roller-coasters, good heist movies, and is undecided between “The Godfather” and “The Shawshank Redemption” as the greatest movie of all time.

Masudur Rahaman Sayem is a Specialist Solution Architect for Analytics at AWS. He is passionate about distributed systems. He also likes to read, especially the classic comic books.

Felix John is a Solutions Architect and data streaming expert at AWS, based in Germany. He focuses on supporting small and medium businesses on their cloud journey. Outside of his professional life, Felix enjoys playing Floorball and hiking in the mountains.

Audit History

Last reviewed and updated in November 2024 by Felix John | Sr. Solutions Architect</p