Containers
Shipping logs to third-parties with Kinesis Data Firehose and Fluent Bit for Amazon EKS on AWS Fargate
AWS Fargate is a technology that provides on-demand capacity for running pods on EKS clusters. Fargate provides a more hands-off experience, helping you run container applications without needing to manage the EC2 instances underneath. AWS Fargate runs each Kubernetes pod in its own isolated security boundary. This means it has a slightly different operating model than Kubernetes pods that run on EC2 instances.
In this post, we’ll demonstrate how you can enjoy the convenience of AWS Fargate while meeting compliance for centralized logging by routing application logs from containers running within Amazon EKS on AWS Fargate to Splunk.
To retrieve application metrics, Amazon CloudWatch Container Insights for Amazon EKS Fargate using AWS Distro for OpenTelemetry lets you view the CPU and memory use of EKS Fargate Pods in Amazon CloudWatch. For application logs, AWS Fargate provides a fully managed, built-in log router based on Fluent Bit, so no additional components need to be defined in the workload manifest.
To use the Fluent Bit log-router, create a Kubernetes ConfigMap and define FluentBit Filters and Parsers. Then define either CloudWatch, Amazon OpenSearch, Amazon Kinesis Data Firehouse, or Kinesis Streams as the supported destination for the output.
Many organizations also use third-party logging and observability solutions, such as Splunk, Datadog or New Relic. In its current state, AWS Fargate’s log router doesn’t directly support these instances, so instead, you can use Amazon Kinesis Data Firehose to create a logging pipeline. From the log router, AWS Fargate can automatically send log data to Kinesis Data Firehose before streaming it to a third-party destination.
The above architecture follows this diagram:
Overview
This walkthrough can be broken down into three high-level steps.
- Configure a Splunk deployment
- Create the Kinesis Firehose delivery stream
- Configure the EKS Cluster, Fargate Profile, and Fluent Bit ConfigMap
Prerequisites
- An AWS account with the relevant permissions to create an EKS cluster and Kinesis Data Firehose
- Installation of AWS CLI, kubectl, eksctl, and Git
Configure the Splunk deployment
- Deploy a Splunk Server. If you are going to install Splunk on an EC2 Linux instance, refer to the Splunk Manual – Install on Linux. This step might not be required if an existing Splunk deployment is available.
- The Splunk platform must be publicly accessible. To ensure Kinesis Data Firehose can reach the Splunk deployment, ensure that the following IP Ranges are able to reach your Splunk Deployment and that port 8088 is open on the Splunk deployment. The latter will allow Splunk to receive data from Kinesis Data Firehose.
- The Splunk endpoint needs to be secured with a TLS Certificate. Consider Let’s Encrypt to generate a fully trusted certificate, since self-signed certificates are not supported. In the example server.conf below,
/opt/splunk/etc/auth/mycerts/myServerCert.pem
is the file containing both public certificate and private keys. - Following the Splunk documentation, create a HTTP Event collector, and ensure that you’ve selected Indexer acknowledgement. After creating this, copy the token value , for example,
BD274822-96AA-4DA6-90EC-18940FB2414C
, into your workstation’s clipboard.
- To test if the Splunk deployment is ready to receive data, replace
BD274822-96AA-4DA6-90EC-18940FB2414C
with the token you just copied. Adjust the value of theX-Splunk-Request-Channel
to any random UUID, as we use that for testing only.
Create a Kinesis Firehose delivery stream
- On the Amazon Kinesis Console page, under Delivery Streams click Create delivery stream.
- When choosing the Source field, select Direct PUT. For the Destination field, choose Splunk.
- Enter the Splunk cluster endpoint (such as http://mysplunkhost.com:8088 ) and the value of Authentication Token (for example,
BD274822-96AA-4DA6-90EC-18940FB2414C
). - Leave the other fields as default values, and select Create delivery stream.
You can now navigate into the new delivery stream you just created. Select Start sending demo data, and the testing data should appear in your Splunk environment. If no logs appear in your Splunk environment, you can troubleshoot further by viewing the CloudWatch metrics, Delivery to Splunk success and Destination error logs.
Configure the EKS cluster, Fargate profile, and Fluent Bit
The final part of setup is to run an EKS Cluster with Fargate, and configure the built-in log router to send logs to Kinesis Data Firehose.
1. Create an eksctl YAML manifest that defines an EKS cluster with a Fargate profile:
2. Run the eksctl
command to create the EKS cluster. Please note that it will take a few minutes for the EKS Cluster and Fargate profiles to be completed.
3. Create the dedicated aws-observability
namespace and the ConfigMap for Fluent Bit. Replace PUT-SPK-k0itr
to the value as the name of the Kinesis Firehose you created in a previous part. Create the Fluent Bit configuration using the following command:
4. Provide the relevant IAM permissions for the log router running on AWS Fargate to write to the Kinesis Data Firehose by attaching an IAM policy to the pod execution role. In doing so, the underlying Fargate resource will gain permission to write to the Kinesis Data Firehouse instead of the workload running in the Kubernetes Pod.
First define a policy called allow_kinesis_put_permission.json
. Replace the ARN with the relevant Firehouse Delivery Stream ARN when you create the file.
Deploy sample applications
To generate logs and test that the log pipeline is working, deploy a Nginx pod that’s running on AWS Fargate.
Once the pod is running, retrieve the logs using the kubectl command to compare to the logs within Splunk.
Ideally, the same log lines will appear in Splunk, and they’ll include additional metadata enriched by Kubernetes filter.
Cleaning up
To avoid incurring future charges, delete all resources, including the Kinesis Firehose and EKS cluster using the following commands:
Conclusion
In this post, we demonstrated how to send logs from Fargate to a third-party logging solution with Kinesis Data Firehose, using Splunk as an end target.
With this architecture, you can enjoy the ease of Fargate while meeting a requirement for centralized logging.
To learn more about EKS Fargate logging, Kinesis Data Firehose, and Splunk, please refer to the links below: