Containers
Kubernetes Logging powered by AWS for Fluent Bit
September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.
Centralized logging is an instrumental component of running and managing Kubernetes clusters at scale. Developers need access to logs for debugging and monitoring applications, operations teams need access for monitoring applications, and security needs access for monitoring. These teams have different requirements for processing and storage of logs. In this blog post, we will look at a solution to centralize your logs using AWS for Fluent Bit combined with Amazon CloudWatch.
AWS for Fluent Bit is a container built on Fluent Bit and is designed to be a log filter, parser, and router to various output destinations. AWS for Fluent Bit adds support for AWS services such as Amazon CloudWatch, Amazon Kinesis Data Firehose, and Amazon Kinesis Data Streams.
Before I dive into the solution, let’s look at how logs are processed by Fluent Bit and sent to the output destination. Logs are first ingested via an Input. For Kubernetes, our Input is the container log files generated by Docker from the stdout and stderr of the containers on that host. This input processes the Docker log format and ensure that the time is properly set on the log entry.
Next, logs are filtered by a set of Fluent Bit filters. This solution leverages the Kubernetes filter to enrich the log entries with Pod Labels and Annotations for easy querying in the log storage solution.
By default, the Kubernetes filter assumes the log data is in the JSON format and attempt to parse that data. For example, if our log entry was serialized from JSON to a string, we want the structured data available in our log entry. For example, if we had the following log entry, we would want the structured data to be available in our backend system. The plugin will also preserve the original entry from the application.
Input:
Output:
The parser can be customized to use custom parsers such as NGINX or Apache. By adding an annotation to the Kubernetes Pod, we can override the default JSON parser. Now, different log formats can be parsed and deserialized from their string formats into structured formats.
The Kubernetes Filter also enriches the data with Kubernetes metadata. It calls the Kubernetes API Server and query for information about that pod. This adds an additional key to the log entry called Kubernetes.
Now that we have our logs parsed and enriched with metadata we send them to our output destination. It is common to have multiple outputs for different use cases. For example, you may want all logs sent to Amazon CloudWatch so that developers can access the logs and also export them to S3 for long-term storage. Some development teams may use an ELK (Elasticsearch, Logstash, Kibana) stack. For ELK, we can leverage the Kinesis Data Firehose plugin to stream the logs to Amazon Elasticsearch and S3. This toolset is commonly used for developers to live stream logs for debugging while maintaining long-term audit requirements via S3. For more information on splitting logs to multiple outputs, see this blog post about Fluent Bit streams. We keep this example simple and just use Amazon CloudWatch Logs.
First, we need to configure the output for CloudWatch Logs. We configure Fluent Bit to send the logs to a specific log group and to create that group if it doesn’t exist.
Now that we have our Fluent Bit configuration we need to deploy our cluster and DaemonSet. First, we use the eksctl to create a new cluster with IAM Roles for Service Accounts enabled. For more information on how to do this, visit the eksctl documentation. When setting up your cluster, make sure that there is a Service Account called “alb-ingress-controller” in the “kube-system” namespace with the proper ALB Ingress Controller permissions and a “fluentbit” in the “fluentbit-system” namespace with permission to write to Amazon CloudWatch Logs. Now that we have an EKS cluster with the FluentBit service account.
Once the cluster is provisioned, we deploy our DaemonSet. To start, we create a fluentbit.yml file, which defines the ClusterRole, ClusterRoleBinding, ConfigMap, and DaemonSet used to deploy our Fluent Bit agents.
Once this is deployed, you should see logs streaming into your Amazon CloudWatch Log Group for the system containers such as CoreDNS and the AWS Node container. To show how this can be used for applications, we deploy an NGINX container and specify a custom Fluent Bit parser. Below is the demo.yml file used for our demo application.
Once this is deployed to the cluster, we wait for the ALB to be configured. You can get the ALB endpoint by running the following command:
Then use to simulate HTTP requests, like hey, to simulate load on the application and generate logs:
You should now see entries in your logs that look similar to this:
Awesome! The AWS for Fluent Bit DaemonSet is now streaming logs from our application, adding Kubernetes metadata, parsing the logs, and sending it to Amazon CloudWatch for monitoring and alerting.
Note: If you are running your containers on AWS Fargate, you need to run a separate sidecar container per Pod as Fargate doesn’t support DaemonSets.
In conclusion, this architecture can be configured to stream logs to Amazon CloudWatch, Amazon Kinesis Data Firehose, Amazon Kinesis Data Streams, and many other backends in a structured manner that can be monitored and analyzed. This enables administrators to provide access to the necessary groups within the organization.