Containers
How to capture application logs when using Amazon EKS on AWS Fargate
Update 12/05/20: EKS on Fargate now supports capturing applications logs natively. Please see this blog post for details.
Amazon Elastic Kubernetes Service (Amazon EKS) now allows you to run your applications on AWS Fargate. You can run Kubernetes pods without having to provision and manage EC2 instances. Because Fargate runs every pod in VM-isolated environment, the concept of daemonsets currently doesn’t exist in Fargate. Therefore to capture application logs when using Fargate, you need to reconsider how and where your application emits logs. This tutorial shows how to capture and ship application logs for pods running on Fargate.
Kubernetes logging architecture
According to the Twelve-Factor App manifesto, which provides the gold standard for architecting modern applications, containerized applications should output their logs to stdout and stderr. This is also considered best practice in Kubernetes and cluster level log collection systems are built on this premise.
The Kubernetes logging architecture defines three distinct levels:
- Basic level logging: the ability to grab pods log using kubectl (e.g.
kubectl logs myapp– wheremyappis a pod running in my cluster) - Node level logging: The container engine captures logs from the application’s
stdoutandstderr, and writes them to a log file. - Cluster level logging: Building upon node level logging; a log capturing agent runs on each node. The agent collects logs on the local filesystem and sends them to a centralized logging destination like Elasticsearch or CloudWatch. The agent collects two types of logs:
- Container logs captured by the container engine on the node.
- System logs.
Kubernetes, by itself, doesn’t provide a native solution to collect and store logs. It configures the container runtime to save logs in JSON format on the local filesystem. Container runtime – like Docker – redirects container’s stdout and stderr streams to a logging driver. In Kubernetes, container logs are written to /var/log/pods/*.log on the node. Kubelet and container runtime write their own logs to /var/logs or to journald, in operating systems with systemd. Then cluster-wide log collector systems like Fluentd can tail these log files on the node and ship logs for retention. These log collector systems usually run as DaemonSets on worker nodes. But running DaemonSets is not the only way to aggregate logs in Kubernetes.
Shipping container logs to a centralized log aggregation system
There are three common approaches for capturing logs in Kubernetes:
- Node level agent, like a Fluentd daemonset. This is the recommended pattern.
- Sidecar container, like a Fluentd sidecar container.
- Directly writing to log collection system. In this approach, the application is responsible for shipping the logs. This is the least recommended option because you will have to include the log aggregation system’s SDK in your application code instead of reusing community build solutions like Fluentd. This pattern also disobeys the principle of separation of concerns, according to which, logging implementation should be independent of the application. Doing so allows you to change the logging infrastructure without impacting or changing your application.
For pods running on Fargate, you need to use the sidecar pattern. You can run a Fluentd (or Fluent Bit) sidecar container to capture logs produced by your applications. This option requires that the application writes logs to filesystem instead of stdout or stderr. A consequence of this approach is that you will not be able use kubectl logs to view container logs. To make logs appear in kubectl logs, you can write application logs to both stdout and filesystem simultaneously. In the tutorial below, I am using tee write to file and stdout.
We understand that, if your application logs to stdout/stderr, you may need to make changes to your applications to capture cluster level logs in EKS on Fargate. We have heard from customers that this is undesirable and we are working to create a solution that doesn’t need application refactoring. Until then, if you want to run your workloads without managing EC2 instances, you can use the sidecar pattern to capture cluster level application logs. Note that, if you only need to capture basic logging at the pod-level, kubectl logs will do without any application refactoring.
Pods on Fargate get 20GB of ephemeral storage, which is available to all the containers that belong to a pod. You can configure your application to write logs to the local filesystem and instruct Fluentd to watch the log directory (or file). Fluentd will read events from the tail of log files and send the events to a destination like CloudWatch for storage. Ensure that you rotate logs regularly to prevent logs from usurping the entire volume.
Tutorial
The demo container produces logs to /var/log/containers/application.log. Fluentd is configured to watch /var/log/containers and send log events to CloudWatch. The pod also runs a logrotate sidecar container that ensures the container logs don’t deplete the disk space. In the example, cron triggers logrotate every 15 minutes; you can customize the logrotate behavior using environment variables.
You will need the latest version of eksctl to create the cluster and Fargate profile.
The command below will create an EKS cluster. All pods in kube-system and default namespaces will run on Fargate. There will be no EC2 nodes in this cluster.
eksctl create cluster \
--name eksfargate-logging-demo \
--fargateCreate a new namespace that will run the demo application.
kubectl create namespace logdemo
Create a new Fargate profile for logdemo namespace. This tells EKS to run the pods in logdemo namespace on Fargate.
eksctl create fargateprofile \
--namespace logdemo --cluster \
eksfargate-logging-demo \
--name logdemo
Create an IAM OIDC identity provider for the cluster.
eksctl utils associate-iam-oidc-provider \
--cluster=eksfargate-logging-demo \
--approve
Create an IAM role and a Kubernetes service account for Fluentd. This role permits Fluentd container to write log events to CloudWatch.
eksctl create iamserviceaccount \
--name logdemo-sa \
--namespace logdemo \
--cluster eksfargate-logging-demo \
--attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
--approve
You can review the service account created in the previous step.
kubectl get serviceaccount -n logdemo
NAME SECRETS AGE
default 1 50m
logdemo-sa 1 1mCreate a manifest for Fluentd ClusterRole,RoleBinding, and ConfigMap.
cat <<EOF > fluentd_rbac.yaml
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: fluentd-spring
rules:
- apiGroups: [""]
resources:
- namespaces
- pods
- pods/logs
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: fluentd-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: fluentd-spring
subjects:
- kind: ServiceAccount
name: logdemo-sa
namespace: logdemo
---
apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd-spring-config
namespace: logdemo
labels:
k8s-app: fluentd-cloudwatch
data:
fluent.conf: |
@include containers.conf
<match fluent.**>
@type null
</match>
containers.conf: |
<source>
@type tail
@id in_tail_container_logs
@label @containers
path /var/log/containers/*.log
pos_file /usr/local/fluentd-containers.log.pos
tag *
read_from_head true
<parse>
@type none
# @type json
# time_format %Y-%m-%dT%H:%M:%S.%NZ
</parse>
</source>
<label @containers>
<filter **>
@type kubernetes_metadata
@id filter_kube_metadata
</filter>
<filter **>
@type record_transformer
@id filter_containers_stream_transformer
<record>
stream_name springlogs #
</record>
</filter>
<filter **>
@type concat
key log
multiline_start_regexp /^\S/
separator ""
flush_interval 5
timeout_label @NORMAL
</filter>
<match **>
@type relabel
@label @NORMAL
</match>
</label>
<label @NORMAL>
<match **>
@type cloudwatch_logs
@id out_cloudwatch_logs_containers
region "#{ENV.fetch('REGION')}"
log_group_name "/aws/containerinsights/#{ENV.fetch('CLUSTER_NAME')}/springapp"
log_stream_name_key stream_name
remove_log_stream_name_key true
auto_create_stream true
<buffer>
flush_interval 5
chunk_limit_size 2m
queued_chunks_limit_size 32
retry_forever true
</buffer>
</match>
</label>
EOFApply the manifest.
kubectl apply -f fluentd_rbac.yamlThe result should look like this:
clusterrole.rbac.authorization.k8s.io/fluentd-spring created
clusterrolebinding.rbac.authorization.k8s.io/fluentd-role-binding created
configmap/fluentd-spring-config createdCreate a manifest for the sample application. The pod contains an initContainer that copies the Fluentd ConfigMap and copies it to /fluentd/etc/. This directory is mounted in the Fluentd container.
cat <<EOF > application_deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: fargate-log-gen
namespace: logdemo
spec:
replicas: 1
selector:
matchLabels:
app: appf
k8s-app: fluentd-cloudwatch
template:
metadata:
labels:
app: appf
k8s-app: fluentd-cloudwatch
annotations:
iam.amazonaws.com/role: logdemo-sa
spec:
volumes:
- name: fluentdconf
configMap:
name: fluentd-spring-config
- name: app-logs
emptyDir: {}
serviceAccount: logdemo-sa
serviceAccountName: logdemo-sa
containers:
- name: app
image: busybox
command: ['sh', '-c']
args:
- >
while true;
do echo "Time: $(date) $(cat /dev/urandom | tr -dc a-zA-Z0-9 | fold -w 1024 | head -n 1)" | tee -a /var/log/containers/application.log;
sleep 1;
done;
imagePullPolicy: Always
volumeMounts:
- mountPath: /var/log/containers
name: app-logs
resources:
requests:
cpu: 200m
memory: 0.5Gi
limits:
cpu: 400m
memory: 1Gi
securityContext:
privileged: false
readOnlyRootFilesystem: false
allowPrivilegeEscalation: false
- name: logrotate
image: realz/logrotate
volumeMounts:
- mountPath: /var/log/containers
name: app-logs
env:
- name: CRON_EXPR
value: "*/15 * * * *"
- name: LOGROTATE_LOGFILES
value: "/var/log/containers/*.log"
- name: LOGROTATE_FILESIZE
value: "50M"
- name: LOGROTATE_FILENUM
value: "5"
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:v1.9.3-debian-cloudwatch-1.0
env:
- name: REGION
value: us-east-2
- name: AWS_REGION
value: us-east-2
- name: CLUSTER_NAME
value: eksfargate-logging-demo
- name: CI_VERSION
value: "k8s/1.0.1"
resources:
limits:
memory: 400Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: fluentdconf
mountPath: /fluentd/etc
- name: app-logs
mountPath: /var/log/containers
EOF? Edit the value of REGION, AWS_REGION, and CLUSTER_NAME to match your environment.
Deploy the sample application with the command.
kubectl apply -f application_deployment.yaml
You can see the written logs using the AWS CLI or CloudWatch console. Using AWS CLI:
aws logs get-log-events \
--log-group-name "/aws/containerinsights/eksfargate-logging-demo/springapp" \
--log-stream-name "springlogs" You should see log events generated by the demo container:
{
"timestamp": 1593026958792,
"message": "{\"message\":\"Time: Wed Jun 24 19:29:18 UTC 2020=0tfx\"
}",To view in the CloudWatch console, search for log group “/aws/containerinsights/eksfargate-logging-demo/springapp.”
Conclusion
If you want to use Fargate to run your pods, you will need to use the sidecar pattern to capture application logs. Consider writing to stdout and file simultaneously so you can view logs using kubectl. You can still use the daemonset pattern for applications running on EC2 nodes.
We are working to provide a native solution for application logging for EKS on Fargate. The FireLens on EKS Fargate issue on the AWS Containers Roadmap includes the proposal we’re considering. Leave us a comment, we would love to hear your feedback.