Announcing Firelens – A New Way to Manage Container Logs

Today, the fantastic team that builds our container services at AWS have launched an excellent new tool called AWS FireLens that will make dealing with logs a whole lot easier.

Using FireLens, customers can direct container logs to storage and analytics tools without modifying deployment scripts, manually installing extra software or writing additional code. With a few configuration updates on Amazon Elastic Container Service (Amazon ECS) or AWS Fargate, you select the destination and optionally define filters to instruct FireLens to send container logs to where they are needed.

FireLens works with either Fluent Bit or Fluentd, which means that you can send logs to any destination supported by either of those open-source projects. We maintain a web page where you can see a list of AWS Partner Network products that have been reviewed by AWS Solution Architects. You can send log data or events to any of these products using FireLens.

I find the simplest way to understand FireLens is to use it, so in the rest of this blog post, I’m going to demonstrate using FireLens with a container in Amazon Elastic Container Service (Amazon ECS), forwarding the container logs on to Amazon CloudWatch.

First, I need to configure a task definition, I got an example definition from the Amazon ECS FireLens Examples on GitHub.

I replaced the AWS Identity and Access Management (IAM) roles with my own taskRoleArn and executionRoleArn IAM roles, I also added port mappings so that I could access the NGINX container from a browser.

{
	"family": "firelens-example-cloudwatch",
	"taskRoleArn": "arn:aws:iam::365489000573:role/ecsInstanceRole",
	"executionRoleArn": "arn:aws:iam::365489300073:role/ecsTaskExecutionRole",
	"containerDefinitions": [
		{
			"essential": true,
			"image": "906394416424.dkr.ecr.us-east-1.amazonaws.com/aws-for-fluent-bit:latest",
			"name": "log_router",
			"firelensConfiguration": {
				"type": "fluentbit"
			},
			"logConfiguration": {
				"logDriver": "awslogs",
				"options": {
					"awslogs-group": "firelens-container",
					"awslogs-region": "us-west-2",
					"awslogs-create-group": "true",
					"awslogs-stream-prefix": "firelens"
				}
			},
			"memoryReservation": 50
		 },
		 {
			 "essential": true,
			 "image": "nginx",
			 "name": "app",
			 "portMappings": [
				{
				  "containerPort": 80,
				  "hostPort": 80
				}
			  ],
			 "logConfiguration": {
				 "logDriver":"awsfirelens",
				 "options": {
					"Name": "cloudwatch",
					"region": "us-west-2",
					"log_group_name": "firelens-fluent-bit",
					"auto_create_group": "true",
					"log_stream_prefix": "from-fluent-bit"
				}
			},
			"memoryReservation": 100
		}
	]
}

I saved the task definition to a local folder and then used the AWS Command Line Interface (AWS CLI) to register the task definition.

aws ecs register-task-definition --cli-input-json file://cloudwatch_task_definition.json

I already have an Amazon ECS cluster set up, but if you don’t, you can learn how to do that from the Amazon ECS documentation. The command below creates a service on my Amazon ECS cluster using my newly registered task definition.

aws ecs create-service --cluster demo-cluster --service-name demo-service --task-definition firelens-example-cloudwatch --desired-count 1 --launch-type "EC2"

After logging into the Amazon Elastic Container Service (Amazon ECS) console and drilling into my service, and my tasks, I find the container definition that exposes an External Link. This IP address is exposed since I asked for the container to map port 80 of the container port to port 80 of the host port inside of the task definition.

If I go to that IP adress in a browser then the NGINX container which I used as my app, serves its default page. The NGINX container logs any requests that it receives to Stdout and so FireLens will now forward these logs on to CloudWatch. I added a little message to the URL so that when I take a look at the logs, I should be able to quickly identify this request from all the others.

I then navigated over to the Amazon CloudWatch console and drilled down into the firelens-fluent-bit log group. If you remember this is the log group name that I set up in the original task definition. Below you will notice I have several logs in my log stream and the last one is the request that I just made in the browser. If you look closely at the log, you will find that “IT WORKS” is passed in as part of the GET request.

So there we have it, I successfully set up FireLens and had it forward my container logs on to CloudWatch I could, of course, have chosen a different destination, for example, a third-party provider like Datadog or an AWS destination like Amazon Kinesis Data Firehose.

If you want to try FireLens, it is available today in all regions that support Amazon Elastic Container Service (Amazon ECS), and AWS Fargate.

Happy Logging!