How do I send my container logs to multiple destinations in Amazon ECS on AWS Fargate?

Last updated: 2020-08-27

I want to forward the logs of my application container that runs on AWS Fargate to multiple destinations. These destinations could be Amazon CloudWatch, Amazon Kinesis Data Firehose, or Splunk.

Short description

An Amazon Elastic Container Service (Amazon ECS) task definition allows you to specify only a single log configuration object for a given container, which means that you can forward logs to a single destination only. To forward logs to multiple destinations in Amazon ECS on Fargate, you can use FireLens.

Note: FireLens works with both Fluent Bit and Fluentd log forwarders. The following resolution considers Fluent Bit because Fluent Bit is more resource-efficient than Fluentd.

Consider the following:

  • FireLens uses the key-value pairs specified as options in the logConfiguration object from the ECS task definition to generate the Fluent Bit output definition. The destination where the logs are routed is specified in the [OUTPUT] definition section of a Fluent Bit configuration file. For more information, see Output on the Fluent Bit website.
  • FireLens creates a configuration file on your behalf, but you can also specify a custom configuration file. You can host this configuration file in either Amazon Simple Storage Service (Amazon S3), or create a custom Fluent Bit Docker image with the custom output configuration file added to it.
  • If you're using Amazon ECS on Fargate, then you can't pull a configuration file from Amazon S3. Instead, you must create a custom Docker image with the configuration file.

Resolution

Create AWS Identity and Access Management (IAM) permissions

Create IAM permissions to allow your task role to route your logs to different destinations. For example, if your destination is Kinesis Data Firehose, then you must give the task permission to call the firehose:PutRecordBatch API.

Note: Fluent Bit supports several plugins as log destinations. Permissions required by destinations like CloudWatch and Kinesis streams include logs:CreateLogGroup, logs:CreateLogStream, logs:DescribeLogStreams, logs:PutLogEvents, and kinesis:PutRecords.

Create a Fluent Bit Docker image with a custom output configuration file.

1.    Create a custom Fluent Bit configuration file called logDestinations.conf with your choice of [OUTPUT] definitions defined in it. For example, the following configuration file includes configurations defined for CloudWatch, Kinesis Data Firehose, and Splunk.

[OUTPUT]
    Name                firehose
    Match               YourContainerName*
    region              us-west-2
    delivery_stream     nginx-stream  
[OUTPUT]
    Name                cloudwatch
    Match               YourContainerName*
    region              us-east-1
    log_group_name      firelens-nginx-container
    log_stream_prefix   from-fluent-bit
    auto_create_group   true   
[OUTPUT]
    Name                splunk
    Match               <ContainerName>*
    Host                127.0.0.1
    Splunk_Token        xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
    Splunk_Send_Raw     On

Note: Different destinations require different fields to be specified in the [OUTPUT] definition. For examples, see Amazon ECS FireLens examples.

2.    Create a Docker image with a custom Fluent Bit output configuration file using the following Dockerfile example:

FROM amazon/aws-for-fluent-bit:latest
ADD logDestinations.conf /logDestinations.conf

Note: For more information, see Dockerfile reference on the Docker website.

3.    To create the custom fluent-bit Docker image using the Dockerfile that you created in step 2, run the following command:

docker build -t custom-fluent-bit:latest .

Important: Be sure to run the docker build command in the same location as the Dockerfile.

4.    To confirm that the Docker image is available to Amazon ECS, push your Docker image to Amazon Elastic Container Registry (Amazon ECR) or your own Docker registry. For example, to push a local Docker image to Amazon ECR, run the following command:

docker push aws_account_id.dkr.ecr.region.amazonaws.com/custom-fluent-bit:latest

5.    In your task definition (TaskDefinition), update the options for your FireLens configuration (firelensConfiguration). For example:

{
   "containerDefinitions":[
      {
         "essential":true,
         "image":"aws_account_id.dkr.ecr.region.amazonaws.com/custom-fluent-bit:latest",
         "name":"log_router",
         "firelensConfiguration":{
            "type":"fluentbit",
            "options":{
               "config-file-type":"file",
               "config-file-value":"/logDestinations.conf"
            }
         }
      }
   ]
}

When you update the options for your FireLens configuration, consider the following:

To specify a custom configuration file, you must include the config-file-type and config-file-value options in your FireLens configuration file. You can only set these options when you create a task definition with the AWS Command Line Interface (AWS CLI).

You must modify the image property in the containerDefinition section of your configuration to reflect a valid Amazon ECR image location. You can specify images in Amazon ECR repositories by using the full registry/repository:tag naming convention. For example:

aws_account_id.dkr.ecr.region.amazonaws.com/custom-fluent-bit:latest

To use other repositories, see the image property of the task definition.


Did this article help?


Do you need billing or technical support?