How do I use a Splunk log driver with an Amazon ECS task on Fargate?

Last updated: 2020-11-12

I want to use a Splunk log driver with an Amazon Elastic Container Service (Amazon ECS) task on AWS Fargate.

Short description

You can use the Splunk log driver (also called a "logging driver") from Docker in Fargate to send your container logs to the Enterprise Splunk log collector or Splunk Cloud. For more information, see Splunk logging driver on the Docker website.

Resolution

Set up your Splunk Cloud environment to receive Fargate logs

1.    Create a Splunk Cloud account using the Free Splunk option. If you already have a Splunk Cloud account, skip to step 2. For more information, see Free Splunk on the Splunk website.

2.    Sign in to your Splunk Cloud account.

3.    In the navigation menu, choose Settings.

4.    In the DATA section, choose Data inputs.

5.    In the Local inputs section, in the Actions column, choose Add new.

Note: You create an HTTP Event Collector (HEC) token so that you can send data and application events to a Splunk deployment over HTTP and Secure HTTP (HTTPS) protocols.

6.    For Name, enter a name for your token, and then choose Next.

7.    Step through the rest of the pages in the setup wizard, and then choose Submit to create the HEC token.

Note: You can leave the default values in place.

8.    Test the new token using either curl or Docker.

Note: By default, Splunk listens on port 8088 and accepts events on the /services/collector path.

Using curl:

curl -k  https://prd-p-9l79p.splunkcloud.com:8088/services/collector -H "Authorization: Splunk splunk-token" -d '{"event": "Put this message in Splunk"}'

Note: Replace splunk-token with the HEC token that you created earlier. 

Using Docker:

docker run --publish 80:80 --log-driver=splunk --log-opt splunk-token=splunk token --log-opt splunk-url=https://prd-p-u7z1u.splunkcloud.com:8088 --log-opt splunk-insecureskipverify=true --log-opt splunk-sourcetype=docker-test --log-opt splunk-index=main  nginx

Note: Replace splunk-token with the HEC token that you created earlier. 

Configure AWS Secrets Manager to use your HEC token

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.

To store your HEC token in AWS Secrets Manager so that Fargate can use your token securely, run the following command:

aws secretsmanager create-secret --name splunk-token --secret-string token-value

Note: Replace splunk-token with your HEC token. Replace token-value with your token value.

Configure a Task Execution role with permission to access secrets

1.    Open the AWS Identity and Access Management (IAM) console.

2.    From the navigation pane, choose Roles.

3.    From the list of roles, search for and choose ecsTaskExecutionRole.

4.    Choose Add inline policy.

5.    Choose the JSON tab of the editor, and then enter the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "kms:Decrypt"
      ],
      "Resource": [
        "arn:aws:secretsmanager:<region>:<aws_account_id>:secret:<secret_name>",
        "arn:aws:kms:<region>:<aws_account_id>:key/<key_id>"
      ]
    }
  ]
}

Note: If you're using a custom AWS Key Management Service (AWS KMS) key to encrypt your secrets, you must provide the kms:Decrypt permission and Amazon Resource Name (ARN) for your KMS key.

6.    Choose Review policy.

7.    For Name, enter a name for your policy.

8.    Choose Create policy.

Configure your task definition for your Amazon ECS task on Fargate

1.    Open the Amazon ECS console.

2.    From the navigation pane, in the Amazon ECS section, choose Task Definitions.

3.    Choose Create new Task Definition.

4.    Choose Fargate, and then choose Next step.

5.    In the Configure task and container definitions section, for Task definition name, enter a name for your task definition.

6.    For Task Role, choose the role that you configured earlier.

7.    In the Task size section, for Task memory (GB) and Task CPU (vCPU), enter the appropriate values for your tasks.

8.    In the Container Definitions section, choose Add container.

9.    Choose the appropriate options for your container needs.

10.    In the STORAGE AND LOGGING section, clear the Log configuration check box. 

11.    For Log driver, select splunk.

12.    For the key and value options in Log options, enter the following:

Key: splunk-url
Value: Your splunk endpoint (For example: https://prd-p-9l79p.splunkcloud.com:8088/)

Key: plunk-insecureskipverify
Value: True

Key: splunk-token
Value: The Secret ARN that you created earlier

13.    Choose Add

Note: For more details on code and configurations for Splunk, see the log driver code on GitHub and Splunk options on the Docker website.

14.    To create your task definition, choose Create.

15.    Create an Amazon ECS service or run a task that uses the task definition that you created earlier.

Check your logs in Splunk Cloud

Be sure that your task is running before you check your logs in Splunk Cloud.

Note: By default, your logs go to splunk-index in the main index because you haven't specified splunk-index in your log configuration.

1.    Connect to your Splunk Cloud account.

2.    In the navigation menu, choose Search.

3.    Enter index="main" in the search box.

Now, you can see your logs in the Splunk log collector.

Note: You can follow the preceding steps to create a task definition in Amazon ECS for an Amazon Elastic Compute Cloud (Amazon EC2) launch type. Then, you can send your logs to the Splunk log collector using the Splunk log driver form any of your ECS tasks.  


Did this article help?


Do you need billing or technical support?