Dilip shows you how to process
Amazon CloudFront logs using
Amazon Elasticsearch Service

dilip_cloudfront_logs_elasticsearch

How can I process Amazon CloudFront logs using Amazon Elasticsearch Service (Amazon ES)?

To build custom reports to meet specific requirements:

  1. Enable logging for your CloudFront distribution, and then deliver your CloudFront logs to an S3 bucket.
  2. Use an Amazon Elastic Compute Cloud (Amazon EC2) instance running Logstash as a client to process the CloudFront logs.
  3. Push the logs to an Amazon ES domain.
  4. Build and visualize the reports using Kibana.

1.    (Optional) If you don't have an Amazon Simple Storage Service (Amazon S3) bucket in the same Region where you will run your Elasticsearch domain, you must create one. For more information, see How Do I Create an S3 Bucket?

2.    In the Amazon CloudFront console, select your CloudFront distribution, choose Distribution Settings, and then choose Edit. For Logging, select On. In the Bucket for Logs field, choose your S3 bucket. In the Log Prefix field, type a prefix for the names of the logs.

3.    Choose Yes, Edit. Note: It might take up to 24 hours before you see log requests delivered to the S3 bucket.

4.    Create a new Amazon ES domain in the Elasticsearch Service console by following the instructions in Creating and Configuring Amazon Elasticsearch Service Domains.

5.    Launch a new Amazon EC2 instance. You'll use this EC2 instance as the Logstash client. Note: You might need to configure an AWS Identity and Access Management (IAM) role that has access to S3 (GET object) and Elasticsearch (PUT document). For more information, see Creating IAM Roles.

6.    Connect to your new EC2 instance using SSH.

7.    From the command line, run the following command to download and install the Logstash client on your EC2 instance:

wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.0.tar.gz

8.    Extract the Logstash client by using the following command:

tar xvf logstash-5.5.0.tar.gz

9.    Install the Logstash plugin for Elasticsearch by using the following command:

cd logstash-5.5.0
bin/logstash-plugin install logstash-output-amazon_es

10.    Create your cloudfront.template.json and cloudfront.conf files similar to the following: Note: You might need to edit the templates to meet your requirements.

cloudfront.template.json

#cloudfront.template.json
{
  "template": "cloudfront-logs-*",
  "mappings": {
    "logs": {
      "_source": {
        "enabled": false
      },
      "_all": {
        "enabled": false
      },
      "dynamic_templates": [
        {
          "string_fields": {
            "mapping": {
              "index": "not_analyzed",
              "type": "string"
            },
            "match_mapping_type": "string",
            "match": "*"
          }
        }
      ],
      "properties": {
        "geoip"  : {
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip" },
            "location" : { "type" : "geo_point" },
            "latitude" : { "type" : "float" },
            "longitude" : { "type" : "float" }
          }
        }
      }
    }
  }
}

cloudfront.conf

#cloudfront.conf
input {
  s3 {
    bucket => "<CLOUDFRONT_LOG_BUCKET>"
    prefix => "<CLOUDFRONT_LOG_KEY_PREFIX>"
    region => "<BUCKET_REGION_NAME>"
  }
}


filter {
  grok {
    match => { "message" => "%{DATE_EU:date}\t%{TIME:time}\t%{WORD:x_edge_location}\t(?:%{NUMBER:sc_bytes:int}|-)\t%{IPORHOST:c_ip}\t%{WORD:cs_method}\t%{HOSTNAME:cs_host}\t%{NOTSPACE:cs_uri_stem}\t%{NUMBER:sc_status:int}\t%{GREEDYDATA:referrer}\t%{GREEDYDATA:User_Agent}\t%{GREEDYDATA:cs_uri_stem}\t%{GREEDYDATA:cookies}\t%{WORD:x_edge_result_type}\t%{NOTSPACE:x_edge_request_id}\t%{HOSTNAME:x_host_header}\t%{URIPROTO:cs_protocol}\t%{INT:cs_bytes:int}\t%{GREEDYDATA:time_taken}\t%{GREEDYDATA:x_forwarded_for}\t%{GREEDYDATA:ssl_protocol}\t%{GREEDYDATA:ssl_cipher}\t%{GREEDYDATA:x_edge_response_result_type}" }
  }

  mutate {
    add_field => [ "listener_timestamp", "%{date} %{time}" ]
  }

  date {
    match => [ "listener_timestamp", "yy-MM-dd HH:mm:ss" ]
    target => "@timestamp"
  }

  geoip {
    source => "c_ip"
  }

  useragent {
    source => "User_Agent"
    target => "useragent"
  }

  mutate {
    remove_field => ["date", "time", "listener_timestamp", "cloudfront_version", "message", "cloudfront_fields", "User_Agent"]
  }
}

output {
  amazon_es {
    hosts => ["<AMAZON_ES_DOMAIN_ENDPOINT>"]
    region => "<AMAZON_ES_DOMAIN_REGION_NAME>"
    index => "cloudfront-logs-%{+YYYY.MM.dd}"
    template => "/path-to-file/cloudfront.template.json"
  }
} 

11.    Use a text editor such as vi to edit your cloudfront.conf file to reflect the following: For bucket, use the name of your S3 bucket. For prefix, use the prefix you specified for Log Prefix in step 2. For hosts, use the endpoint of your CloudFront distribution. For region, use the Region your resources are in. For template, use the directory your CloudFront template is in.

12.    Save the changes you made to cloudfront.conf.

13.    Run Logstash with the -f option and specify cloudfront.conf as the configuration file. For more information, see Command-Line Flags.

In a few minutes, Logstash will publish documents to the Elasticsearch domain you specified. To be sure that the documents are published successfully, open the Amazon ES console, select your ES domain, and then check the Indices tab.

You can now use Kibana to create custom reports and visualizations for your logs. For more information, see Kibana and Logstash.

Note: You might need to configure an access policy to be sure that Kibana can access the logs stored in your Amazon ES domain.


Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center

Published: 2018-01-09

Updated: 2018-06-20