AWS Contact Center

Use Amazon Connect data in real time with Elasticsearch and Kibana

In this blog post, we demonstrate how you use Amazon Elasticsearch Service (Amazon ES) and Kibana for real-time analytics for your Amazon Connect contact center. You can monitor your contact center performance to improve a variety of service metrics, such as call times, service level, efficiency, agent performance, and customer satisfaction.

We also review a solution to process contact trace records (CTRs), agent event streams, and contact flow logs from Amazon CloudWatch using Amazon ES, and build visualizations for near real-time interaction using Kibana. Elasticsearch is an open-source, distributed search, and analytics engine used for log analytics and full text search. Kibana is a data aggregation and visualization tool. Kibana enables you to explore, visualize, analyze, and discover data in real time with Amazon ES.

Amazon Connect provides contact flow logs for real time details about events in interactions with your users. Contact flows define your customers’ experience when they contact you, and define interactions such as prompts, customer inputs, and queue transfers.

Additionally, Amazon Connect can stream CTRs to an Amazon Kinesis Data Firehose, and agent events to an Amazon Kinesis Data Streams to export data for analysis. Contact trace records capture the events, attributes, queue, and agent interactions with a contact in your Amazon Connect instance. Agent events record agent activity within your Amazon Connect instance such as login, logout, and status change.

Solution overview

The following diagram illustrates the architecture for processing contact flow logs, CTR, and agent events from Amazon Connect and visualizing data using Amazon ES and Kibana.

Architecture diagram showing the data flow betwen AWS services for data analytics

Architecture diagram for the contact center analytics solution

In this solution, Amazon Connect is configured to send contact flow logs to Amazon CloudWatch. CloudWatch is a monitoring service that enables you to track metrics, collect log files, and set alarms based on monitored events. We created a CloudWatch Logs subscription filter to invoke an AWS Lambda function to write contact flow logs data to the Amazon ES, and copy the data in Amazon Simple Storage Service (Amazon S3) for back-up storage and further analysis.

Additionally, Amazon Connect is configured to stream CTRs to an Amazon Kinesis Data Firehose and agent event logs to an Amazon Kinesis Data Stream. CTRs are exported to the stream in JSON format and contain events, attributes, queues, and agent interaction data for contacts. Agent events are similarly exported in JSON format and contain agent activity details such as agent login, logout, or status changes.

Kinesis Data Firehose writes CTR data to Amazon ES and copies the data to Amazon S3 for back-up storage and further analysis. A separate Kinesis Data Firehose is configured to capture agent events data from the agent events Kinesis Data Stream and send data to Amazon ES. The raw agent event data is similarly copied to Amazon S3 for back-up storage and further analysis.

Once contact flow logs, CTRs, and agent event data are written to Amazon ES, we use Kibana to explore and interact with data in near real time. Contact flow logs and agent activity data are available in real time, whereas CTR data is written to Amazon ES within 80-100 seconds after the call ends.

Configure the solution

Before creating this solution, you need to create an Amazon Connect instance and enable contact flow logs. When you enable contact flow logs, make a note of the CloudWatch Logs location displayed for Your logs will be stored here, such as aws/connect/instancealias. You will use this value to configure the AWS CloudFormation template in the next step.

Once you have created your Amazon Connect instance, you can log in to the AWS Management Console and deploy this solution using an AWS CloudFormation template we created for this article. This template creates the necessary IAM service roles, an AWS Lambda function, an Amazon Kinesis Data Stream, two Kinesis Data Firehose streams, an S3 bucket, and an Amazon ES domain.

To update the CloudFormation template

    1. Open the CloudFormation template:
      • If your Amazon Connect instance is in the US East (N. Virginia) Region, use this template.
      • If your Amazon Connect instance is in the US West (Oregon) Region, use this template.
    2. On the Select Template page, choose Next.
    3. On the Specify Details page, change the values for the following parameters to values for your environment:
      • ContactFlowLogGroup—Specify the name of the CloudWatch Logs group that you made a note of when you enabled contact flow logging.
      • ElasticsearchName—The name of domain created by the CloudFormation template.

      Image showing the Specify Details page for creating the stack using the CloudFormation template provided for this article.

      Specify Details page for creating a stack in CloudFormation

  1. Choose Next, and then on the Options page, leave all values and choose Next.
  2. On the Review page, change the following settings:
    • Select the checkbox for I acknowledge that AWS CloudFormation might create IAM resources.
    • Choose Create Change Set.
    • Choose Execute.
      Wait for the stack to finish being created before proceeding to the next section. When complete, the status column displays CREATE_COMPLETE.

Enable data streaming in Amazon Connect

After the CloudFormation stack is created, configure your Amazon Connect instance to stream contact trace records and agent events to Kinesis Data Streams. The Kinesis streams are created by the CloudFormation template.

To enable data streaming from Amazon Connect

  1. From the Amazon Connect console page, choose the instance alias you are using for this solution.
  2. Choose Data streaming and select Enable data streaming.
  3. Under Contact Trace Records, select Kinesis Firehose and choose the stream name that contains AmznConnectCtrDeliveryStream from the drop-down list.
  4. Under Agent Events, select the stream name that contains AgentKinesisStream from the list.
Image showing the Data streaming settings page in Amazon Connect with the streams created for this demo selected

Data streaming settings page in Amazon Connect

Once data streaming is enabled, generate data by making or receiving calls using the Amazon Connect Contact Control Panel (CCP). A few minutes after making or receiving calls, you should see data in your S3 bucket and Amazon ES domain.

Create Visualizations in Kibana

After data is loaded to Amazon ES, you can create visualizations in Kibana. Before we can start creating visualizations, we first need to grant access to the Elasticsearch domain so that we can access Kibana. The CloudFormation template included an access policy for the domain that allows access from a single IP address. For this article, we replaced the IP address in the policy with the public IP address for the computer we’re using. You can also use your own access policies, or take advantage of support for Cognito authentication in Kibana. For more information, see Elasticsearch access policies and Amazon Cognito Authentication for Kibana.

Update the access policy for Kibana

  1. Open the Amazon ES console.
  2. Choose the domain, amznconnect-elk-dev, which is the domain created by the CloudFormation template.
  3. Choose Modify access policy.
  4. For Set the domain access policy to, select the Allow access to the domain from specific IP(s) template.
  5. Enter the public IP address for the computer you are using, then choose OK.  You can search for “my IP address” if you’re not sure what it is.
  6. Choose Submit, then wait for the Domain status to show Active.
  7. On the Overview tab, choose the link next to Kibana.

Once you have loaded Kibana, you need to create index patterns before you can create visualizations.

Create an index pattern for agent events in Kibana

    1. From the Kibana left-hand navigation menu, choose Management.
    2. From the Management page, choose Index Patterns.
    3. For Index pattern, type agent-logs-* and choose Next step.
Image showing the Kbana Management settings page to define an index pattern

Create an index pattern in the Kibana Management settings page

    1. For Time Filter field name, select EventTimestamp and choose Create index pattern.
Image of the Kibana Management settins for creating an index pattern step 2

Kibana Management settings for creating an index pattern step 2 of creating an index pattern.

After creating agent events index pattern, create visualizations on the log data. In this example, I create a vertical bar chart to display the number of agents by status.

To create a visualization

  1. From the left-hand navigation menu, choose Visualize.
  2. Choose Create a visualization.
  3. Under Basic Charts, choose Vertical Bar.
  4. Select agent-logs-* from Select Index.
  5. Under Metrics, expand Y-Axis and provide the following parameters:
    • Aggregation: Unique Count
    • Field: CurrentAgentSnapshot.Configuration.Username.keyword
    • Custom Label: Agent
  6. Under Buckets, choose X-Axis and provide the following parameters:
    • Aggregation: Terms
    • Field: CurrentAgentSnapshot.AgentStatus.Name.keyword
    • Order By: metric: Agents
    • Size: 50
    • Custom Label: Status
  7. Choose Add a filter and provide the following parameter:
      • Filter: EventTimestamp is between now-10s and now.

    Setting the filter for the EventTimestamp to now-10s in Kibana

    Setting the filter for EventTimestamp in Kibana

  8. Choose Save, and then choose Play to apply the changes.
  9. From the top ribbon menu, choose Save.
  10. Give your visualization a name and choose Save.

This visualization displays the number of agents in your Amazon Connect instance and their latest status. Agent statuses are current to the most recent 10 seconds for a near real-time view of agent availability.

Image showing a chart in Kibana that displays agent count by status

Chart showing agent count by status in Kibana

Next, create a visualization using CTRs. In order to create visualizations from CTRs, you need to create an index pattern for CTR data.

To create an index pattern for contact trace records

  1. From the left-hand navigation menu, choose Management.
  2. From the Kibana Management page, choose Index Patterns.
  3. From Index Patterns, choose +Create Index Pattern.
  4. For Index pattern, type ctr-logs-* and choose Next step.
  5. For Time Filter field name, select LastUpdateTimestamp and choose Create index pattern.

After creating the CTR index pattern, create visualizations on the data. In this example, we create a data table to display the number of calls received per day.

To create a visualization for CTR data

  1. From the left-hand navigation menu, choose Visualize.
    Note: If you see the agent events visualization from the previous steps, choose Visualize again to begin with an empty slate.
  2. Click the “+” symbol to create a new visualization.
  3. Under Data, choose Data Table.
  4. Select ctr-logs-* from Select Index.
  5. Under Metrics, expand Metrics and provide the following parameters:
    • Aggregation: Unique Count
    • Field: ContactId.keyword
    • Custom Label: Calls
  6. Under Buckets, choose Split Rows and provide the following parameters:
    • Aggregation: Date Histogram
    • Field: LastUpdateTimestamp
    • Interval: Daily
    • Custom Label: Date
  7. To apply changes, choose the Play button.
  8. From the top ribbon menu, choose Save.
  9. Give your visualization a name and choose Save.

This visualization displays the number of calls received by Amazon Connect per day.

Chart in Kibana that shows the number of calls per day in the Amazon Connect instance

Kibana chart showing the number of calls per day

You can also create visualizations and interact with data from contact flow logs. In order to create visualizations from contact flow logs, you need to create an index pattern for contact flow log data.

To create an index pattern for Contact flow logs

  1. From the left-hand navigation menu, choose Management.
  2. From the Kibana Management page, choose Index Patterns.
  3. From Index Patterns, choose +Create Index Pattern.
  4. For Index pattern, type cwl-* and choose Next step.
  5. For Time Filter field name, select Timestamp and choose Create Index pattern.

You can build multiple visualizations and add them to an interactive dashboard from the left-hand navigation dashboard menu, such as the following:

Kibana dahsboard showing the visualizations created for this article

Kibana dashboard showing the visualizations created for this article

Conclusion

The Amazon Connect service provides metrics for tracking CTRs, agent events, and Contact flows. When integrated with AWS Analytics services such as Amazon Kinesis Data Streams and Amazon Elasticsearch Service, it allows you to interact with and explore data in real time, monitor the operations and productivity of your customer contact center, and enables real-time decision making to improve agent and queue performance.