How Axrail Built an AI-Powered Automated Reply System for Users’ Customer Service Queries

By Clevester Teo, Partner Solutions Architect – AWS
By Abhijit Kalita, Sr. AI/ML Evangelist – AWS
By Kelvin Yap, Lead Architect – Axrail

Axrail

On a typical day, customer service agents communicate with many customers using various channels such as inbound and outbound calls, emails, chats, and social media messaging.

For channels like email or social media messaging, there is typically a target timeframe within which the agent needs to reply to the customer. Often, agents are overwhelmed with hundreds or thousands of emails or messages from customers with specific queries. It becomes difficult for agents to always reply promptly and within the defined target timeline.

The quality and timeliness of the email or message replies can directly impact customer satisfaction. In situations where the volume of incoming emails are beyond the capacity of the agents, timeliness and quality of reply can be impacted, which may reduce the level of customer satisfaction and impact the organization’s reputation.

In this post, we will highlight an artificial intelligence (AI)-powered solution called Smart Reply that is developed by Axrail, an AWS Advanced Tier Services Partner that provides outcome-based digital transformation through analytics and application modernization.

Solution Overview

The purpose of the Smart Reply solution is to automate repetitive tasks related to agents replying to common emails from customers.

This solution has the following features:

• It automatically classifies incoming emails into specific categories to understand the key intent of the email. Once the intent is known based on classification, it creates a templated email reply which will be shown to the agents for review. The agent can review the email reply, make changes, and send it to customers.
• The solution uses Amazon Kendra for semantic search where keywords extracted from the email are used to form the Kendra query, and the answers returned by Kendra are shown to the agents. This gives flexibility to the agents to enhance their email reply with information fetched from the semantic search engine.
• The solution does sentiment analysis of the email message so agents can prioritize the email reply or be more empathetic when replying. This can also be used to understand long-term sentiment trends and assess if trends are improving.
• It has a dashboard using Amazon QuickSight which shows the top keywords for a defined time period and a corresponding word cloud. This gives insight to management on the latest trends in customer queries, key problems customers are facing, and gaps in the FAQ page that need to be updated, for example. The dashboard can be easily customized based on a user’s needs.

AI-Powered Automated Reply System

The first step in the Smart Reply process is email classification to understand the intent of the email. Amazon Comprehend custom classifier is used for this purpose.

There are two modes that users can use in custom classification: multi-class and multi-label. Multi-class is used when the email belongs to one specific class, and multi-label is used when the email could belong to a few different classes.

For example, if an agent is classifying the emails into categories such as login issues, application freeze, and application slow, an email from a user could fall into both the “application freeze” and “application slow” categories if the user is complaining about both issues.

Amazon Comprehend outputs common machine learning (ML) metrics such as precision, recall, F1 score, and accuracy, which can be used to judge the performance of the model. Once the model is trained and performance metrics are within target, a real-time endpoint is created, which can be invoked to get inference for a new email.

With Smart Reply, agents can process the emails asynchronously and get the prediction for the email intent. In this solution, asynchronous analysis is done for custom classification.

Amazon Comprehend is also used to get the sentiment scores and category of sentiment for the emails. Sentiment analysis does not need any training dataset, as it uses pre-trained models and outputs the sentiment category in four categories: positive, negative, neutral, and mixed. The solution’s user interface (UI) shows the sentiment category against each email that is received.

Once the email intent is classified and sentiment analyzed, a templated reply is retrieved from the database corresponding to the specific intent category. The template replies are pre-curated by the organization based on best practices.

On top of the templated reply that is shown to the agent, the UI fetches the answer from the semantic search engine built using Amazon Kendra.

Kendra uses natural language understanding (NLU) to run queries that identify semantic relationships in text using models like reading comprehension that extract answers directly from customer documents.

Before the Kendra query is invoked, a set of key phrases are extracted from the email message using Amazon Comprehend. Key phrases with confidence more than a certain threshold (for example, 0.99) are appended together to form the query text. For more details on key phrases extraction using Comprehend, refer to the documentation. Kendra’s real-time query API is then invoked to get the top “x” results to show to the agent.

Reference Architecture

Now, let’s look at the detailed solution architecture.

Figure 1 – Axrail’s Smart Reply solution architecture.

1. An AWS Lambda function is triggered by Amazon EventBridge every five minutes. This function extracts and aggregates the emails from the email server, encrypts and stores them in Amazon DynamoDB.
2. With DynamoDB streams, the Lambda function automatically reacts to new records. A batching window is configured to avoid invoking the function with a small number of records. This function will run Amazon Comprehend custom classifier to analyze multiple emails in asynchronously and the sentiment analysis API to return the overall sentiment of a text.
3. Valuable insights are stored in the DynamoDB table. Amazon Athena queries the transformed data stored in Amazon Simple Storage Service (Amazon S3) buckets, and the analysis, reports, or dashboards will be shown in QuickSight. The analysis includes visual types like word clouds to show word or phrase frequency. Athena is an interactive query service that makes it easy to analyze data in S3 using standard SQL, while QuickSight is a cloud-scale business intelligence (BI) service that delivers easy-to-understand insights.
4. Smart Reply is a serverless platform built on cloud-native principles which perform at scale and provides a cost-effective, automated solution. Amazon S3 is configured to host the static resources for the front end, and Amazon CloudFront speeds up the distribution of the content by a low latency network.
5. After users are authenticated by Amazon Cognito, the JSON Web Tokens (JWT) received back will be used to authenticate against the GraphQL-based API that is built with AWS AppSync. By leveraging the AppSync resolvers, direct mappings are created between the API and DynamoDB. AppSync exposes a single endpoint without needing to worry about servers or create complex backend abstractions.
6. When an agent shift starts, they might review emails with higher prioritization, such as those with unhappy sentiment. Amazon OpenSearch Service offers keyword queries for the tabular data store in DynamoDB. Users can search for keywords like email sender or topic, sorting the email and optimize the table app via OpenSearch.
7. When agents start to formulate the reply to the customer, Amazon Kendra acts as a semantic knowledge base to provide a suggested reply draft from automated search results. The templated replies are stored in DynamoDB and the agents can choose to append or modify them with additional inputs from the Kendra search results.

Summary

Axrail understands that customer service agents spend significant time responding to customer queries and searching appropriate responses for common topics.

The Smart Reply solution helps organizations drive operational excellence by improving productivity of customer service agents. It can improve customer satisfaction by reducing the lead time for response to customers’ queries.

Smart Reply embraces AWS artificial intelligence services and other fully-managed services that feature automatic scaling, built-in high availability, and a pay-per-use billing model to increase agility and optimize cost.

.

.

Axrail – AWS Partner Spotlight

Axrail is an AWS Partner that provides outcome-based digital transformation through analytics and app modernization using cloud-native capabilities.

Contact Axrail | Partner Overview