AWS Machine Learning Blog
Automatically scale Amazon Kendra query capacity units with Amazon EventBridge and AWS Lambda
Data is proliferating inside the enterprise and employees are using more applications than ever before to get their jobs done, in fact according to Okta Inc., the number of software apps deployed by large firms across all industries world-wide has increased 68%, reaching an average of 129 apps per company.
As employees continue to self-serve and the number of applications they use grows, so will the likelihood that critical business information will remain hard to find or get lost between systems, negatively impacting workforce productivity and operating costs.
Amazon Kendra is an intelligent search service powered by machine learning (ML). Unlike conventional search technologies, Amazon Kendra reimagines search by unifying unstructured data across multiple data sources as part of a single searchable index. It’s deep learning and natural language processing capabilities then make it easy for you to get relevant answers when you need them.
Amazon Kendra Enterprise Edition includes storage capacity for 100,000 documents (30 GB of storage) and a query capacity of 8,000 queries per day (0.1 queries per second), and allows you to adjust index capacity by increasing or decreasing your query and storage capacity units as needed.
However, usage patterns and business needs are not always predictable. In this post we’ll demonstrate how you can automatically scale your Amazon Kendra index based on a time schedule using Amazon EventBridge and AWS Lambda. By doing this you can increase capacity for peak usage, avoid service throttling, maintain flexibility, and control costs.
Solution overview
Amazon Kendra provides a dashboard that allows you to evaluate the average number of queries per second for your index. With this information, you can estimate the number of additional capacity units your workload requires at a specific point in time.
For example, the following graph shows that during business hours, a surge occurs in the average queries per second, but after hours, the number of queries reduces. We base our solution on this pattern to set up an EventBridge scheduled event that triggers the automatic scaling Lambda function.
The following diagram illustrates our architecture.
You can deploy the solution into your account two different ways:
- Deploy an AWS Serverless Application Model (AWS SAM) template:
- Clone the project from the aws-samples repository on GitHub and follow the instructions.
- Create the resources by using the AWS Management Console. In this post, we walk you through the following steps:
- Set up the Lambda function for scaling
- Configure permissions for the function
- Test the function
- Set up an EventBridge scheduled event
Set up the Lambda function
To create the Lambda function that we use for scaling, we create a function using the Python runtime (for this post, we use the Python 3.8 runtime).
Use the following code as the content of your lambda_function.py
code:
You must modify the following variables to match with your environment:
- INDEX_ID – The ID for your index; you can check it on the Amazon Kendra console.
- DEFAULT_UNITS – The number of query processing units that your Amazon Kendra Enterprise Edition requires to operate at minimum capacity. This number can range from 0–100 (you can request more capacity). 0 represents that no extra capacity units are provisioned to your Amazon Kendra Enterprise Edition index, which leaves it with a default capacity of 0.1 queries per second.
- ADDITIONAL_UNITS – The number of query capacity units you require at those times where additional capacity is required. This value can range from 1–100 (you can request additional capacity).
Configure function permissions
To query the status of your index and to modify the number of query capacity units, you need to attach a policy to your Lambda function AWS Identity and Access Management (IAM) execution role with those permissions.
- On the Lambda console, navigate to your function.
- On the Permissions tab, choose the execution role.
The IAM console opens automatically.
- On the Permissions tab, choose Attach policies.
- Choose Create policy.
A new tab opens.
- On the JSON tab, add the following content (make sure to provide your account and user information):
- Choose Next: Tags.
- Choose Next: Review.
- For Name, enter a policy name (for this post, we use
AmazonKendra_UpdateIndex
). - Choose Create policy.
- On the Attach permissions page, choose the refresh icon.
- Filter to find the policy you created.
- Select the policy and choose Attach policy.
Test the function
You can test your Lambda function by running a test event. For more information, see Invoke the Lambda function.
- On the Lambda console, navigate to your function.
- Create a new test event by choosing Test.
- Select Create new test event.
- For Event template, because your function doesn’t require any input from the event, you can choose the
hello-world
event template.
- Choose Create.
- Choose Test.
On the Lambda function logs, you can see the following messages:
Set up an EventBridge scheduled event
An EventBridge scheduled event is an EventBridge event that is triggered on a regular schedule. This section shows how to create an EventBridge scheduled event that runs every day at 7 AM UTC and at 8 PM UTC to trigger the kendra-index-scaler
Lambda function. This allows your index to scale up with the additional query capacity units at 7 AM and scale down at 8 PM.
When you set up EventBridge scheduled events, you do so for the UTC time zone, so you need to calculate the time offset. For example, to run the event at 7 AM Central Standard Time (CST), you need to set the time to 1 PM UTC. If you want to accommodate for daylight savings, you have to create a different rule to account for the difference.
- On the EventBridge console, in the navigation pane, under Events, choose Rules.
- Choose Create rule.
- For Name, enter a name for your rule (for this post, we use
kendra-index-scaler
).
- In the Define pattern section, select Schedule.
- Select Cron expression and enter
0 7,20 * * ? *
.
We use this cron expression to trigger the EventBridge event every day at 7 AM and 8 PM.
- In the Select event bus section, select AWS default event bus.
- In the Select targets section, for Target, choose Lambda function.
- For Function, enter the function you created earlier (
lambda_function_kendra_index_handler
).
- Choose Create.
You can check Amazon CloudWatch Logs for the lambda_function_kendra_index_handler
function and see how it behaves depending on your index’s query capacity units.
Conclusion
In this post, you deployed a mechanism to automatically scale additional query processing units for your Amazon Kendra Enterprise Edition index.
As a next step, you could periodically review your usage patterns in order to plan the schedule to accommodate your query volume. To learn more about Amazon Kendra’s use cases, benefits, and how to get started with it, visit the webpage!
About the Authors
Juan Bustos is an AI Services Specialist Solutions Architect at Amazon Web Services, based in Dallas, TX. Outside of work, he loves spending time writing and playing music as well as trying random restaurants with his family.
Tapodipta Ghosh is a Senior Architect. He leads the Content And Knowledge Engineering Machine Learning team that focuses on building models related to AWS Technical Content. He also helps our customers with AI/ML strategy and implementation using our AI Language services like Amazon Kendra.
Tom McMahon is a Product Marketing Manager on the AI Services team at AWS. He’s passionate about technology and storytelling and has spent time across a wide-range of industries including healthcare, retail, logistics, and ecommerce. In his spare time he enjoys spending time with family, music, playing golf, and exploring the amazing Pacific northwest and its surrounds.