AWS Database Blog

Automate the stopping and starting of Amazon Neptune environment resources using resource tags

Automating the management of the compute resources associated with your Amazon Neptune database cluster can save you time and money. The most significant cost when running Neptune for your graph workloads are the compute resources in the database cluster. If you’re also using associated resources such as Amazon SageMaker notebook instances, which you can use to write queries that run against your graph database, these add to your workload cost. Additionally, with the introduction of Amazon Neptune auto-scaling, where your Neptune database cluster now has the ability to automatically add additional read replicas to meet your workload requirements, costs can potentially grow over time.

A recommended approach is that you stop any resources that you aren’t using, however it can often be difficult to identify which resources should be stopped. If you’re running your graph workload across multiple environments such as development, QA, and production, using automation can help you save additional time and money, and improve your carbon footprint.

One way of doing this is to use resource tags. Tags help identify associated resources by means of a key-value pair that can group them together. For example, tagging all resources associated to a specific environment can help automated tasks be able to identify resources and, if necessary, stop them to reduce cost. Tagging resources is an AWS best practice, and can be used to as part of a wider tagging strategy, for example to help group resource costs in AWS Cost Explorer.

You can use scheduled tasks that run at specific times of the day to automate the stopping and starting of resources. The task can query all Neptune resources based on a specific set of tags. Automation gives better control over which Neptune resources are included within the process, and you can also use it to provide a mechanism for notifying specific teams of when their resources have been updated.

In this post, we show you how to create a serverless, event-driven solution to manage your Neptune database cluster and associated resources. First, we add tags to the resources you want to manage as part of the workflow, for example your Neptune database cluster and SageMaker notebooks. Then we create a workflow to automatically stop or start the resources based on a scheduled trigger. Finally, we publish messages containing information about the modified resources to an Amazon Simple Notification Service (Amazon SNS) topic, which is received as an email.

Overview of solution

Creating an event-driven, serverless workflow to automate the steps of stopping or starting your resources includes the following steps:

  1. Create two Amazon EventBridge rules to run on a schedule: one to stop the resources, and the other to start them.
  2. Use Amazon SNS to receive EventBridge rule notifications and pass them on to AWS Lambda functions.
  3. Write a stop function and a start function using Lambda, which are invoked depending on whether the workflow is to stop the resources or start them.

Depending on the timing of the scheduled stop and start triggers, for example, 17H00 (stop) and 08H00 (start), you may need to run this solution in your account for at least 24 hours to see the effects. The approximate cost of provisioning this solution in the N. Virginia (us-east-1) Region is as follows:

    • Without the example Neptune environment (including database cluster, instance, and notebook) the cost is less than $1 per day
    • With the example Neptune environment (including database cluster, instance, and notebook), the cost is less than $4 per day

The easiest way to follow along with this post is to use the provided AWS CloudFormation template. This template creates the necessary EventBridge rules, SNS topics, Lambda functions, and AWS Identity and Access Management (IAM) roles and policies that form the foundation of the solution. The template also creates the required AWS Systems Manager Parameter Store parameters based on input given during the stack creation. In addition, you can create a Neptune database instance for testing purposes. The following diagram illustrates the architecture for the stack.

Reference event-driven architecture

Prerequisites

For this walkthrough, you should have the following prerequisites:

  • An AWS account.
  • A Neptune database cluster and optionally a SageMaker notebook. The accompanying CloudFormation template can provision this for you if required.
  • Optionally, at least one Neptune database cluster tagged using the tag key and one of the tag values provided during stack creation.
  • Optionally, at least one SageMaker notebook instance tagged using the tag key and one of the tag values provided during stack creation.

Provision your resources with AWS CloudFormation

To deploy the CloudFormation template, complete the following steps:

  1. On the AWS CloudFormation console, choose Create stack.
  2. Choose With new resources (standard).
  3. Download the CloudFormation template locally to your machine. Choose Upload a template, then choose Choose file and choose the template you downloaded. Alternatively, select Amazon S3 URL and copy and paste the following link to the CloudFormation template into the text box:https://aws-blogs-artifacts-public.s3.amazonaws.com/artifacts/DBBLOG-1998/Neptune_Db_Cluster_Scheduled_Start_Stop_CFn_Template.yml
  4. Choose Next.
  5. For Stack name, enter a stack name of your choosing.
    CloudFormation stack name
  6. In the Parameters section, provide the following values:
    1. For CreateExampleNeptuneEnvironment, choose Yes to create a Neptune database cluster tagged with the specified tag values; otherwise, choose No.
    2. For NotificationEmailAddress, enter an email address to receive stop and start notifications.
    3. For StartScheduleExpression, enter a valid CRON schedule expression for starting resources.
    4. For StopScheduleExpression, enter a valid CRON schedule expression for stopping resources.
    5. For TagKey, enter the tag key used to identify resources to stop and start.
    6. For TagValues, enter a comma-separated list of values that have been populated for the tag key.

    The following screenshot is an example configuration that will run the start process at 08:00, and the stop process at 20:00, Monday to Friday, on any day of the month, during any month, in any year. This will be applied to Neptune resources that have a resource tag named Environment with a value of Dev, development, dev or DEV.

    CloudFormation stack parameter

  7. Choose Next.
  8. Continue through the remaining sections.
  9. Read and select the check boxes in the Capabilities section.
  10. Choose Create stack.

The stack takes approximately 5 minutes to complete. You will receive an email to the address specified in the NotificationEmailAddress parameter requiring you to confirm your subscription to the newly created SNS topic. You need to do this prior to being able to receive notifications generated by the solution.

EventBridge rule CRON schedules

This solution uses EventBridge to act as the initiator for the event-driven workflow. You create two rules—one for the stop workflow, and one for the start workflow—both of which include a specific CRON schedule. For example, the CRON schedule for the stop workflow is as follows.

Example stop EventBridge CRON schedule

The preceding schedule expression states the rule runs only at 17:00 on Monday to Friday, on any day of the month, during any month, in any year.

Similarly, the CRON schedule for the start workflow is as follows.

Example start EventBridge CRON schedule

This schedule expression states the rule runs only at 08:00 on Monday to Friday, on any day of the month, during any month, in any year. The combination of both rules result in any affected Neptune database cluster and associated resources only running during the period of 08:00 to 17:00, Monday to Friday.

For more information about CRON statements, refer to Schedule Expressions for Rules.

Create targets for the scheduled events

After you create the EventBridge rules, you need to provide at least one target to receive the notification event from each rule. In this case, we use Amazon SNS to receive the event and invoke further downstream services. Amazon SNS is included in this design to provide a decoupled mechanism for processing events. After an event is received by Amazon SNS, we invoke a Lambda function, passing it the details of the event.

SNS subscription to EventBridge

Store reusable parameters

Each of the Lambda resources in the design performs a single function: to identify Neptune and SageMaker notebook instances within the same account that have been tagged with a specific tag key, and either stop or start it. We use Parameter Store parameters to store the tag key and the comma-separated list of tag values that are used to identify these resources. Doing so provides a centralized, cost-effective, and secure repository that enables both functions to work using the same data, and any updates are viewed by both. From a security perspective, the Lambda functions are assigned a single role that permits only a narrow set of actions pertaining to Neptune, SageMaker, Amazon SNS, and AWS Systems Manager.

System Manager Parameter Store parameters

Apply resource tags in your own environment

After you deploy the solution, you can apply the tag key and one of the tag values specified in the CloudFormation template to your resources. This causes the tagged resources to be included in the automated process. When applying the resource tag to your Neptune database, you need only apply the resource tag to your Neptune cluster, rather than the writer instance, as shown in the following screenshot.

Resources tags applied to Neptune database cluster

Send post-action notifications

After each Lambda function has finished either stopping or starting identified resources, it publishes a message to a separate SNS topic, which is then sent to the email address specified in the CloudFormation deployment. Included in this message is the name of each resource that has been modified. The following screenshot illustrates an example notification email.

SNS Email Notification

Clean up

When you’re done automatically stopping and starting your resources, you can visit the AWS CloudFormation console and delete the stack that we created. This removes all the resources that we created as part of this solution.

If you created your own Neptune database cluster or SageMaker notebooks, you need to manually delete them to avoid additional cost.

Conclusion

In this post, we demonstrated how to create an automated serverless workflow that runs at scheduled instances to stop or start Neptune database clusters and associated SageMaker notebook instances using tags applied to these resources. Stopping these resources during periods where they’re not required can reduce cost and improve your carbon footprint. In addition, because the solution is event-driven and serverless, there are no instances to manage.

You can use the solution in this post as a foundation, and then extend it to include additional services that you want to stop and start as part of your Neptune environment.

Furthermore, you can run multiple instances of this solution in your account to monitor for different resource tag keys, such as cost center, owner, or workload values.

We encourage you to try out this solution in your AWS account where you’re running your Neptune resources. If you have any feedback, please leave a comment below.

About the author

Kevin Phillips is a Neptune Specialist Solutions Architect working in the UK at Amazon Web Services. He has 18 years of development and solutions architectural experience, which he uses to help support and guide customers.