AWS Cloud Operations Blog

Monitor your private internal endpoints 24×7 using CloudWatch Synthetics

Introduction

Since Amazon CloudWatch Synthetics launched in 2019, Synthetics canaries have become the first line of defense to reliably alert developers if their public endpoints, including REST APIs and URLs, show unexpected latencies or availability drops. In addition, Synthetics canaries can also monitor for broken links, or unauthorized content changes resulting from phishing, code injection, or cross-site scripting. With Synthetics canaries, developers no longer have to wait for accidental discovery of bugs, or until a frustrated customer calls in. They can proactively monitor to discover these issues, react faster, and avoid high cost to business.

Today, organizations rely on an intranet, or a private network to securely run their business infrastructure. Accessible only to authorized users, the network is shielded from external internet access and acts as an essential hub for communication, collaboration, and more. Microservices, APIs, or internal web tools inside a private network often power customer-facing websites and applications. Continuously monitoring both internal and external endpoints using synthetic monitoring is vital to ensure smooth operations on a consistent basis. This blog post covers:

  • Why monitoring internal endpoints should be part of your user experience monitoring strategy
  • How to configure Synthetic canaries inside your firewall including set up inside the Amazon Virtual Private Cloud (Amazon VPC)

Monitoring internal endpoints

Here’s why you should use synthetic monitoring to monitor internal endpoints in addition to public endpoints:

  1. Many modern applications are built such that various elements on a single webpage are powered by different underlying microservices and stacks. Taking a simple example, recommendations, search, product listings and filtering options on a Bookstore app are powered by one or more underlying services. A failure in the underlying image microservice causes images to stop loading in the recommendations pane. You want to detect and correlate the internal or external failure the instant it occurs and fix it before your customers send in a complaint or file this issue with support.
  2. Because APIs run in the background of an application, and are not directly accessed by users, the concept of real-user API monitoring would not make much sense. Using synthetic monitoring to generate traffic that mimics real user traffic to monitor internal APIs and being able to correlate with other parts of the application helps developers get to the root cause of the issue quickly and effectively.
  3. It is not possible to gauge the performance of new features in a pre-production environment with limited or no real users. In such a scenario, synthetic monitoring allows you to test the impact of changes to the application before the changes go live. Synthetic monitoring can simulate real use cases for detailed testing in a staging environment.
  4. The lifeline of many businesses relies on a range of internal tools and critical applications that are regularly used by the entire workforce within the company. From accounting, invoicing, health, payroll, Human Resources (HR) portals, Customer Relationship Management (CRM) systems, corporate email, collaboration tools, and so on, the list of applications your employees rely on is long! Continuously monitoring for changes in latency and availability for critical internal applications has a direct impact on increased productivity for your organization.

Getting started

There are two parts for setting up the Synthetics canary. First, ensure and create the resources required for an internal endpoint. Next, create and configure Synthetics canary to monitor the internal endpoint.

  1. Part A: Setting up VPC pre-requisites
    1. (Optional) Create an internal endpoint to use in this exercise
    2. Setup networking pre-requisites for internal endpoints running on your existing Amazon VPC or other VPC networks/firewall setups
  2. Part B: Monitoring the internal endpoint
    1. You create a canary using CloudWatch Synthetics to monitor the internal endpoint

Solution overview

For internal endpoints on Amazon VPC

Below is a schematic of components for setting up a Synthetics canary to monitor an internal endpoint on Amazon VPC. It also shows network traffic flow from the Synthetics canary to the internal endpoint and other AWS services. The VPC does not have internet access enabled.

For internal endpoints on other VPC providers or firewall setups

Below is a schematic of components involved for setting up a Synthetics canary to monitor an internal endpoint running on other VPC networks/firewall setups. It also shows the network traffic flow from the Synthetics canary to your internal endpoint and other AWS services.

Part A: Setting up VPC pre-requisites

Following is a walkthrough of ensuring and creating resources before we setup up the Synthetics canary.

Step 1: (Optional) Setup a dummy internal endpoint

  1. If you have an internal endpoint available to use with Synthetics canary, you can skip this optional step.
  2. For this exercise, we create a static website using Amazon Simple Storage Service (Amazon S3). To create the endpoint using AWS Management Console, follow Configure a bucket as Static website guide.
  3. After you’ve created a static website, depending on your Region, you will have an Amazon S3 website in one of the following formats (Learn more about endpoint formatting). 
    http://bucket-name.s3-website.Region.amazonaws.com
    http://bucket-name.s3-website-Region.amazonaws.com
  4. Please note this endpoint so that it can be used when configuring the Synthetics canary.
  5. If you already have an Amazon VPC that you can use, go to Step 2 for internal endpoints on Amazon VPC. Go to Step 3: For internal endpoints on other VPC providers or firewall setups to create a new VPC.

Note: So far, the endpoint is public and should be accessible via public internet. In the coming steps, we restrict access for this endpoint to simulate an internal endpoint.

Step 2: For internal endpoints on Amazon VPC

  1. Go to Step 3: For internal endpoints on other VPC providers or firewall setups if your endpoint is on other VPC providers or firewall setups
  2. For your Amazon VPC that you use, note down the VPC id, private subnet IDs and security group IDs to be used when configuring the Synthetics canary
  3. If your Amazon VPC has internet access enabled, follow steps here otherwise go to next step.
    1. Ensure that public subnets in your VPC have a NAT Gateway (or NAT instance) and you must have one or more private subnets, which are used for creating the Synthetics canary.

      Note: Whether a subnet is private or public depends on its route table. A public subnet has a route pointing to an internet gateway, and a private subnet does not.

  4. (Optional) If your Amazon VPC does not have internet access enabled, follow steps here:
    1. Provision two VPC endpoints by following the guidelines at VPC endpoint for S3 and Using CloudWatch with VPC endpoints.
    2. Ensure that DNS resolution and DNS hostname options are enabled for the VPC. Refer to Using DNS with your VPC.
  5. (Optional) If you followed Step 1: (Optional) Set up a dummy internal endpoint, then let’s replace the S3 bucket policy to restrict the access to your VPC only. Using Adding a bucket policy guide to add the following bucket policy after replacing the example bucket name and the VPC id with your information.

    Note: If you open your website after applying the following bucket policy, you should not be able to access it from the public internet.

    {
	"Version": "2012-10-17",
	"Id": "Policy1585781604328",
	"Statement": [
		{
			"Sid": "Stmt1585781599291",
			"Effect": "Allow",
			"Principal": "*",
			"Action": "s3:GetObject",
			"Resource": [
				"arn:aws:s3:::example-bucket", 
				"arn:aws:s3:::example-bucket/*"
			],
			"Condition": {
				"StringEquals": {
					"aws:sourceVpc": "vpc-1234567"
				}
			}
		}
	]
}

Step 3: For internal endpoints on other VPC providers or firewall setups

  1. Skip this step if you followed the Step 2: For internal endpoints on Amazon VPC.
  2. Create a public-private Amazon VPC using Creating a VPC with Public and Private Subnets guide. Once the VPC creation is successful, note the VPC id, Elastic IP address from the NAT Gateway page, private subnet IDs, and security group IDs in order to use when configuring the Synthetics canary.
  3. (Optional) If you followed Step 1: (Optional) Set up a dummy internal endpoint, then let’s replace the S3 bucket policy to restrict the access to your IP only. Use Adding a bucket policy guide to add the following bucket policy. Make sure to replace the example bucket name and the aws:SourceIp with the Elastic IP address in the following policy.

    Note: If you open your website after applying the following bucket policy, you should not be able to access it from the public internet.

    {
    "Version": "2012-10-17",
    "Id": "Policy1585081442378",
    "Statement": [
        {
            "Sid": "Stmt1585081440021",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::example-bucket/*",
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": "13.XX.YYY.ZZ"
                }
            }
        }
    ]
}

Part B: Monitoring the internal endpoint

Following is a walkthrough of how to create a canary using the resources from the setup above.

Step 1: Configure canary details

  1. Navigate to Create canary page on the AWS Management Console
  2. Select Use a blue print and Heartbeat monitoring blue print
  3. Enter a Name for the canary
  4. Enter your Application or endpoint URL or dummy endpoint we created

    Note: Select the http:// option if you’re using the dummy endpoint

  5. (Optional) Schedule: Use the default or select your desired schedule
  6. (Optional) Data Retention: Use defaults or select your desired retention period for this canary’s run results.
  7. (Optional) Data Storage: Leave this empty or select an S3 folder where you would like to store the artifacts created by each canary run
  8. (Optional) Access Permissions: Use Create a new role or provide your own role with required permissions

    Note: Synthetics console will create a new role that includes all necessary permissions to run the canary. If you selected your own S3 bucket, you must select an existing role.

  9. (Optional) CloudWatch Alarms: If your canary detects any issues, you can set alarms to get notified.

Step 2: Configure VPC settings

  1. Select Virtual Private Cloud (VPC) based on the information available from pre-requisites
  2. Select one or more private subnets from Subnets options (At least two subnets are recommended for high availability)
  3. Select one or more security groups from Security groups options

Step 3: Create canary

  1. Click Create canary

    Note: This step should take about a minute to process. Wait until the canary is created and started.

Step 4: View canary runs

  1. Navigate to canary details page by selecting the canary from the canaries list page.

    Note: You should see that the canary is in Running state. You can view your canary run data including screenshots, HTTP archive (HAR) files, and log files.

  2. Click on Configuration tab to view canary details including the VPC configuration you specified when creating the canary.

    Note: Visit the troubleshooting guide Troubleshoot a canary on a VPC if you run into errors after you have created the canary.

 

Cleaning up

To avoid any unwanted costs, remember to clean up the following resources that may be created during this exercise

  1. The Synthetics canary and associated resources.
  2. The static S3 endpoint.
  3. The Public-Private VPC, subnets and security groups.

Conclusion

In this post, we used Amazon CloudWatch Synthetics to route a CloudWatch Synthetics canary to originate from the same network that your internal endpoints are accessible from or from a single IP address owned by you.

Try it on other types of internal endpoints that benefit from 24×7, automated synthetic monitoring:

  1. Internal APIs that power critical customer-facing components in your applications
  2. Critical internal tools that are exposed to customers/partners with contractual SLAs
  3. Pre-production endpoints (alpha, beta, gamma stage environments) where there is no real-customer traffic
  4. Microservices powering business-critical internal applications including legal, HR, billing, and email

To learn more about CloudWatch Synthetics, see the CloudWatch Synthetics documentation.

———————————————————————————————————————

About the authors

Nitya Sheth is a Software Engineer working on Amazon CloudWatch Synthetics at AWS. He has also worked on user experience implementations for Database, Analytics & AI AWS consoles including Amazon Lex, Amazon Translate, Amazon DeepRacer, and Amazon Redshift. He is an outdoor enthusiast who likes to explore new hiking places and adventure sports.

 

 

 

Surbhi Dangi is a Product & Design Manager at AWS. She works on several Database and Analytics services and broadly adopted internal tools for AWS teams – including Amazon Redshift, Amazon CloudWatch Synthetics and others.

 

 

 

 

Raja Bhogi is an Engineering Manager at AWS. He is responsible for CloudWatch Synthetics service and building delightful and easy-to-use web experiences for analytics & blockchain products. His work includes launching CloudWatch Synthetics service, web experiences for new analytics products, and working on new feature launches for existing products. He is passionate about web technologies, performance insights, monitoring, and tuning. He is a thrill seeker and enjoys everything from roller coasters to bungee jumping.