Networking & Content Delivery

How Northwestern Mutual optimized and improved efficiency with Amazon Route 53 Profiles

Managing DNS configurations across multiple Amazon Virtual Private Clouds (Amazon VPCs) and Amazon Web Services (AWS) accounts can be a daunting task for network administrators, especially in complex environments with numerous Private Hosted Zones (PHZs) and Amazon Route 53 Resolver rules. Traditionally, they relied on outbound and inbound Route 53 Resolver endpoints to transport DNS queries between VPC resources across accounts, which led to operational overhead and increased costs. Enter Route 53 Profiles, a feature that streamlines DNS management by providing a centralized method for configuring and managing DNS settings across multiple VPCs and accounts.

This post showcases how Northwestern Mutual, an AWS customer, transformed their environment by migrating to Route 53 Profiles, reducing costs and operational overhead in the process.

Customer environment

At Northwestern Mutual, we manage hundreds of AWS accounts and have adopted a distributed deployment strategy for Amazon Route 53 PHZs. This approach involves provisioning a dedicated PHZ within each AWS account. Implementing this pattern provides development teams with access to their own isolated PHZ, which they use to manage DNS record sets specific to their AWS workloads. This approach empowers teams with dedicated DNS management capabilities and mitigates the potential impact of DNS changes made by a single team on the broader DNS ecosystem across the organization. Figure 1 shows our current architecture for DNS resolution between AWS and the corporate data center.

As part of our established and architected account provisioning process and landing zone, we automatically provision a PHZ within each new AWS account. We associate this PHZ with a VPC local to the account where we provision it. To maintain consistency and ease of identification, we follow a standardized naming convention for these PHZs: <account-identifier>.aws.example.com

In this naming structure, ”account-identifier” represents a unique identifier for the specific AWS account, while ”example.com” is our internal corporate domain. For example, for an account with an identifier of “ABC”, the resulting PHZ name would be abc.aws.example.com. We follow this pattern for all accounts across our ecosystem.

As shown in the following figure, our organization has a dedicated central networking account named the “Network Services Account”. Within this central account, we have provisioned a dedicated VPC called the ”DNS Hub” VPC. This ”DNS Hub” VPC serves as the centralized focal point for DNS resolution across our entire AWS environment.

Figure 1. Northwest Mutual current environment showing DNS resolution between AWS and Corporate Data Center

Resolution from on-premises to Route 53 PHZs

To facilitate DNS resolution from our on-premises environment (desktops, servers, etc.) to the DNS record sets hosted in our Route 53 PHZs, we used the Route 53 Resolver Inbound Endpoint feature. As shown in Figure 1, we deployed this Inbound Endpoint in a dedicated ”DNS Hub” VPC to provide it with a comprehensive view of our entire private DNS ecosystem.

This ”DNS Hub” VPC serves as a centralized reference point for private DNS across our organization. As well as each PHZ associating with its account-local VPC, we established a cross-account association between all PHZs and this ”DNS Hub” VPC. Consequently, the ”DNS Hub” VPC provides a comprehensive view of all Route 53 PHZs across our AWS accounts.

The Resolver Inbound Endpoint deployed within the ”DNS Hub” VPC is associated with every PHZ in our organization. This association provides a comprehensive private DNS view, including all PHZs across our AWS ecosystem. To enable on-premises resolution, we configured our on-premises DNS server to forward all DNS queries for the *.aws.example.com domain to this centralized Resolver Inbound Endpoint.

Through this architecture we have established a seamless DNS resolution mechanism that allows our on-premises resources (desktops, servers, etc.) to resolve DNS record sets hosted in our Route 53 PHZs across multiple AWS accounts.

Resolution from VPCs to on-premises

To enable the resolution of DNS records for applications hosted in our on-premises environment from resources within our VPCs, we used the following AWS services and components: a Route 53 Resolver Outbound Endpoint, a Route 53 Resolver Rule, and an AWS Resource Access Manager (AWS RAM) Resource Share. Refer to Figure 1 for a visual representation of this architecture.

  1. Resolver Outbound Endpoint: We created a Resolver Outbound Endpoint within our dedicated “DNS Hub” VPC. This Outbound Endpoint serves as the gateway for DNS queries originating from our VPC-bound resources to our on-premises DNS environment.
  2. NEW Resolver Rule: We created a new Resolver Rule specifically for our on-premises corporate domain *.example.com and associated it with the Outbound Endpoint. The target IPs of this rule are our on-premises DNS server IPs. This rule defines the routing behavior for DNS queries destined for our on-premises domain.
  3. NEW AWS RAM Resource Share: To facilitate sharing this Resolver Rule across our entire AWS Organization, we used the AWS RAM service. We created a Resource Share and included the previously created Resolver Rule as the shared resource. Then, we specified our organization as the principal with which this Resource Share would be shared.
  4. Rule Association: The *.example.com Resolver Rule is associated with all VPCs in our environment. We modified our VPC build process to make sure that all newly created VPCs automatically associate with the shared Resolver Rule. We use Amazon Default DHCP options, which dictate that all VPC-bound resources default to using the ”Amazon Provided DNS” service, effectively using the Route 53 Resolver. Associating the shared Resolver Rule with each new VPC lets us make sure that resources launched within these VPCs can seamlessly resolve DNS records hosted in our on-premises DNS environment.

Resolution between accounts within AWS (a resource in a VPC in one account needing to resolve a record located in a PHZ in another account):

Enabling DNS resolution between resources in a VPC in one AWS account and a DNS record hosted in a PHZ located in another account, posed a challenge that needed a unique solution. We used additional components and built upon the existing architecture, as shown in Figure 1, to address this.

The cross-account query resolution flow relies on the following key components and configurations:

  1. Resolver Rule: We created a new Resolver Rule for the *.aws.example.com domain and associated it with the same Outbound Endpoint as described previously. The target IP addresses for this rule are the IP addresses belonging to the Inbound Endpoint in the ”DNS Hub” VPC.
  2. AWS RAM Resource Share: To share the *.aws.example.com Resolver Rule across all accounts, we created a new AWS RAM Resource Share, similar to how we shared the .example.com rule for on-premises resolution.
  3. Rule Association: The *.aws.example.com Resolver Rule is associated with all VPCs in our environment, just like the *.example.com rule for on-premises resolution.

With these additional components in place, the cross-account query resolution flow works as follows:

  1. When a resource in a VPC within the “ABC” account attempts to resolve a DNS record hosted in a PHZ located in the “XYZ” account (for example “app1.xyz.aws.example.com”), then the shared Resolver Rule for *.aws.example.com is referenced in the originating VPC.
  2. The *.aws.example.com Resolver Rule dictates that matching queries be sent out through the Outbound Endpoint, destined for the Inbound Endpoint. The ”xyz.aws.example.com” PHZ is associated with the ”DNS Hub” VPC, thus the Inbound Endpoint answers the query authoritatively and provides the response to the originating resource.
  3. The ”DNS Hub” VPC’s Inbound Endpoint answers the query and returns the response to the requesting resource in the “ABC” account.

This configuration establishes a cross-account DNS resolution mechanism. It allows resources in one AWS account to resolve DNS records in PHZs of another account. This creates a unified DNS ecosystem across our entire AWS environment, spanning multiple accounts and private zones.

Challenges

To fully use the resiliency of Route 53, AWS recommends directly associating each PHZ with the VPC that needs it, even in cross-account scenarios. This approach creates a full mesh of VPCs and hosted zones, which becomes challenging to manage when dealing with hundreds of VPCs.

As a result, we sought an alternative solution described earlier that avoids the need for a full mesh of VPCs to hosted zones. However, this design presents some challenges. DNS queries need to be forwarded through multiple Resolver endpoints to get a response, which adds complexity and introduces potential performance issues if an Availability Zone (AZ) experiences degradation. Furthermore, managing the cross-account association between VPCs and hosted zones is a multi-step process, involving actions in both the hosted zone’s account and the “Hub” VPC’s account. Although we have automated this process, it’s still a step we’d prefer to avoid.

Moving to Route 53 Profiles/benefits

  • A more streamlined DNS query routing eliminates “hopping” through multiple Resolver Endpoints. With Route 53 Profiles, we can eliminate hopping through multiple endpoints altogether for VPC-to-VPC resolution.
  • A streamlined setup for Route 53 in new accounts. Instead of creating cross-account associations between PHZs and VPCs manually, we can use AWS RAM shared Profiles. We reference the shared Profiles and associate them with the new account’s local VPC and PHZ. Profiles handle the rest, streamlining the new account onboarding process.
  • We can use the Route 53 built-in resiliency by effectively having a “full mesh” of VPC to hosted zone associations, managed through a Route 53 Profile.
  • We use Route 53 DNS Firewall, because implementing through a Route 53 Profile is direct. Previously, we didn’t use DNS Firewall primarily due to the overhead of associating a rule group to multiple VPCs individually.

Steps for migration

Figure 2 represents our environment after completing the migration. To complete the migration, we followed these steps to create a base setup without affecting our existing DNS resolution workflow:

  1. Create Route 53 Profiles in the Network Service Account.
  2. Associate the “DNS Hub” VPC under Network Service Account with Route53 Profiles.
  3. Associate both Route 53 Resolver rules, which are set up to forward DNS queries to the on-premises DNS server IPs and to the inbound endpoint, with Route53 Profiles.
  4. Share Route 53 Profiles with Organization accounts using AWS RAM.
  5. Associate the VPCs from each account in our organization with the shared Route 53 Profiles.
  6. Associate the PHZs from each account in our organization with the shared Route 53 Profiles.

Next, we discuss the steps we took to migrate to Route 53 profiles for DNS resolution. We started the migration with a single account in our organization and repeated the process for other accounts.

  • Removed the association of Route 53 Resolver rule-2 with the VPC in the App1 Dev Account for AWS-hosted application to application DNS resolution.
  • Removed the cross-account association of PHZ from App1 Dev Account with the “DNS Hub” VPC of Network Service account. This PHZ was previously used for DNS resolution from on-premises to AWS-hosted applications through inbound endpoints.
  • Removed the association of Route 53 Resolver rule-1 with the VPC in the App1 Dev Account for DNS resolution for on-premises hosted applications from AWS through outbound endpoints.

We repeated these steps for other accounts in our organization by removing the associations of PHZs and Route 53 Resolver rules with their respective VPCs. Finally, we deleted the sharing of Resolver rules with our organization.

Figure 2. Northwest Mutual environment showing DNS resolution between AWS and Corporate Data Center after migrating to Route53 Profiles

Summary

Our organization is delighted to bring Route 53 Profiles into our growing mix of foundational AWS services. It reduces complexity by eliminating the need for a few other components. Furthermore, it enhances resiliency by effectively creating a full mesh of Route 53 private zones to VPCs in our environment. Adopting the service also streamlines our new account onboarding process, where we no longer need to perform a cross-account hosted zone to VPC associations.

Author bios

Ankush Goyal.jpg

Ankush Goyal

Ankush Goyal is a Senior Technical Account Manager at AWS Enterprise Support, specializing in helping customers in the travel and hospitality industries optimize their cloud infrastructure. With over 20 years of IT experience, he focuses on leveraging AWS networking services to drive operational efficiency and cloud adoption. Ankush is passionate about delivering impactful solutions and enabling clients to streamline their cloud operations.

TravisDiener.jpg

Travis Diener (Guest)

Travis Diener is a Lead Software Engineer at Northwestern Mutual, based in Milwaukee, Wisconsin. He is a passionate Cloud advocate, and enjoys providing technical guidance, mentoring others, and leading cloud infrastructure design and implementation efforts within his organization, ensuring their success on AWS. He holds a BS in Computer Science and several AWS certifications. He has been building on AWS for nearly eight years.

Anand.jpg

Anandprasanna Gaitonde

As an AWS Principal Solutions Architect, Anandprasanna Gaitonde is responsible for helping customers design and operate Well-Architected solutions to help them adopt the AWS cloud successfully. He focuses on AWS Networking and Serverless technologies to design and develop solutions in the cloud across industry verticals. He has Solutions Architect Professional and Advanced Networking certifications and holds a Master of Engineering degree in Computer Science and a postgraduate degree in Software Enterprise Management.