Building a portfolio of self-service databases with AWS Service Catalog and AWS CloudFormation

Modern distributed applications are moving towards a “purpose-built” database strategy. This means that the selection of database type, size, and configuration should match the problem the database is trying to solve. AWS customers are also requiring that these databases have the appropriate level of security control and organizational governance to operate in customer environments. AWS Service Catalog solves these problems by enabling AWS customers to create and deliver standardized services that provide the necessary control, while still empowering developers to choose the services that best fit their needs. By also using AWS CloudFormation templates, you can distribute your services across many accounts in many Regions with the hub-and-spoke model, ensuring a consistent service offering across your environment.

In this blog post, I’ll walk you through automating the creation of a portfolio of self-service Amazon Relational Database Service (Amazon RDS) databases and demonstrate creating and associating various constraints on those databases to provide both security and governance. I’ll limit access to the portfolio so that only certain people are allowed to provision a database from the portfolio. I’ll also show you how to create and subscribe to notifications when a database is launched or terminated. Lastly, I’ll show you how you can take the CloudFormation template from this blog post and automate the distribution across the accounts and AWS Regions in your environment.

Download the CloudFormation template to follow along.

Before diving into the example, let’s go over some of the key concepts of AWS Service Catalog and AWS CloudFormation.

In AWS Service Catalog:

A portfolio is a collection of products, together with the configuration information. You can use portfolios to manage user access to products. You can grant portfolio access at an AWS Identity and Access Management (IAM) user, IAM group, and IAM role level.
A product is a blueprint for building AWS resources that you want to make available for deployment on AWS. A product can belong to multiple portfolios.
Constraints control the way users are able to deploy a product.

AWS CloudFormation allows you to model the state of your infrastructure as code and to deploy your AWS resources in a safe, repeatable manner.

Step 1: Creating the portfolio

I’ll start by creating a portfolio resource that acts as a container for the database products I want in my catalog. A portfolio allows you to apply permissions to control who can access it and therefore which products you choose to offer to a certain user or group. Here, I am creating a portfolio with the name “Database Portfolio” that contains the different database products available for self-service deployment.

DatabasePortfolio:
    Type: "AWS::ServiceCatalog::Portfolio"
    Properties:
      Description: "A portfolio of self-service databases."
      DisplayName: "Database Portfolio"
      ProviderName: "IT"

For instructions on creating a portfolio using the AWS Management Console, see Creating and Deleting Portfolios.

Step 2: Adding products to the portfolio

The three database products that I want to make available in my portfolio are a MySQL database, a PostgreSQL database, and a Microsoft SQL Server database. These databases are all running on Amazon RDS. The preconfigured CloudFormation templates used to create the database products in this example are taken from AWS Service Catalog reference architectures. Each template outputs a JDBC connection string that you can use to connect to a database launched from the portfolio.

Note
There are multiple CloudFormation templates as part of this solution. The template described in this blog post creates the AWS Service Catalog resources that allow you to launch the database products that are made up of additional templates. Because of this template separation, you can simply replace the database reference architecture templates in this blog post with your own templates by uploading the template to Amazon Simple Storage Service (Amazon S3) and modifying the LoadTemplateFromURL property.

The CloudFormationProduct resource that creates the MySQL database product is shown in the following code and allows the specification of multiple provisioning artifacts (also known as versions). Each version of the product maps to a separate AWS CloudFormation template. The databases in this blog post only provide a single version, but additional versions can be added by appending to the ProvisioningArtifactParameters list. Each database that is offered in the portfolio has a separate CloudFormationProduct resource.

Example MySQL database product:

MySQLDatabaseProduct:
  Type: "AWS::ServiceCatalog::CloudFormationProduct"
  Properties: 
    Name: "MySQL Database"
    Description: "MySQL Database with corporate configuration."
    Distributor: "IT"
    Owner: "IT"
    SupportDescription: "Please call any time for support"
    SupportEmail: "info@example.com"
    SupportUrl: "https://www.example.com"
    ProvisioningArtifactParameters:
      -
        Name: "MySQL Database"
        Description: "MySQL Database"
        Info:
          LoadTemplateFromURL: "https://raw.githubusercontent.com/aws-samples/aws-service-catalog-reference-architectures/master/rds/sc-rds-mysql-ra.yml"

Associating the MySQL database product with the Database Portfolio is then done by creating an association resource and referencing the product and portfolio to associate. I’ll need to also add an association for each of the other databases in my portfolio.

MySQLtoDatabasePortfolioProductAssociation:
  Type: "AWS::ServiceCatalog::PortfolioProductAssociation"
  Properties:
    PortfolioId: !Ref DatabasePortfolio
    ProductId: !Ref MySQLDatabaseProduct

Creating products and adding products to a portfolio can also be done using the AWS Service Catalog console.

Step 3: Granting portfolio permissions

Now that I’ve associated database products to my portfolio, I need to limit access to the portfolio so that only members of my engineering team can access it and deploy database products. I accomplish this by creating an IAM group and granting that IAM group a policy that allows them to use AWS Service Catalog as a non-administrative user.

EngineeringGroup:
  Type: "AWS::IAM::Group"
  Properties:
    ManagedPolicyArns: 
      - arn:aws:iam::aws:policy/AWSServiceCatalogEndUserFullAccess

I’ll then associate that IAM group to the Database Portfolio by adding a different association resource. This association effectively grants members of the engineering group access to the portfolio and the ability to provision their choice of database. Granting additional IAM users access to the portfolio is then simply done by adding them to the engineering group.

DatabasePortfolioPrincipalAssociation:
  Type: "AWS::ServiceCatalog::PortfolioPrincipalAssociation"
  Properties:
    PortfolioId: !Ref DatabasePortfolio
    PrincipalARN: !GetAtt EngineeringGroup.Arn
    PrincipalType: "IAM"

Note
After deploying the template, you need to create and manually add an IAM user to the engineering group in order to access the Database Portfolio.

If you prefer to manage portfolio permissions outside of your CloudFormation template, see Granting Access to Users.

Step 4: Adding tag options

Tagging allows you to define custom metadata on an AWS resource to group resources for organizational or cost accountability purposes. AWS Service Catalog tag options enforce tagging standards by allowing an administrator to predefine the set of possible tags that can be selected when provisioning a product. Here, I associate the ProductType tag option to the Database Portfolio and the DatabaseType tag option to each database product. When a user provisions a database, they will be given the option to select from a list of associated tag options that come from both the product and its portfolio.

DatabaseTagOption:
  Type: "AWS::ServiceCatalog::TagOption"
  Properties:
    Key: "ProductType"
    Value: "Database"

DatabasePortfolioTagOptionAssociation:
  Type: "AWS::ServiceCatalog::TagOptionAssociation"
  Properties:
    ResourceId: !Ref DatabasePortfolio
    TagOptionId: !Ref DatabaseTagOption

PostgreSQLDatabaseTagOption:
  Type: "AWS::ServiceCatalog::TagOption"
  Properties:
    Key: "DatabaseType"
    Value: "PostgreSQL"

PostgreSQLDatabaseTagOptionAssociation:
  Type: "AWS::ServiceCatalog::TagOptionAssociation"
  Properties:
    ResourceId: !Ref PostgreSQLDatabaseProduct
    TagOptionId: !Ref PostgreSQLDatabaseTagOption

For information on managing tag options through the AWS Service Catalog console, see Managing TagOptions.

Step 5: Adding portfolio constraints

Constraints allow an administrator to have tighter control over the product provisioning process. There are currently three types of constraints available for AWS Service Catalog: launch constraints, notification constrains, and template constraints. In this step, I’ll demonstrate how to apply each constraint to products in the Database Portfolio.

Note
The AWS Service Catalog constraint CloudFormation resources expect the product to be associated to the specified portfolio before they are applied. If the product is associated to the portfolio in the same template in which the constraint is created, you must use a DependsOn attribute to declare a dependency on the PortfolioProductAssociation for the product to ensure that it is associated to the portfolio prior to applying the constraint.

Launch constraint

The first of the constraints is the launch role constraint. Applying a launch role constraint to databases in the portfolio allows you to avoid giving the members of the engineering group permissions to Amazon RDS directly. Engineering group members will only need to have access to AWS Service Catalog. AWS Service Catalog will handle creating the CloudFormation stack and RDS database on their behalf by assuming the role defined by the launch constraint.

The following role and attached policy grants AWS Service Catalog the ability to interact with AWS CloudFormation and provision Amazon RDS instances. The constraint takes that role and associates it to a database in the portfolio. The example template creates a constraint for each database in the portfolio.

MySQLDatabaseLaunchConstraint:
  Type: "AWS::ServiceCatalog::LaunchRoleConstraint"
  Properties:
    Description: "Allows the product to launch with the policies granted by the associated role."
    PortfolioId: !Ref DatabasePortfolio
    ProductId: !Ref MySQLDatabaseProduct
    RoleArn: !GetAtt RDSLaunchConstraintRole.Arn
  DependsOn: MySQLtoDatabasePortfolioProductAssociation

RDSLaunchConstraintRole:
  Type: "AWS::IAM::Role"
  Properties:
    Path: "/"
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: "Allow"
          Principal:
            Service: "servicecatalog.amazonaws.com"
          Action: "sts:AssumeRole"
    Policies:
      - PolicyName: "AllowProductLaunch"
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Resource: '*'
              Effect: "Allow"
              Action:
                # Permissions required for the provisioning of the database
                - rds:*
                - ec2:DescribeAccountAttributes
                - ec2:DescribeSubnets
                - ec2:DescribeVpcs
                - ec2:DescribeSecurityGroups
                - ec2:DescribeSecurityGroupReferences
                - ec2:DescribeStaleSecurityGroups
                - ec2:CreateSecurityGroup
                - ec2:DeleteSecurityGroup
                - ec2:AuthorizeSecurityGroupIngress
                - ec2:AuthorizeSecurityGroupEgress
                - ec2:UpdateSecurityGroupRuleDescriptionsIngress
                - ec2:UpdateSecurityGroupRuleDescriptionsEgress
                - ec2:RevokeSecurityGroupEgress
                - ec2:RevokeSecurityGroupIngress
                - ec2:CreateTags
                # Permissions required by AWS Service Catalog to create stack                  
                - cloudformation:GetTemplateSummary
                - s3:GetObject
            - Resource:
                - "arn:aws:cloudformation:*:*:stack/SC-*"
                - "arn:aws:cloudformation:*:*:changeSet/SC-*"
              Effect: "Allow"
              Action:
                # Permissions required by AWS Service Catalog to create stack  
                - cloudformation:CreateStack
                - cloudformation:DeleteStack
                - cloudformation:DescribeStackEvents
                - cloudformation:DescribeStacks
                - cloudformation:SetStackPolicy
                - cloudformation:ValidateTemplate
                - cloudformation:UpdateStack
            - Resource: !Sub "arn:aws:sns:${AWS::Region}:${AWS::AccountId}:${EngineeringDatabaseCreatedTopic.TopicName}"
              Effect: "Allow"
              Action:
                - sns:Publish

Notification constraint

A notification constraint allows an interested third party, such as an engineer’s manager, to be notified when a new database has been provisioned or terminated via AWS Service Catalog. This constraint instructs AWS Service Catalog to publish stack events from AWS CloudFormation to an Amazon Simple Notification Service (Amazon SNS) topic. The third party can then subscribe to this SNS topic and receive updates via e-mail for CloudFormation stack events. You can even attach an AWS Lambda function to the SNS topic and take automated action when a new database is launched.

The following snippet creates an SNS topic and associates it with a single database. To receive notifications in the form of e-mail from this notification constraint, you must manually subscribe to the “Engineering Database Created” topic after deploying the template.

EngineeringDatabaseCreatedTopic:
  Type: "AWS::SNS::Topic"
  Properties:
    DisplayName: "Engineering Database Created"

MicrosoftSQLDatabaseCreatedLaunchNotification:
  Type: "AWS::ServiceCatalog::LaunchNotificationConstraint"
  Properties:
    Description: "Publishes notifications when a database is provisioned."
    NotificationArns: 
      - !Ref EngineeringDatabaseCreatedTopic
    PortfolioId: !Ref DatabasePortfolio
    ProductId: !Ref MicrosoftSQLDatabaseProduct
  DependsOn: MicrosoftSQLtoDatabasePortfolioProductAssociation

Template constraint

A launch template constraint allows you to limit the set of values for a parameter that users can choose when launching a product. In the following example, I want to ensure that all databases launched from the Database Portfolio are launched into more than one Availability Zone, and are therefore highly available by default. To accomplish this, I have added a rule that forces anyone who launches this template to select a “true” value for the MultiAZ property.

ConstrainMySQLDatabaseInstanceSize:
  Type: "AWS::ServiceCatalog::LaunchTemplateConstraint"
  Properties:
    Description: "Constrains the size of a database instance "
    PortfolioId: !Ref DatabasePortfolio
    ProductId: !Ref MySQLDatabaseProduct
    Rules: >
      {
        "Rule1" : { 
          "Assertions" : [{ 
            "Assert" : {
              "Fn::Equals" : [
                "true", 
                {"Ref" : "MultiAZ"}
              ]
            }, 
            "AssertDescription" : "Databases must be deployed with the MultiAZ property enabled."
          }]
        }
      }
  DependsOn: MySQLtoDatabasePortfolioProductAssociation

Step 6: Deploying the template

Launch the stack to deploy the template to a single account and follow the steps for creating an AWS CloudFormation stack.

Deploying to many AWS accounts

In organizations that have multiple AWS accounts, you will find it easier to manage a single, central portfolio in a master account and distribute that portfolio to child accounts. This is known as the hub-and-spoke model.

To support the hub-and-spoke model, there are two additional AWS Cloudformation resources required that are not included in the template referenced in this blog post.

Portfolio share

The PortfolioShare resource allows you to specify the child (“spoke”) account to share a portfolio with. This resource would be included as part of an AWS Cloudformation template that is deployed to the central (“hub”) account.

SharePortfolioWithChildAccountA:
  Type: "AWS::ServiceCatalog::PortfolioShare"
  Properties:
    AccountId: "AccountId that this portfolio should be shared with"
    PortfolioId: !Ref EngineeringPortfolio

Accepting a portfolio share

After you’ve shared the portfolio with a child account from the central account, each child account must then accept the share using an AcceptedPortfolioShare resource. This resource must be part of an AWS CloudFormation template that is deployed in each child account.

AcceptEngineeringPortfolioShare:
  Type: "AWS::ServiceCatalog::AcceptedPortfolioShare"
  Properties:
    PortfolioId: "port-000000000"

The value for the PortfolioId property can be found by the following steps:

Navigate to the AWS Service Catalog console in the master account.
Choose Portfolios List on the left navigation pane.
Select the portfolio that you created a share for.
Copy the ID from the portfolio details.

Conclusion

In this blog post, I have demonstrated how to create an AWS Service Catalog portfolio that enables an organization to empower their developers with self-service provisioning of their database of choice, while still maintaining the necessary governance and controls required to operate safely.

For further information, please see the AWS Service Catalog CloudFormation documentation.

If you have questions about implementing the solution described in this blog post, you can start a new thread on the AWS Service Catalog Forum or contact AWS Support.

About the Author

Matt Luttrell is a Cloud Application Architect with AWS Professional Services. He specializes in serverless applications, containers, and architecting distributed systems. When he’s not spending time with his family, he enjoys skiing, cycling, and the occasional video game.

AWS Cloud Operations Blog