How BBVA USA delivered security and governance at scale using management tools

As BBVA USA began its digital transformation journey, the security operations team had to improve its processes around provisioning and baselining of AWS accounts. The demand for new AWS accounts continued to increase from multiple application teams within the bank. In an effort to standardize new accounts within the enterprise, BBVA USA built an automated account creation process. This process is designed with a multi-account strategy in mind supporting development, test, and production environments. The team also achieved security and governance controls by using the management tools within AWS services portfolio. Here we provide a tutorial of the account creation process explaining these management tools and services were used at scale.

About BBVA USA

In the US, BBVA is a sunbelt-based financial institution that operates 641 branches, including 330 in Texas, 89 in Alabama, 63 in Arizona, 61 in California, 44 in Florida, 37 in Colorado and 17 in New Mexico. The bank ranks among the top 25 largest US commercial banks based on deposit market share and ranks among the largest banks in Alabama (2nd), Texas (4th), and Arizona (6th). In the US, BBVA has been recognized as one of the leading small business lenders by the Small Business Administration (SBA). It is also ranked 14th nationally in terms of dollar volume of SBA loans originated in fiscal year 2019.

The Challenge

BBVA was rapidly scaling their AWS Cloud infrastructure. However, the existing account creation process was manual and used PowerShell and AWS CLI commands. This process was not secure or scalable. The security team was looking for a way to centrally manage security and governance controls for both existing and new accounts. The goal is to automate the account creation process using AWS CloudFormation. This would standardize the baseline account creation and facilitate continuous integration and continuous delivery. BBVA also started using a more robust language such as Python boto3 and state machine. They also simplified existing automation by using service-managed AWS CloudFormation StackSets and AWS Organizations Service Control Policies (SCP) and moved all security services to AWS CloudFormation templates.

Architecture

As a part of the account creation process, new accounts were provisioned using AWS Organizations API. Once an account is provisioned, the account is moved into an AWS Organization Unit (OU) within AWS Organizations. The OU matches the environment’s intended use – development, test, or production. Movement into this OU automatically deployed service-managed AWS CloudFormation StackSets and applied inherited SCPs.These policies prohibit resources from modification such as security managed AWS CloudTrail trails, AWS Config rules, and remediation AWS Lambda functions.

The first phase tackled the need for automatic account creation. Once the create-account API call returned new the account numbers, the account information is passed to a Lambda function. This Lambda function is authenticated to the AWS Organizations main account. It moved the new account from the root OU to appropriate OU based on the ticket, or JSON object’s data. Moving the account auto deployed OU specific StackSets to build standard roles for each account. In addition to the inherited SCPs, one is attached directly to the account to provide a limited availability of specific allow list of services. Lastly, this function calls the AWS Support API to open a ticket to request Enterprise Support to be added.

1. Service-managed CloudFormation templates assigned to auto deploy create stack instances.
2. Service Control Policies applied to the new OU take effect in the new account applying security and governance controls.
3. Apply an account-level SCP to replace the default one granting full access.
4. Once in the OU, service-managed stacks with auto-deployment options enabled begin to deploy standard IAM resources.
5. Default roles, policies, and boundary permissions that function as governance guardrails are created.

The second phase focused on hardening security and governance across these accounts by implementing centralized logging and monitoring to assets, audits, security controls, and tagging.

To centralize logging, AWS Config is used to continuously monitor and record AWS resource configurations. It allows you to automate the evaluation of recorded configurations against desired configurations. For near-real time security control, the recommendation is to use Amazon GuardDuty. GuardDuty is used to detect threat and continuously monitor for malicious activity and unauthorized behavior to protect AWS accounts and workloads.  The findings for the centrally managed GuardDuty implementation were passed to Amazon EventBridge. They are aggregated in the security logging account for ingestion into Security Information Event Management (SIEM) tool for SOC analysis and response. The inclusion of EventBridge improved the centralization of the solution and contributed to redefining how responses are handled. This approach is also used for other security services to provide a repeatable solution for security logging and response.

Finally, account activities, and network event logs such as AWS CloudTrail, Amazon VPC Flow Logs, and AWS Systems Manager logs were collected and aggregated for potential threat detection. These logs are integrated with Amazon CloudWatch Events to generate actionable insights and feed a third-party SIEM. The design facilitated continuous integration and delivery in addition to creating a standardized baseline cloud Infrastructure across BBVA USA.

1. Lambda function runs on a cron schedule to scan for new tickets in a specific Jira queue and sends relevant account information to an SNS topic.
2. SNS sends the message to an initial Lambda function that assumes a role in the primary Organization account and runs the create-account API call.
3. The returned account information is added to the event message and passed into the Step Function service.
4. A ten-minute wait is started to allow the accounts to register with all needed services.
5. The first function moves newly created accounts into the correct OU in Organizations based on its environment type (dev, test, prod).
6. Additional OU and account-specific actions are taken in the primary Master account.
7. The support API call requests that the new account be added to the Enterprise Support plan associated with the organization.
8. Complete the execution of remaining, locally-deployed stacks.

The StackSets included standard AWS Identity and Access Management (IAM) roles associated with IAM policies used for assumption by IAM users. With the addition of conformance packs, the collection of pre-defined rules with common regulatory needs are provided as templates. This reduced the work needed by compliance and security teams to map rules to controls with templates provided for NIST, PCI, CIS, and others. Using SCPs naming conventions and tagging standards, the deployment of templates can be done through IT teams at an organizational level without any concern for modification of another teams’ StackSets. This further removes security as a gatekeeper due to access limitations, or concerns.

Conclusion

By automating the account provision and account baselining process, BBVA USA’s security operations team was able to improve the account setup turnaround time. This also enabled the DevOps team and other efforts within the bank such as CI/CD and multi-account strategy. To get started on automating your infrastructure with code visit the AWS CloudFormation documentations.

Removing the manual steps also removed the possibility of human error in addition to a consistent experience for newly created accounts.  The completion from receipt of ticket to the handover of the account to the DevOps team is down from couple of hours to 20 mins. Additionally, this removed routine, manual work from the security team and paved the way for the bank to implement its cloud-first digital transformation strategy.

About the authors

Neel Sendas is a Senior Technical Account Manager at Amazon Web Services. Neel works with Enterprise customers to design, deploy, and scale cloud applications to achieve their business goals. He is also a Machine Learning enthusiast. When he is not helping customers, he dabbles in golf and salsa dance.

John is a Cloud Security Manager at BBVA. John works to implement and operationalize cloud native and third party tools with the existing cloud environment, as well as building the strategy for new tools and services. John can usually be found in the kitchen learning new dishes and cooking techniques when not writing blog posts about his AWS experiences.