AWS Cloud Operations & Migrations Blog
How BBVA automated responses through event management at scale
In this blog post, we describe how BBVA USA, a financial institution that ranks among the top 25 largest commercial banks, used AWS services to implement event management at scale and centralize its event response. Generally speaking, security compliance in a monolithic environment is easier to monitor and enforce when a small number of hands are administrating infrastructure. But as infrastructure moves more to code, a democratized and decentralized approach allows for drift and shadow configurations to find their way into environments without compliance oversight.
By identifying known good states of resources and then managing the deviations through notifications, you can use security event notification and baseline configurations to ensure visibility or enforce adherence through automated remediation.
AWS provides companies like BBVA USA with services that help efficiently secure their cloud platforms. BBVA USA deployed an event management solution at scale by identifying and maturing a standard design that uses Amazon EventBridge. Their solution offers a repeatable approach to managing events from numerous native security and compliance services.
About BBVA USA
In the United States, BBVA operates 641 branches, including 330 in Texas, 89 in Alabama, 63 in Arizona, 61 in California, 44 in Florida, 37 in Colorado, and 17 in New Mexico. The bank ranks among the top 25 largest U.S. commercial banks based on deposit market share. It ranks among the largest banks in Alabama (2nd), Texas (4th), and Arizona (6th). In the U.S., BBVA has been recognized as one of the leading small business lenders by the Small Business Administration (SBA). It is also ranked 14th nationally in terms of dollar volume of SBA loans originated in fiscal year 2019.
Challenge
To reduce the management complexity of handling multiple services uniquely, BBVA USA decided to use an event management pipeline. By deploying a pipeline for event management services like CloudTrail, GuardDuty, Macie, and AWS Config, events can be aggregated and acted on from a centrally managed operational account. CloudTrail can log API calls with detail-rich events that provide an audit trail for actions taken by any identity in the account. GuardDuty and Macie both use machine learning—GuardDuty to provide deviations from expected behaviors and Macie to provide data analysis of object storage. AWS Config makes it possible to analyze resources and detect deviations from established baselines through defined (or custom) rules. You only need to determine which actions should be taken on the noncompliant resource.
By first creating a custom EventBridge event bus in the operational account, events sent from other accounts are responded to through notification and near real-time remediation. After this route is opened, EventBridge rules are deployed to match on AWS Config noncompliance changes, CloudTrail high-risk events, and findings from Macie and GuardDuty to an operational account for action.
Lambda functions are then used in a central operational account to accomplish remediation through cross account assumption of roles into member accounts. This allows the deployment of Lambda remediation code in a central location instead of managing it at the member account level. AWS CloudFormation enables the deployment of the resources required in member accounts to support the solution. This ensures the design is handled as a baseline at the time of account creation.
Services used in the solution
Architecture
The following section includes diagrams that show the interaction between the centralized operational account, member accounts, and AWS services.
Event pipeline backbone
In the operational account, BBVA US deployed a custom event bus in EventBridge with the permissions to allow other accounts in the organization (or specific accounts) to send events to this bus. This account has a rule for each service from which events are being matched. This allows all rules to be developed at the account level without coordination of rules across multiple buses.
Figure 1: Interaction between EventBridge rules in member accounts and event bus in operational account
1. In each member account in the organization, EventBridge rules are deployed to match high-risk or noncompliant events from various services.
2. The events are passed to a custom event bus in a centralized operational account.
3. On the event bus, rules are implemented for each service that generates events because these events do not have the same standard format.
Inputs
In the member accounts, event-based rules are deployed for each service-specific response. GuardDuty and Macie provide a native delegated administration solution which allows for the aggregation into the operational account without per account management of rulesets. Matching only noncompliant events in AWS Config makes it possible to handle only the data that has been deemed meaningful. CloudTrail provides insight into identified, high-risk events.
For API calls that are required for operations, but high risk enough to trigger further investigation or notification, BBVA USA relied on CloudTrail’s integration with CloudWatch to provide EventBridge the ability to respond with rules for each event pattern that requires a response, similar to AWS Config.
Using the operational account as the delegated administrator for Macie and GuardDuty allows events from these services to be aggregated in that account. This means the event rules exist in the operational account and do not have to be passed across accounts.
Figure 2: Interaction between delegated administrator account, aggregated events, and GuardDuty and Macie
1. The administrator functionality in GuardDuty and Macie has been delegated to the operational account to ensure that the services are enabled and collecting data at the account level.
2. Through the delegated administrator account, events are aggregated in the operational account with all data relevant to the event, including the account of origin.
3. Matching to rules that capture GuardDuty and Macie findings in the operational account provides a point of integration with these services.
AWS Config rules are deployed to member accounts as the accounts are created. Each rule that has an automated response has a corresponding EventBridge rule with an event pattern to match on noncompliance of the deployed rule.
{
"detail-type": [
"Config Rules Compliance Change"
],
"detail": {
"configRuleName": [
"AutomatedResponse_PublicIP"
],
"messageType": [
"ComplianceChangeNotification"
],
"newEvaluationResult": {
"complianceType": [
"NON_COMPLIANT"
]
}
},
"source": [
"aws.config"
]
}
Figure 3: Interaction between AWS Config rule and Eventbridge rule
1. An AWS Config rule changes the compliance state to noncompliant and triggers a notification.
2. The member account has an EventBridge rule that monitors for changes that are noncompliant. The rule is sent to the operational account’s event bus.
3. The event bus in the operational account has a rule that matches on the service and then passes the message for further processing.
Operational execution
When the events are matched to a rule, the event message is passed across accounts into the operational account’s custom event bus and matched to a rule for each service. The purpose of accepting all events from a service indiscriminately is to reduce the need for managing matching rules in the operational account. This provides an input for ingestion without requiring the deployment of matching rules in the centralized account. By providing the custom event bus to a team, you can add high-value security and compliance alerts without a constant feedback loop. This ensures that any delay in building automation or notification does not result in the loss of any high-value events.
{ "source": [ "aws.config"] }
After these events are aggregated in the operational account, EventBridge passes these events to appropriate endpoints based on the use case. For events that need to be ingested by third party tools, Amazon Simple Queue Service (Amazon SQS) and Amazon Simple Notification Service (Amazon SNS) provide methods, respectively, for being pulled or pushed from, depending on the solution or use case. For SIEM integration, SQS provided a queue from which to pull. Similarly, SNS allowed integration through direct posts to HTTPS endpoints allowing integration with third party applications for response.
For automated remediation steps, a role is deployed using least privilege based on the needed remediation actions in the member account. With these roles deployed, the operational account executes AWS Lambda functions which invokes an execution role that has sts:AssumeRole permissions to the role deployed in the member accounts. From that role, the needed actions are taken in the member account. As opposed to a distribution of the compute tasks for remediation in each account, the centralized approach reduces the number of resources across all accounts, thereby reducing the complexity and management overhead.
Figure 4: Interaction between Eventbridge rule, its targets, and the assumed execution role
1. A resource generates an event (finding, noncompliance, API call), which is captured by EventBridge event pattern rule.
2. The event message is sent to an EventBridge custom event bus in the operational account that matches rules looking for all events from specific services.
3. The rule has a target of various services including SQS, SNS, and Lambda.
4. When invoked through the event management process, Lambda assumes an execution role in the operational account. The role assumes a cross account role into the originating account.
5. The role deployed into the local account uses least privilege access to take action on the specific use case.
AWS CloudFormation is used to implement auto-deployed, service-managed StackSets, ensuring a standard baseline. This guarantees that all standards are applied at the time the account is created. This means that new accounts are immediately integrated with established and deployed EventBridge rules.
Figure 5: Interaction between member accounts, StackSets, and EventBridge rules
1. Accounts are moved into the appropriate organizational unit (OU) as they are created.
2. AWS CloudFormation StackSets are assigned to autodeploy with service-managed roles to specified OUs.
3. These StackSet instances are used to deploy EventBridge rules and any required supporting resources.
Conclusion
If there is a need, you can use a fanout architecture to send to multiple targets. The solution described in this post matures as event notifications evolve into remediation or ingestion into other solutions for analytics. BBVA USA used AWS services to centralize and automate its event response. The use of EventBridge enables the bridging of services that already have adopted delegated administration for aggregation with the services that have other implementation patterns into a single, standardized approach.
About the authors
Neel Sendas is a Senior Technical Account Manager at Amazon Web Services. Neel works with Enterprise customers to design, deploy, and scale cloud applications to achieve their business goals. He is also a Machine Learning enthusiast. When he is not helping customers, he dabbles in golf and salsa dance.
John was a Cloud Security Manager at BBVA. Working as a Cloud Security Manager at BBVA, John implemented and operationalized cloud native and third party tools with the existing cloud environment, as well as building the strategy for new tools and services. When he is not tinkering with AWS and writing blog posts about the experiences, John entertains himself with science documentaries and learning new cooking techniques.