Inside BBVA’s MLOps transformation: from data platform to scalable ML on AWS

BBVA is continuing to industrialize and scale its Machine Learning (ML) capabilities through the evolution of ADA, its global Analytics, Data and AI platform. The ADA platform currently serves data scientists across seven countries, providing centralized​​ access to development environments, shared data assets, and production deployment capabilities. It also offers standardized ML pipelines that data scientists use to build and deploy models consistently across business units.

As adoption expanded and use cases diversified across the bank, it became necessary to build on this strong foundation to achieve greater flexibility, scalability, and automation. This included empowering teams to build more modular and reusable ML pipelines tailored to specific business needs, as well as support faster experimentation and delivery.

To implement this framework, BBVA leverages AWS technologies that provide the technical foundation while supporting the bank’s high standards for governance, risk management and auditability.

Challenges

ADA’s ML workflows, built on a robust and well-governed foundation, successfully support the platform’s growth. As the scale and complexity of use cases increased, new opportunities for optimization and scalability emerged.

On the development side, data scientists leveraged standardized scaffolding and notebook-based workflows that enabled rapid prototyping and early value delivery. As adoption grew, the focus shifted towards enhancing modularity, improving code reuse, and strengthening reproducibility and version control practices across teams. Aligning development patterns across business units became a natural step to ensure consistency and long-term maintainability at scale.

At the same time, there was an opportunity to further empower data scientists with greater autonomy to experiment and iterate, while maintaining the platform’s stability and governance standards.

From an operational perspective, the definitions of development and governance processes is clear and effective, but operate as separate workflows. As the number of models and projects increased, streamlining lifecycle management through greater automation and integration became a priority. Consolidating related assets (training pipelines, models, inference pipelines) into unified project structures and automating approval and audit processes would reduce coordination overhead and enhance traceability.

To address these opportunities, BBVA and AWS collaborated to design and implement a modern MLOps framework built on Amazon SageMaker AI, introducing cloud-native automation and reinforcing best practices across the entire model lifecycle.

Solution overview

The MLOps architecture provides the infrastructure and tooling required for data scientists and machine learning engineers to build ML pipelines for training, batch inference, and monitoring, while maintaining governance and traceability.

Within ADA’s platform, a user triggers the creation of a new project by selecting an MLOps template asset from a catalog. This helps teams across countries and business units work consistently from the beginning of each project. This deployment collects important regulatory metadata, such as the model description, business owner, and model tier associated with the models that will be developed.

Each project automatically provisions a dedicated GitHub repository with a standardized code scaffold based on SageMaker Pipelines for training models, registering them in Amazon SageMaker Model Registry alongside metrics and metadata.

The platform enables a fully automated ML development workflow, where standardized building blocks and self-service capabilities allow teams to develop, test, and deploy ML solutions independently, while relying on a shared, production-ready foundation.

It also provides increased flexibility, shifting from a monolithic to a loosely coupled approach.

This lets teams focus on delivering business impact, rather than managing infrastructure and orchestration complexity. Instead of implementing end-to-end logic in a single step, users can compose reusable pipeline components, parametrize them, and iterate more efficiently.

To reduce testing time, the solution deploys dedicated ephemeral development workflows to test new features before merging them to the main branch. See the Ephemeral development workflows section for more details.

The development and production environments transition through GitHub Actions, which are attached to the associated repository branches in each environment. Amazon EventBridge and AWS Lambda work together to register the model in the production environment, following the approval workflow established by the model governance policy.

Figure 1 – MLOps architecture on AWS

Workflow

The following workflow shows how these capabilities are applied end to end within the platform.

The end-to-end workflow for implementing ML models follows these steps:

1. A data scientist creates a project in the ADA console. The platform automatically provisions the project resources, such as a dedicated GitHub repository and the necessary CI/CD workflow with GitHub Actions.
2. Users create and work in a feature branch and commit their code.
3. Users create pull requests (PRs) against the main branch for sandbox deployments.
4. The status of the PR triggers the GitHub Actions. While the PR remains open, a new ephemeral environment is created, provisioning temporary resources that allow users to test their new feature in the sandbox.
5. When a developer merges the PR into the main branch, GitHub Actions remove the ephemeral environment and deploy all the SageMaker Pipelines in the sandbox account.
6. Once the developer merges the PR into the main branch, deploying those resources to the Production account requires to first generate a manual release from main. In this case, the platform provisions only the inference pipelines, centralizing the deployment.
7. During the execution of a training pipeline and registration of a new model, a user promotes the model version as production-ready, triggering the model approval workflow. See section Approval workflows.
8. Downstream applications then consume the approved models and inference pipelines.

Ephemeral development workflows

One of the most transformative innovations in BBVA’s MLOps evolution has been the introduction of ephemeral development workflows. This approach redefines how data scientists and machine learning engineers validate and iterate on their work by moving from merge-based validation to a branch-based model that supports faster experimentation while maintaining quality standards.

Before implementing ephemeral workflows, validating changes required merging code into shared branches (main branch), introducing additional friction in the development cycle. This highlighted the need for more flexible validation mechanisms that supported faster iteration without impacting shared environments. Moreover, this iterative development generated a new pipeline named after a sequential version for each merge to main, resulting in the proliferation of unused pipelines as the version number increased. This sequential approach introduced inefficiencies and potential risks in development cycles.

The ephemeral workflow implementation leverages GitHub Actions and SageMaker to create isolated, temporary resources for each pull request. When a developer opens a PR, the CI/CD platform automatically provisions a complete set of resources (SageMaker Pipelines, SageMaker Model Package Groups, and associated resources) within the project’s sandbox account using a naming convention like {PROJECT_NAME}-pr-{PR_NUMBER}.

The platform now introduces SageMaker Pipelines management through a flexible configuration system that allows teams to enable or disable specific pipelines, preventing deployment even when code exists in the repository. It also enables automated pipeline execution upon creation or update, with the ability to specify execution parameters directly in the configuration, accelerating the development feedback loop. Most importantly, it supports environment-specific configurations, allowing teams to define different behaviours for ephemeral workflows, sandbox and production environments. This capability is a key enabler for scaling ML development across teams.

Figure 2 – Ephemeral development workflow lifecycle

Pipeline versioning and resource management

BBVA teams can now manage pipeline evolution more effectively thanks to SageMaker’s new pipeline versioning capability. Instead of creating entirely new pipeline definitions with version suffixes, SageMaker natively generates new versions only when the pipeline definition or the code of the steps changes. This reduces the total number of pipeline resources while improving traceability, and the consolidation improves quota management, helping teams reduce quota consumption and enabling precise rollback and comparison capabilities. This new feature allows teams to track changes in pipeline definitions and compare performance across versions.

While using these new ephemeral workflows, multiple users can work across branches and pull requests, creating resources (pipelines, models, etc.). The ephemeral workflow system implements resource cleanup to prevent cost accumulation. When a PR closes, the CI/CD platform executes a multi-step destruction process: stopping running executions, deleting model versions, removing pipeline definitions, and cleaning up model package groups. This resource lifecycle management has become an important operational discipline within BBVA’s MLOps model. Across pilot use cases, the automated cleanup contributed to cost reductions of 40–55%.

Parallel development and corporate integration

Multiple users or teams can work simultaneously on different features, each testing and running independently in their own isolated environments.

Developers can iterate rapidly, experiment freely, and test complex scenarios with confidence, without impacting other teams or shared resources. This parallelism enables the 20–75% reduction in development time observed across pilot use cases.

While ephemeral workflows provide unprecedented flexibility, they don’t compromise on quality or governance. The GitHub workflows that deploy ephemeral resources fully integrate with BBVA’s corporate validation framework, triggering quality checks including SonarQube integration, security vulnerability scanning, compliance verification, automated testing, and mandatory code reviews. BBVA has integrated these MLOps-specific workflows into its centralized repository of GitHub Actions, making them easily reusable across all MLOps projects throughout the bank and ensuring consistent quality while reducing effort for new projects.

Model governance framework

BBVA has embedded model governance directly into the ML lifecycle. Approval rules, separation of duties, risk-tier-based promotion, and auditability are no longer handled as disconnected manual steps, but as part of the platform workflow itself. SageMaker Model Registry integrates with the bank’s model governance framework to enforce these rules programmatically while maintaining a streamlined, controlled experience for data scientists.

The framework addresses three core challenges:

• Ensure that no data scientist can approve their own model (self-approval prevention).
• Automate promotion decisions based on model risk tier.
• Maintain a centralized audit trail of every model lifecycle event.

These controls help ensure that all model lifecycle transitions are traceable, auditable, and aligned with BBVA’s internal risk management policies.

Model lifecycle stages

Models registered in SageMaker Model Registry progress through four stages: Development, QA, PreProduction, and Production. Each stage, paired with a specific status (InProgress, PendingApproval, Approved, or Rejected), determines the allowed transitions and the responsible role for each action. Refer to Staging Construct for your Model Lifecycle for additional information.

The diagram in Figure 3 illustrates the allowed stage and status transitions.

Figure 3 – Model approval workflow

All data scientists within a sandbox account share the same AWS Identity and Access Management (IAM) execution role in SageMaker Studio AI (refer to Part 3: How BBVA built a global analytics and machine learning platform on AWS for role configuration details). To distinguish individual users, the platform uses sourceIdentity, an attribute propagated through AWS Security Token Service (STS) that identifies the individual user behind each API call made from Studio. The reactive approval workflow uses this identity to enforce separation of duties across the model lifecycle.

Approval workflows

The governance framework implements different approval workflows, each addressing a different governance concern.

Auto-approval workflow
Production models frequently require retraining because of performance degradation, updated datasets, or changing business requirements. Requiring a full approval cycle each time prevents teams from keeping models current.

The auto-approval workflow addresses this: if an already production-approved model finishes training and has not materially changed, the workflow automatically approves the new model version without repeating the governance process. And if the pipeline definition has changed, the model must go through the full approval cycle.

Auto-promotion requires meeting all the following conditions:

• The training pipeline creates the model package with Development/InProgress status.
• The Git branch associated with the model package is main.
• A previous version of the same SageMaker Model Package Group exists with Production/Approved status.
• That previous production-approved version was generated with the same training pipeline version as the current one.

Manual-based approval workflow
When auto-promotion conditions are not met, a data scientist requests QA approval by updating the model lifecycle from Development/InProgress to QA/PendingApproval. A different data scientist reviews the model and either promotes it to the next stage or rejects it.

Once a model reaches PreProduction/PendingApproval, the platform automatically evaluates whether the Business Owner approval step can be skipped based on the model’s risk tier. The workflow automatically promotes lower-risk models to Production. And models with higher risk require another data scientist to promote them.

Centralized approval workflow with AWS Step Functions

ADA built a centralized reactive approval workflow using AWS Step Functions. Every CreateModelPackage or UpdateModelPackage action in SageMaker Model Registry triggers an EventBridge event, which starts a Step Functions state machine that validates the action in real time and takes corrective action when needed.

Sandbox accounts report model lifecycle state to an Amazon DynamoDB table in a central account. This table stores the stage, status, and sourceIdentity for every model package across sandbox accounts, enabling transition validation, self-approval detection, and rollbacks.

The diagram in Figure 4 shows the simplified flow of the state machine.

Figure 4 – AWS Step Functions model approval workflow

At a high-level, the state machine performs the following steps:

1. An EventBridge event triggers the state machine when a model package is created or updated in SageMaker Model Registry.
2. Guard checks verify the event does not originate from a previous rollback, the model belongs to an MLOps project, and the action has a valid sourceIdentity.
3. The state machine retrieves the previous lifecycle state from the centralized DynamoDB table.
4. For UpdateModelPackage events, it detects self-approval scenarios. For CreateModelPackage events, it evaluates whether the model can be automatically promoted to production.
5. The workflow validates the transition against the responsibility matrix and rolls back any invalid transitions.
6. By meeting the tier-based skip conditions, the model is auto-promoted to production.
7. Persist the model lifecycle state to DynamoDB and send an event to the central account for notifications and auditing.

Notifications across the model lifecycle
One of the most important process improvements is the tighter integration between ADA and the Model Risk Management platform. BBVA has reduced manual handoffs between development, governance, and risk management by automatically notifying key lifecycle events, such as project initialization or production promotion. This keeps both platforms aligned and improves traceability across the full model lifecycle.

Conclusion

BBVA’s MLOps transformation shows how modernizing ML practices can deliver tangible business value when applied to real use cases.

Across four pilot initiatives: risk modeling, pricing optimization, personalized recommendations, and financial forecasting teams achieved a 20–75% reduction in development time and a 40–55% reduction in costs, depending on the use case and initial level of MLOps maturity. These improvements were driven by a new MLOps operating model within BBVA: standardized project creation, ephemeral validation environments, and automated CI/CD workflows enable teams to move faster while maintaining the level of control required in a regulated banking environment.

Beyond efficiency gains, the platform embeds governance, compliance, and traceability by design. Centralized governance and automated approval workflows reduce operational risk while supporting alignment with regulatory requirements.

More importantly, this transformation represents a shift toward treating machine learning as a first-class engineering discipline within BBVA, supported by platform capabilities that enable scale, consistency, and continuous innovation.

In Part 2 of this series, we will explore the use cases that enabled these results and the next set of enhancements driving further improvements, including capabilities that will further improve model reliability and enable more proactive lifecycle management as adoption continues to scale.