Scaling ML in production: how BBVA accelerated delivery with MLOps

In Part 1 of this series, we described the MLOps architecture that BBVA built on Amazon SageMaker AI, including ephemeral development workflows and model governance.

While the global architecture was being designed, BBVA worked in parallel with four business units to modernize existing ML use cases. This parallel approach allowed the team to validate architectural decisions against real production requirements from the beginning.

The use cases spanned different domains (risk modeling, pricing optimization, personalized recommendations, and financial forecasting), data scales, compute requirements, and team maturity levels. The common patterns identified across them helped define a reusable MLOps foundation that became the basis for a generic production-ready template, while also shaping the platform capabilities required to support custom templates for different ML workloads.

This post describes how BBVA used these pilots to identify reusable ML patterns, standardize operational workflows, and design extensible MLOps templates that accelerate ML delivery while maintaining governance and flexibility across teams and business domains.

Challenges

Four business units participated in the initial pilots and, despite operating across different domains and geographies, shared opportunities to develop and standardize their ML delivery practices.

While teams already leveraged code repositories, experiment tracking, and model lifecycle capabilities, implementation patterns had naturally evolved independently over time, reducing reuse across teams and increasing onboarding effort for new projects with similar technical foundations.

Operational processes around model promotion and pipeline management also combined automated and manual activities, creating opportunities to improve scalability as the number of use cases grew. The absence of isolated testing environments meant validating pipeline changes required merging to shared branches, slowing feedback loops.

Teams already had governance capabilities such as experiment tracking, model registration, and basic monitoring, but implementation approaches varied across teams. Standardizing these practices offered an opportunity to improve traceability, consistency, and visibility across the full ML lifecycle.

The opportunity was clear: accelerate and industrialize ML delivery by standardizing common patterns into reusable templates, simplifying operational workflows, and embedding governance and traceability directly into the development experience.

Methodology

Modernizing ML workflows across a large organization requires more than introducing new tools. At BBVA, the objective was to define a scalable way of building and operating machine learning that aligned people, processes, governance, and technology.

To achieve this, BBVA and AWS Professional Services jointly conducted a series of discovery and assessment sessions involving several business units. The team conducted these sessions parallel to the MLOps architecture design, embedding business, operational, and governance requirements from the outset. By combining BBVA’s domain expertise with Amazon’s Working Backwards approach, this alignment grounded the resulting architecture in real business needs.

We kicked off with structured discovery sessions across each of the four business units that agreed to be early adopters of the solution. Each business unit brought distinct ML technical use cases (from XGBoost regression to Deep Neural Networks), different development maturity levels, and operational constraints. Rather than prescribing a single path, we used these sessions to map these diverse workflows to identify common patterns, challenges, and divergences.

This dual-track approach was crucial to align real-world business needs with the general MLOps platform being designed. The result was a shared understanding of what “good” looked like across the organization with real data points.

The findings from these discovery sessions fed directly into the design of reusable MLOps templates and standardized operational workflows. As a result, the technical proposal balanced standardization across four dimensions:

• People, where we provided guidance to move from individual notebook work to collaborative, version-controlled pipelines.
• Processes, establishing standards for code review, testing, and deployment.
• Governance, embedding compliance controls, model approval workflows, and traceability to reduce manual oversight and ensure regulatory alignment.
• Technology, integrating the MLOps templates into the ADA platform, to start a new ML project with a proven, production-ready foundation.

Solution overview

As described in Part 1, users select an MLOps asset from the ADA console to provision a new project with a dedicated GitHub repository and standardized code scaffold. This scaffold provides an end-to-end setup for the ML lifecycle, including reusable SageMaker Pipelines for training, inference and data and model monitoring. This means that teams customize for their specific ML use case while inheriting the platform’s built-in governance, experiment tracking, and CI/CD integration.

Training Pipeline

Figure 1: Default training pipeline in the reusable MLOps template.

The diagram in Figure 1 shows the default training pipeline included in the reusable MLOps template. The template shows the model build process’ discrete steps: a data processing step (SageMaker Processing or an Amazon EMR step, depending on the use case needs), a training step, a model evaluation step, a performance-based condition check, both data and model quality checks and the registration step into SageMaker Model Registry. Teams extend this base structure by adding steps for their specific needs, such as additional preprocessing stages.

This modular design enables:

• Step-level caching, which skips unchanged steps between runs, reducing execution time and compute costs during iterative development. For example, when experimenting with different models while the data processing logic remains the same.
• A CPU-first compute strategy where the pipeline uses GPU instances only for steps that require them, while processing and evaluation run on cost-effective CPU instances.
• Python-based execution replacing PySpark where feasible. Not every ML use case requires distributed compute. For those cases, running steps as Python-based SageMaker Processing jobs simplifies the development experience and removes the dependency on EMR clusters. When use cases require large-scale data preparation, teams can combine EMR-backed PySpark steps with Python-based downstream steps in the same pipeline. This hybrid pattern is covered in detail below.
• Conditional model registration based on performance thresholds, promoting only models that meet quality criteria to SageMaker Model Registry.
• MLflow experiment tracking integrated in the training step, logging hyperparameters, metrics, and artifacts to a centralized tracking server with a standardized naming convention that maps each run to its corresponding pipeline execution and Model Registry version.

Figure 2: MLFlow captures metrics from training and visualizes them in dashboards.

Inference Pipeline

Figure 3: Inference pipeline from the MLOps template.

The figure above shows the inference pipeline template. An AWS Lambda step retrieves the latest approved model version from SageMaker Model Registry. Then a processing step prepares the input data, executes the predictions, and stores the results in Amazon Simple Storage Service (Amazon S3). As with the training pipeline, teams extend this base with additional steps. Key capabilities include:

• Automated model version retrieval through a centralized function that resolves the corresponding model artifacts in Amazon S3, removing the need to manage artifact paths manually.
• Seamless model updates: users configure the inference pipeline to retrieve the latest approved model version every time. When a retrained model transitions through the approval workflow described in Part 1 (no material changes in the training pipeline), the inference pipeline selects it automatically without requiring a new production deployment or re-approval of the inference pipeline itself.

Across both pipelines, these design decisions reduced pipeline execution time during iterative development and optimized compute costs by matching instance types to actual step requirements. On the operational side, automated model retrieval and integrated monitoring closed the gap between model deployment and ongoing model health, enabling teams to keep production predictions current with minimal manual coordination.

Extending the MLOps template

While the default template covers the most common ML development pattern, different business domains introduced additional technical and operational requirements. Rather than creating disconnected workflows, we extended the reusable template foundation to support specialized workloads without diverging from the standardized pipeline structure.

GPU-accelerated training for deep learning

One of BBVA’s business units required training several compute-intensive deep learning candidate models in parallel. The goal was to select only the best-performing model dynamically based on the evaluation metrics, thus qualifying this model for registration. The team implemented this scenario by enabling SageMaker Pipelines to run multiple training jobs in parallel. A subsequent pipeline step evaluates all trained models and selects the best candidate according to predefined business criteria, automatically registering the selected model into SageMaker Model Registry.

For such use cases involving deep neural networks or architectures requiring matrix-intensive computation, we created a template that extends naturally to GPU-backed training steps. The key adaptations include:

1. The training step configuration switches to GPU-optimized instance families (such as ml.g5 or ml.p4d), applying exclusively to the training step itself. Processing, evaluation, and monitoring steps continue running on cost-effective CPU instances, preserving the cost-optimization strategy of the base template.
2. The container environment shifts to a CUDA-enabled framework image (for example, PyTorch or TensorFlow with GPU support), while the pipeline orchestration, experiment tracking integration, and conditional registration logic remain unchanged.
3. Parallel training steps, allowing teams to build competing models in shorter times, and maintaining environment isolation during training jobs.

This extension shows a core design principle: the pipeline skeleton (data processing, training, evaluation, condition check, and registration) stays consistent regardless of the compute backend. Teams adopting GPU workloads inherit the same caching, monitoring, and MLflow tracking capabilities as CPU-based use cases, reducing the operational overhead of supporting heterogeneous model architectures.

Figure 4: Example of an extended pipeline from the reusable MLOps template.

Hybrid PySpark and Python pipelines

Some business domains operated on datasets large enough to require distributed processing, while their downstream training and inference workloads could execute efficiently on single-instance Python environments. To support these scenarios, we extended the reusable MLOps template with a hybrid execution model that combines PySpark-based distributed processing with Python-based SageMaker training and inference jobs.

1. The data processing steps leverage EMR steps within SageMaker Pipelines, enabling distributed transformations (joins, aggregations, feature engineering) across large-scale datasets using PySpark, feeding directly from the ADA platform data lake. This step produces a processed dataset stored in Amazon S3 that downstream steps consume.
2. After converting the data to a training-ready format, subsequent steps execute as standard Python-based SageMaker Processing or Training jobs. This avoids provisioning EMR clusters for steps that don’t benefit from distributed compute.
3. The pipeline maintains a clear contract between the EMR and Python stages through well-defined S3 paths and data schemas, enabling teams to iterate on either stage independently without breaking the other.

This hybrid execution model allows teams to combine the scalability of distributed processing with the simplicity and flexibility of Python-based ML development, while preserving a consistent orchestration, governance, and monitoring framework across workloads. It also provides a gradual migration path for teams evolving from fully Spark-based ML workflows toward modular and cost-efficient Python-based execution patterns.

Common extensibility patterns

Both extensions share a set of patterns that any team can apply when adapting the template:

• Instance-level compute specialization: each pipeline step declares its own compute requirements, allowing GPU instances, EMR clusters, and CPU resources to be allocated where they need them.
• Framework-agnostic orchestration: the pipeline definition layer (step sequencing, caching, conditional logic) decouples from the execution framework, whether that is PyTorch on a GPU, Scikit-Learn on a CPU, or PySpark on EMR.
• Consistent governance surface: regardless of the template, every pipeline execution logs to MLflow, registers models through the same Model Registry workflow, and integrates with the CI/CD automation described in Part 1.
• Multi-model pipelines: whether intended for training or inference, pipeline patterns can branch out into parallel paths to train and register or run inference on different models, reducing the total execution time compared to sequential approaches.

These extensions validated that the template architecture could absorb the diversity discovered during the initial assessment sessions, from lightweight gradient-boosted models to GPU-intensive deep learning and large-scale distributed processing, without fragmenting into disconnected, team-specific solutions.

Lessons learned and best practices

The first pilots confirmed that MLOps standardization is most effective when teams build it from real use cases. They helped BBVA identify common patterns across projects while preserving the flexibility required by different data volumes, model architectures, compute needs, and governance requirements.

One key lesson was that templates should act as accelerators, not constraints. The generic MLOps template provides a production-ready foundation for training, inference, data monitoring, and model monitoring pipelines, with CI/CD, experiment tracking, model registration, and governance capabilities already integrated. Teams can extend these pipelines when needed, while still following a consistent lifecycle model.

The pilots also showed the importance of improving the developer experience. Pull-request-based ephemeral environments allow teams to validate changes safely before merging them into shared branches, enabling faster feedback, parallel development, and better collaboration. Combined with automated tests, code reviews, and security checks, this approach improves both speed and quality.

Another best practice was to optimize pipelines at the step level. By assigning the right compute resources to each stage, using caching, and combining PySpark, Python, CPU, and GPU workloads where appropriate, teams were able to reduce execution times and improve cost efficiency.

The team also embedded governance directly into the ML lifecycle. The standard workflow handles model versioning, approval workflows, MRM integration, monitoring, and traceability, reducing manual effort and making compliance easier to follow.

Overall, the first pilots showed that a standardized but flexible MLOps operating model can accelerate development, improve execution efficiency, reduce costs, increase visibility, and strengthen governance. The pilots identified the following best practices:

• Start from real use cases.
• Reuse production-ready templates.
• Enable ephemeral validation environments.
• Optimize resources per pipeline step.
• Integrate monitoring from the beginning.
• Embed governance by design.

Conclusion and next steps

The first wave of MLOps adoption at BBVA confirms the value of moving towards a more standardized, reusable, and governed way of delivering machine learning solutions at scale. By starting from real production use cases, BBVA defined a common MLOps foundation that accelerates delivery while preserving the flexibility required by different business domains, technical patterns, and regulatory requirements.

This new way of working is already helping teams reduce fragmentation, improve reuse, and strengthen how they control, monitor, and operate ML solutions in production. These improvements ranged from 30% to 75%. The lower end came from step caching, which eliminated redundant processing. The higher end reflected migrations from notebook-based workflows to modular pipelines combining CPU-first compute and Python-based execution.

BBVA will continue to develop and improve these capabilities by:

• Expanding adoption across more teams, geographies, and ML projects.
• Enriching the developer experience with generative AI capabilities such as assisted documentation, template generation, and intelligent guidance.
• Building on the same operational foundations to move towards broader AIOps capabilities.

Overall, this evolution represents an important step in BBVA’s journey to industrialize AI and ML delivery. It enables the organization to deliver reliable, governed, and production-ready AI solutions more efficiently, with the ultimate goal of providing better, faster, safer, and more personalized services to customers across the countries where BBVA operates.