This post is co-written with Dr. Jens Kohl, Dr. Sirko Molau and Sascha Kallin from BMW Group.

In this blog, we describe BMW’s migration journey, and share the key lessons learned along the way. We also explain how BMW ensured the digital sovereignty of customer data, user interfaces, and customer-facing functions throughout the migration. We also outline the automated workflows and processes BMW implemented to effectively scale governance of the migrated workloads.

BMW Connected Company is a division within the BMW Group responsible for developing and operating premium digital services for BMW’s connected vehicle fleet, which currently numbers more than 23 million vehicles worldwide.

In 2019, BMW Connected Company made the strategic decision to migrate the entire ConnectedDrive backend, comprised of over 1,300 microservices, from on-premises to the AWS Cloud. The rationale behind this decision was to improve reliability, availability, scalability, and performance of BMW ConnectedDrive services. In addition, BMW aimed to reduce time-to-market and lower operational overheads to enable BMW DevOps teams to remain focused on developing premium digital services for customers.

Migration Approach

BMW’s migration strategy was structured in three major steps as illustrated in Figure 1.

Figure 1: Three-step cloud migration strategy

Step 1 – Prepare

The preparation for the BMW Connected Company’s cloud migration comprised of three key elements.

1. Standardize – BMW established a central standardized architecture and set of services for teams to leverage.

2. Refactor – DevOps teams refactored applications from predominately monoliths to microservice architectures.

3. Containerize – BMW transitioned from virtual machines to an OpenShift (Kubernetes-based) platform.

The goal of these efforts was to transform the monolithic backend into a scalable, domain-driven, microservices architecture, ready for migration, as depicted in Figure 2.

This initial work started in 2015, with services being replatformed on OpenStack while still running within BMW’s own data centers. This helped reduce technical complexity and dependencies, while providing efficiency gains through the transition to a microservices-based architecture.

Figure 2: Refactoring and replatforming of the Connected Vehicle on-premises technology stack

BMW also extended the Platform-as-a-Service (PaaS) layer and provided standard solutions for DevOps teams including:

• Runtime – OpenShift as runtime environment for microservices based on Kubernetes, highly integrated with standardized CI/CD pipelines for seamless integration, deployment and runtime behavior.
• CI/CD – BMW tailored and standardized CI/CD platform based on Jenkins, enabled DevOps teams to harmonize the deployments of all microservices across BMW Connected Company.
• APM – Application Performance Monitoring (APM) solution based on Dynatrace to measure operations quality, performance, and alerting.
• Logging – A logging framework based on ElasticSearch, Logstash and Kibana (ELK) to support DevOps teams with a well-defined log analytics platform across hundreds of independent microservices.
• Monitoring/Alarming – Standardized monitoring and alarming solution based on Prometheus and Grafana, that all product teams can rely on.

Applications were packaged in Docker containers using BMW’s Microservice Reference Project (MRP) that seamlessly integrated services with the PaaS layer, providing DevOps teams with a blueprint to standardize the implementation of their services and an agnostic environment to simplify the migration.

Step 2 – Migrate

The BMW ConnectedDrive cloud migration journey was structured in the steps as illustrated in Figure 3.

Figure 3: Migration process steps

1. Migration strategy – BMW initiated a project to define a comprehensive cloud migration strategy. This approach, combined with a clear cloud-first directive set by BMW’s top-level management, was a key factor in ensuring a fast and efficient cloud migration.

The project team’s tasks were to define the technical standards and target architecture for the migrated applications, as well as provide support and training to DevOps teams on best practices. Since BMW had to start from scratch, the team began by working with the most experienced DevOps teams, to collate and leverage common learnings along the way to inform standards. This structured and strategic approach enabled BMW to lay the foundations for a successful and smooth transition to the cloud.

2. Technical Standards – BMW defined a technical solution space, with the refactored infrastructure serving as the foundation for the target architecture as illustrated in Figure 4. The team reused most of the existing technical standards and extended them with standard AWS services such as Amazon Elastic Kubernetes Services (EKS), Amazon Relational Database Services (RDS) and AWS Lambda.

To streamline service deployment to AWS, BMW provided teams with a PaaS based on Amazon EKS, known internally as Orbit. This standardized runtime environment provided seamless integration with CI/CD pipelines and central observability solution which both streamlined service deployment and runtime behavior. In addition, Orbit also addressed the complexities of building a secure, resilient, and high-performing network communication architecture, and delivered common functionality for monitoring, alerting and Identity and Access Management (IAM), allowing DevOps Teams to focus on application development without concerning themselves with underlying infrastructure intricacies.

Figure 4: BMW Connected Vehicle cloud technology stack

3. Establish Specialist Teams – BMW ramped up the project team, and enlisted support from AWS architects to help guide teams through the migration journey and perform Well-Architected Reviews to help ensure applications are aligned to the six pillars of the AWS Well-Architected Framework.

4. Provide Building Blocks – The project team provided rules, guidelines, best practices, and cloud building blocks as Infrastructure-as-Code (IaC) for the DevOps teams, as most were new to the cloud. This helped reduce the complexity of choice and allowed teams to immediately start building and deploying their infrastructure.

5. Application Migration Workflow – BMW streamlined the workflow to scale the migration, while ensuring it remained fast and agile so that teams could incorporate the necessary user stories into their existing development work as illustrated in Figure 5.

Figure 5: BMW streamlined migration workflow 

6. Support and Training – The project team developed and provided hands-on trainings to enable DevOps teams with their first steps in the cloud, including the use of the project’s tool stack. The teams provided valuable feedback to help identify gaps and improvements in the training materials and lessons learned.

7. Establishing Communication Channels – BMW provided an information flow via community communication channels and offered customized training to support the DevOps teams throughout the migration journey.

8. Accelerate via Migration Parties – The team organized various migration parties based on the AWS Experience-Based Acceleration (EBA) offering, which are intensified training sessions aimed at migrating and deploying a fully working service to a project’s tech stack within a tight timeframe.

Step 3 – Optimize

Since completing the migration, BMW continues to optimize the ConnectedDrive backend to improve quality while at the same time cutting costs.

To address these points, BMW is taking the following actions.

1. Develop and operate overarching platform services – BMW has established a centralized platform team responsible for developing and operating shared services and tools that reduce complexity for the various DevOps teams.

2. Improve security – BMW integrated AWS Security Hub and the WIZ security platform to enhance the security posture through continuous monitoring of cloud resources, and to streamline the identification and remediation of security risks.

3. Automate cost optimization and FinOps workflows – BMW implemented mechanisms to automatically optimize resource utilization and reduce costs, and established FinOps practices to gain visibility into cloud spending and proactively manage costs.

4. Enhance resilience – BMW leveraged Multi-AZ (Availability Zone) deployments to ensure ConnectedDrive services can withstand failures and maintain availability for customers. This is regularly tested with fault experiments, executed in production environments. To learn more about how BMW’s Chaos Engineering journey, see this AWS blog post: Engineering Resilient Systems: BMW Group’s Chaos Engineering Journey and Insights.

5. Automate governance workflows – As workloads scaled, BMW identified the need to streamline governance-related tasks, such as policy enforcement, compliance monitoring, and architecture guidance. To address this, BMW developed a cloud governance tool and Generative AI powered architectural guidance bot to assist DevOps teams in making informed decisions and adhere to best practices, as detailed in AWS blog posts: How BMW Group uses automation to achieve end-to-end compliance at scale on AWS and BMW Group Develops a GenAI Assistant to Accelerate Infrastructure Optimization on AWS.

Lessons Learned

BMW Connected Company learned several valuable lessons from this migration journey.

Strategy and Organization

The cloud migration was supported by a clear cloud-first strategy set by BMW leadership which empowered project teams to efficiently push the transition forward efficiently, ensuring issues could be resolved quickly. DevOps teams found they could not perform the cloud migration in isolation. They needed to strike the right balance, leveraging clear leadership direction to empower teams while ensuring the migration work was properly prioritized and coordinated against existing development objectives.

Migrating to AWS also influenced several operational changes. Functions such as backups, infrastructure provisioning, and scaling which were handled by central teams, could now fall under the direct influence and ownership of individual teams. This meant the teams now owned the entire cloud architecture, making them more autonomous. This additional control and flexibility also required them to choose all of the service offerings themselves, rather than relying on a centralized IT team. This shift in ownership and responsibility required the DevOps teams to learn new skills and develop new processes to effectively manage the cloud infrastructure and services.

This highlighted the importance of balancing strong leadership direction with team empowerment, while also emphasizing the need for coordination with existing development goals. The shift to AWS necessitated operational changes, increasing team autonomy but also requiring new skills and processes to manage the expanded responsibilities of cloud infrastructure ownership.

Training

BMW ensured extensive training programs and support channels were made available to DevOps teams early on to help ensure a smooth transition. In addition, the cloud migration project team established walk-in sessions to raise questions and obtain dedicated support from more experienced colleagues. This fostered team collaboration and knowledge sharing which accelerated teams understanding of AWS services and how to effectively architect their services, covering not only the standard use cases but also more complex scenarios.

Partner with AWS

While training established foundational knowledge, understanding the wide range of AWS services still proved challenging for teams new to the cloud. Therefore, to bridge knowledge gaps and accelerate understanding, teams engaged with AWS who responded by facilitating deep-dive sessions and architecture reviews to help ensure solutions aligned to the AWS Well-Architected Framework from the start. As experience increased, teams documented insights, processes, and best practices to scale consistent knowledge sharing across other migration teams. Additionally, teams leveraged AWS Enterprise Support to help ensure a smooth transition and rapidly troubleshoot any issues that arose post-migration.

Technical Architecture

Whilst the AWS Cloud offers numerous advantages over on-premise infrastructure, it’s crucial to remember that the underlying infrastructure is not infallible by default. The mindset of “everything fails all the time” is essential, and the adoption of key high availability concepts, such as Multi-AZ and multi-Region designs are key to building resilient services.

Concepts such as self-healing, auto-scaling, and anomaly-based monitoring were new to teams without prior cloud experience and therefore were not consistently leveraged across all migrated services. As a result, teams identified significant optimization potential during the third project phase which allowed them to fully realize the added benefits that AWS can provide.

Furthermore, the migration involved several large-scale databases that were multiple terabytes in size. Teams closely monitored AWS Database Migration Service (DMS) replication task metrics and adjusted the tuning parameters accordingly to ensure the performance and availability of dependent applications were not adversely affected. Once migrated, teams carefully monitored AWS CloudWatch metrics to ensure Amazon RDS instances and Amazon Elastic Block Store (EBS) volumes were sufficiently sized and the appropriate performance parameters were applied. This proactive monitoring and tuning helped maintain the performance and availability of migrated databases and applications. 

Conclusion

The successful migration of more than 1,300 microservices to AWS enabled BMW Connected Company to improve the reliability, scalability, and performance of critical digital services. The key to this success was a structured and phased approach, combined with a strong focus on empowering their DevOps teams, establishing clear architectural guidance, and automating critical workflows.

BMW was able to lay the groundwork for a smooth cloud migration by replatforming applications to a Kubernetes-based environment and rearchitecting to support a microservices-based design. Through close partnership with AWS, along with the establishment of scalable training programs and communication channels, BMW ensured DevOps teams were equipped with the necessary skills and resources to navigate the transition.

The lessons learned throughout this journey, from the importance of management buy-in to the power of automation, can help other organizations embarking on their own cloud migration strategies.

Overall, this successful migration to the AWS Cloud positioned BMW to accelerate their development, increase reliability, and be more agile and innovative. As illustrated in Figure 6, these improvements enable BMW to continue delivering exceptional digital experiences for over 23 million connected vehicle customers.

Figure 6: KPIs of the BMW ConnectedDrive backend (Status: 10/2024)

About the Authors