bp Scales Its Data Science Machine Learning Operations on AWS
BP p.l.c. (bp) is pivoting from an international oil company producing resources to an integrated energy company delivering solutions for customers, with an ambition to be a net-zero company by 2050 or sooner. Driving digital innovation is a critical component of bp’s strategy to facilitate new ways to engage with customers, create efficiencies, and build new businesses.
For a decade, bp has seen the benefits of using cloud technologies, data analytics, and machine learning (ML) to transform its businesses. bp’s use of data science is growing as an increasing number of business entities use it to extract value from data and inform decision-making. Deploying data science products—such as well sensor data for faster and more accurate production decisions or algorithms that help drive the performance of wind turbines—has rapidly become business and time critical.
In 2020, bp turned to Amazon Web Services (AWS) and engaged AWS Professional Services—which supplements teams with specialized skills and experience—to accelerate data science product delivery at scale through a best practices framework for model management and deployment. Using support from AWS, bp delivered a Model DevOps Framework in 9 months with features that include a serverless architecture, full digital security design, and on-demand compute provisioning. The self-service, cost-effective framework empowers data scientists and engineers within bp to focus on high-value activities, from experimentation to rapid production deployment, by reducing time spent on non–data science activities, such as provisioning and architecture design. Now, the onboarding of data science projects is simplified and accelerated using standardization and support of production models.
The Model DevOps Framework, powered by AWS, creates a standardized approach to provide control, management, monitoring, and auditability of data models while enforcing code quality.”
In-house Data Scientist and Product Owner for the Model DevOps Framework, Innovation & Engineering, bp p.l.c.
Bringing a DevOps Approach to Model Management
bp has a strong architecture and secure-by-design discipline, backed by a digital engineering DevOps culture. However, applying DevOps practices—which combine software development and IT operations—to the lifecycle of data science projects was relatively new. Historically, data scientists experienced challenges accessing cloud infrastructure and aligning with the process required to develop and deploy data solutions at bp. This lack of clarity and standardization often led to bespoke solutions for each project.
bp wanted to adopt a more structured approach to make the initiation of data science projects in the cloud and the deployment of models to production simpler and more efficient, transparent, and robust. At the same time, bp aimed to increase engineering rigor, bridge the data science / engineering divide, build enduring capabilities, and facilitate reusability of code, models, data, environments, and APIs. The company decided to use a DevOps approach for model management and deployment that works alongside existing bp services and tools.
“The Model DevOps Framework, powered by AWS, creates a standardized approach to provide control, management, monitoring, and auditability of data models while enforcing code quality,” says Chris Coley, In house data scientist and product owner for the model DevOps Framework within innovation & engineering at bp. “It gives our data scientists the ability to self-provision environments to experiment and roll out products more quickly without requiring cloud engineering resources.”
bp began work on the framework in October 2020 using support from AWS Professional Services. After establishing an initial proof of concept, the team decided to extend digital security capabilities beyond the required security parameters and incorporate architectural enhancements for a more seamless self-service integration with existing bp systems. By July 2021, bp’s framework was deployable and fully integrated with its Data Hub to virtualize data and perform data handling automatically. “Using AWS Professional Services saved us a lot of time in learning and implementing the technology,” says Coley. “The solution itself was the achievement, and the fact that it only took about 9 months to get there is great.” In November 2021, bp’s framework won an industry award in the Infosys Best DevOps Tool / Product of the Year category at the DevOps Industry Awards 2021.
Focusing on High-Value Activities
The solution relies on Amazon SageMaker—which is used to build, train, and deploy ML models—and infrastructure as code. “Using Amazon SageMaker and the tooling surrounding it supports our serverless training environment and offers us access to more advanced, forward-looking solutions,” says Coley. The framework runs on a cost-efficient serverless architecture, so bp can manage its cloud resource usage and use only the compute that it needs for its training jobs. “By having on-demand compute management facility, Amazon SageMaker reinforces cloud resource optimization seamlessly for our data professionals,” says Abeth Go, vice president for data & analytics platforms at bp. “We anticipate Amazon SageMaker to continue delivering new features to enhance model monitoring and maintenance.”
It was also important for bp’s data science platform team to streamline and align bp’s architectural and digital security review process across data science projects and reduce each project’s future maintenance needs. “Data science projects—and ML in particular—can provide very fast returns in terms of analysis and insights, but they can also build up significant ongoing maintenance requirements as more projects are deployed,” says Coley. “This is especially true if each project follows a slightly different architecture and process.”
Now, bp teams can deploy the framework directly using a service management tool. Teams have a clearly defined path to production, and although projects still require digital security reviews, the process will be faster because the overall architecture is standardized. “We hear from a lot of our data scientists that navigating things like digital security and the ongoing model and code maintenance slows them down,” says Coley. “We’re trying to put the power back into the hands of the teams doing the data science with a standardized framework that will scale to support the majority of use cases.”
Having a common framework also creates more consistency in the way that bp monitors and maintains its data models and code, making it simpler to alert data scientists to issues. “We’ll be able to see early warning signs if data in the production environment starts to drift away from the data used to train the model,” says Coley. “We can monitor the situation and then guide a team through the steps it needs to take to resolve the problem.”
Delivering Agility with Maintainability
In 2022, bp will explore solution enhancements in the areas of operational monitoring, explainability and bias, and streaming workloads. With the Model DevOps Framework, bp expects to see cost savings of 5–25 percent across solution design and provisioning activities, and its teams can accelerate data science project delivery at scale. Using AWS-native solutions, bp gains agility with maintainability, helping its teams move quickly and sustainably with data innovation.
BP p.l.c. (bp) has a purpose to reimagine energy for people and the planet. With an ambition to be a net-zero company by 2050 or sooner and to help the world get to net zero, bp has set out a strategy for delivering on that ambition. For more information, visit bp.com.
Benefits of AWS
- Implemented the Model DevOps Framework in 9 months
- Empowers data scientists and engineers to focus on high-value activities
- Accelerates data science project delivery at scale
- Facilitates code quality for developing and running models in production
- Reduces reliance on cloud engineering resources
- Standardized architecture and streamlined the digital security review process
- Helps monitor and maintain data models and code
- Expects cost savings of 5%–25% across solution design and provisioning
AWS Services Used
Build, train, and deploy machine learning (ML) models for any use case with fully managed infrastructure, tools, and workflows.
Amazon Professional Services
The AWS Professional Services organization is a global team of experts that can help you realize your desired business outcomes when using the AWS Cloud.
More Oil & Gas Customer Stories
no items found
Organizations of all sizes across all industries are transforming their businesses and delivering on their missions every day using AWS. Contact our experts and start your own AWS journey today.