bp Scales Its Data Science Machine Learning Operations on AWS

bp has a strong architecture and secure-by-design discipline, backed by a digital engineering DevOps culture. However, applying DevOps practices—which combine software development and IT operations—to the lifecycle of data science projects was relatively new. Historically, data scientists experienced challenges accessing cloud infrastructure and aligning with the process required to develop and deploy data solutions at bp. This lack of clarity and standardization often led to bespoke solutions for each project.

bp wanted to adopt a more structured approach to make the initiation of data science projects in the cloud and the deployment of models to production simpler and more efficient, transparent, and robust. At the same time, bp aimed to increase engineering rigor, bridge the data science / engineering divide, build enduring capabilities, and facilitate reusability of code, models, data, environments, and APIs. The company decided to use a DevOps approach for model management and deployment that works alongside existing bp services and tools.

“The Model DevOps Framework, powered by AWS, creates a standardized approach to provide control, management, monitoring, and auditability of data models while enforcing code quality,” says Chris Coley, In house data scientist and product owner for the model DevOps Framework within innovation & engineering at bp. “It gives our data scientists the ability to self-provision environments to experiment and roll out products more quickly without requiring cloud engineering resources.”

bp began work on the framework in October 2020 using support from AWS Professional Services. After establishing an initial proof of concept, the team decided to extend digital security capabilities beyond the required security parameters and incorporate architectural enhancements for a more seamless self-service integration with existing bp systems. By July 2021, bp’s framework was deployable and fully integrated with its Data Hub to virtualize data and perform data handling automatically. “Using AWS Professional Services saved us a lot of time in learning and implementing the technology,” says Coley. “The solution itself was the achievement, and the fact that it only took about 9 months to get there is great.” In November 2021, bp’s framework won an industry award in the Infosys Best DevOps Tool / Product of the Year category at the DevOps Industry Awards 2021.

The solution relies on Amazon SageMaker—which is used to build, train, and deploy ML models—and infrastructure as code. “Using Amazon SageMaker and the tooling surrounding it supports our serverless training environment and offers us access to more advanced, forward-looking solutions,” says Coley. The framework runs on a cost-efficient serverless architecture, so bp can manage its cloud resource usage and use only the compute that it needs for its training jobs. “By having on-demand compute management facility, Amazon SageMaker reinforces cloud resource optimization seamlessly for our data professionals,” says Abeth Go, vice president for data & analytics platforms at bp. “We anticipate Amazon SageMaker to continue delivering new features to enhance model monitoring and maintenance.”

It was also important for bp’s data science platform team to streamline and align bp’s architectural and digital security review process across data science projects and reduce each project’s future maintenance needs. “Data science projects—and ML in particular—can provide very fast returns in terms of analysis and insights, but they can also build up significant ongoing maintenance requirements as more projects are deployed,” says Coley. “This is especially true if each project follows a slightly different architecture and process.”

Now, bp teams can deploy the framework directly using a service management tool. Teams have a clearly defined path to production, and although projects still require digital security reviews, the process will be faster because the overall architecture is standardized. “We hear from a lot of our data scientists that navigating things like digital security and the ongoing model and code maintenance slows them down,” says Coley. “We’re trying to put the power back into the hands of the teams doing the data science with a standardized framework that will scale to support the majority of use cases.”

Having a common framework also creates more consistency in the way that bp monitors and maintains its data models and code, making it simpler to alert data scientists to issues. “We’ll be able to see early warning signs if data in the production environment starts to drift away from the data used to train the model,” says Coley. “We can monitor the situation and then guide a team through the steps it needs to take to resolve the problem.”

In 2022, bp will explore solution enhancements in the areas of operational monitoring, explainability and bias, and streaming workloads. With the Model DevOps Framework, bp expects to see cost savings of 5–25 percent across solution design and provisioning activities, and its teams can accelerate data science project delivery at scale. Using AWS-native solutions, bp gains agility with maintainability, helping its teams move quickly and sustainably with data innovation.