Customer Stories / Energy
TC Energy Builds an Operations Data Platform for 60,000 Miles of Pipeline Using AWS Data Analytics
Learn how TC Energy in the energy industry is targeting 70 data sources using AWS Glue and Amazon Athena.
With 60,000 miles of pipeline in Mexico, Canada, and the United States, TC Energy invests heavily in operations and reliability to accommodate constantly changing demand and regulations. The company wanted to create a centralized data repository known as a data lake to improve the management of its US natural gas infrastructure by consolidating information from over 70 data sources. TC Energy turned to Amazon Web Services (AWS) and Pariveda Solutions Inc. (Pariveda), an AWS Partner, to facilitate this challenging project.
To break down its ambitious goals into more manageable targets, TC Energy identified 11 use cases for the initiative, which it calls Operations 360 (O360). The team followed an interactive approach to achieve results quickly. “By consolidating data on AWS and bundling pipeline work, we conservatively estimate that we’ll save $1.6 million annually,” says Irfan Ali, director of pipeline integrity data engineering at the US Natural Gas Pipelines business unit of TC Energy.
Opportunity | Using AWS Glue to Consolidate Diverse Data Sources for TC Energy
TC Energy was founded in 1951, and since then, the conglomerate has come to own and operate established pipelines across North America through a series of acquisitions. Its pipes traverse thousands of miles, crossing plains, deserts, and mountains to provide vital energy to consumers across the continent. The sheer scale of its operations creates an urgent need for data insights. For example, when data silos prevented project planning teams from getting a unified overview of planned work, it was difficult to plan and implement projects efficiently. At the same time, TC Energy’s commitment to responsible stewardship challenged employees to minimize the disturbances that work crews can cause for local communities when it is necessary to take multiple trips to pipeline locations.
TC Energy has already had significant success migrating existing systems to AWS. In March 2021, the company decided to look into what benefits could be realized from a centralized operations data lake. It knew that this would be a highly complex task because discovery alone involved over 140 internal stakeholders who connected data, processes, and people. The first step was to bring all these sponsors together to lay out and prioritize 11 unique use cases related to improving operational excellence. Then, TC Energy selected Pariveda as an implementation partner. “Pariveda has been very important in implementing a data lake that is based on industry best practices and architecture,” says Ali. On Pariveda’s recommendation, TC Energy opted to develop its new data lake with serverless architecture using AWS Glue, a serverless data integration service that makes it simpler to discover, prepare and integrate data from multiple sources into secure data lakes. It also used Amazon Athena, a service for analyzing petabyte-scale data where it lives.
Using AWS serverless services, we achieved better scale and performance with reduced costs.”
Principal Consultant, Pariveda
Solution | Building a Data Foundation for Decades of Innovation Using Amazon Athena
The complexity of TC Energy’s O360 implementation necessitated an incremental approach that used a fleet of AWS services. First, the data is ingested and sent to Amazon Simple Storage Service (Amazon S3), an object storage service offering industry-leading scalability, data availability, security, and performance. Then, TC Energy processes data using a pair of solutions: Amazon EventBridge, a serverless event bus, and AWS Glue. Pariveda used the Modern Data Enterprise Framework to transform and enhance the data, which is then queried using Amazon Athena. Finally, TC Energy uses AWS Glue, Amazon EventBridge, and AWS Step Functions, a visual workflow service, to transform the data.
A unique aspect of the requirements for TC Energy’s operations data lake was the need for advanced geospatial processing. The team evaluated vendor offerings, but most were not yet compatible with the patterns and scalability of a data lake environment. Using Apache Sedona, the team could build a set of libraries for running geospatial transformations within the AWS Glue environment, including geo-hashing, linear referencing, and dynamic segmentation. “The end result is a unified source of data that can be consumed in a number of different ways,” says Derrick Bowen, principal consultant for Pariveda. “We’re also establishing patterns that can be reused across the organization to tackle TC Energy’s diverse use cases.” The cost savings generated by these initial steps are expected to fund the rest of the initiative, which will extend across multiple years.
For a complex, high-value use case, the team developed a rich, interactive web application for planners to collaborate on finding opportunities to bundle capital projects to save costs and reduce environmental and community impacts. The web front end is backed by AWS Lambda, a serverless, event-driven compute service, as well as Amazon DynamoDB, a fast, flexible NoSQL database service for single-digit millisecond performance at virtually any scale. The company selected Amazon API Gateway—a fully managed service that makes it simple for developers to create, publish, maintain, monitor, and secure APIs at virtually any scale—to create and maintain secure APIs for the interactive offering. “Using these AWS services, we’ve created a solution that makes it simple for users to see the data, drill into it, and work together to get value for the business,” says Bowen.
TC Energy is seeking more than just a financial return for its efforts with O360. “Our number-one value is safety,” says Shane Taylor, business analyst for the US Natural Gas Pipelines business unit at TC Energy. By improving information access and flow, the company expects to improve safety. It also aims to bolster the sustainability of its operations. As the energy industry advances energy transition initiatives, TC Energy is using innovative solutions like O360 to overcome challenges.
Outcome | Achieving Holistic Business Success Using AWS Services
The results of the early stages of O360 implementation look promising for TC Energy’s future growth plans. “Using AWS serverless services, we achieved better scale and performance with reduced costs,” Bowen says. More than a third of the company’s data sources were added to the new data lake within the first 8 months, and the company now has a solid data foundation for future enhancements. Additionally, as more data sources are ingested, a use case may evolve to focus on identifying emission reduction opportunities in pursuit of TC Energy’s sustainability goals. The team has also identified ways to use the ingested data with advanced machine learning to further continual improvement by using efficiencies in the way that TC Energy operates.
The company has a broad vision for O360. “We know that we have a multi-year journey with this data foundation on AWS,” says Taylor. “But this is a great first step.”
About TC Energy
TC Energy Corporation (TC Energy) is a team of 7,000+ energy problem solvers with 60,000 miles of pipeline across Mexico, the United States, and Canada. Its infrastructure transports 25 percent of all the natural gas consumed in North America.
AWS Services Used
AWS Glue is a serverless data integration service that makes it easier to discover, prepare, move, and integrate data from multiple sources for analytics, machine learning (ML), and application development.
Amazon Athena is a serverless, interactive analytics service built on open-source frameworks, supporting open-table and file formats.
Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading scalability, data availability, security, and performance.
Amazon DynamoDB is a fully managed, serverless, key-value NoSQL database designed to run high-performance applications at any scale.
Organizations of all sizes across all industries are transforming their businesses and delivering on their missions every day using AWS. Contact our experts and start your own AWS journey today.