TC Energy Builds an Operations Data Platform for 60,000 Miles of Pipeline Using AWS Data Analytics

TC Energy was founded in 1951, and since then, the conglomerate has come to own and operate established pipelines across North America through a series of acquisitions. Its pipes traverse thousands of miles, crossing plains, deserts, and mountains to provide vital energy to consumers across the continent. The sheer scale of its operations creates an urgent need for data insights. For example, when data silos prevented project planning teams from getting a unified overview of planned work, it was difficult to plan and implement projects efficiently. At the same time, TC Energy’s commitment to responsible stewardship challenged employees to minimize the disturbances that work crews can cause for local communities when it is necessary to take multiple trips to pipeline locations.

TC Energy has already had significant success migrating existing systems to AWS. In March 2021, the company decided to look into what benefits could be realized from a centralized operations data lake. It knew that this would be a highly complex task because discovery alone involved over 140 internal stakeholders who connected data, processes, and people. The first step was to bring all these sponsors together to lay out and prioritize 11 unique use cases related to improving operational excellence. Then, TC Energy selected Pariveda as an implementation partner. “Pariveda has been very important in implementing a data lake that is based on industry best practices and architecture,” says Ali. On Pariveda’s recommendation, TC Energy opted to develop its new data lake with serverless architecture using AWS Glue, a serverless data integration service that makes it simpler to discover, prepare and integrate data from multiple sources into secure data lakes. It also used Amazon Athena, a service for analyzing petabyte-scale data where it lives.

The complexity of TC Energy’s O360 implementation necessitated an incremental approach that used a fleet of AWS services. First, the data is ingested and sent to Amazon Simple Storage Service (Amazon S3), an object storage service offering industry-leading scalability, data availability, security, and performance. Then, TC Energy processes data using a pair of solutions: Amazon EventBridge, a serverless event bus, and AWS Glue. Pariveda used the Modern Data Enterprise Framework to transform and enhance the data, which is then queried using Amazon Athena. Finally, TC Energy uses AWS Glue, Amazon EventBridge, and AWS Step Functions, a visual workflow service, to transform the data.

A unique aspect of the requirements for TC Energy’s operations data lake was the need for advanced geospatial processing. The team evaluated vendor offerings, but most were not yet compatible with the patterns and scalability of a data lake environment. Using Apache Sedona, the team could build a set of libraries for running geospatial transformations within the AWS Glue environment, including geo-hashing, linear referencing, and dynamic segmentation. “The end result is a unified source of data that can be consumed in a number of different ways,” says Derrick Bowen, principal consultant for Pariveda. “We’re also establishing patterns that can be reused across the organization to tackle TC Energy’s diverse use cases.” The cost savings generated by these initial steps are expected to fund the rest of the initiative, which will extend across multiple years.

For a complex, high-value use case, the team developed a rich, interactive web application for planners to collaborate on finding opportunities to bundle capital projects to save costs and reduce environmental and community impacts. The web front end is backed by AWS Lambda, a serverless, event-driven compute service, as well as Amazon DynamoDB, a fast, flexible NoSQL database service for single-digit millisecond performance at virtually any scale. The company selected Amazon API Gateway—a fully managed service that makes it simple for developers to create, publish, maintain, monitor, and secure APIs at virtually any scale—to create and maintain secure APIs for the interactive offering. “Using these AWS services, we’ve created a solution that makes it simple for users to see the data, drill into it, and work together to get value for the business,” says Bowen. 

TC Energy is seeking more than just a financial return for its efforts with O360. “Our number-one value is safety,” says Shane Taylor, business analyst for the US Natural Gas Pipelines business unit at TC Energy. By improving information access and flow, the company expects to improve safety. It also aims to bolster the sustainability of its operations. As the energy industry advances energy transition initiatives, TC Energy is using innovative solutions like O360 to overcome challenges.