Transport for NSW modernizes analytics using AWS Glue cuts data integration costs by 80%

TfNSW's Future Transport Strategy 2056 outlines a 40-year vision to modernize mobility across New South Wales. This long-term masterplan brings together real-time operations, strategic planning, and digital transformation into a single roadmap for the state's transport future.

As part of this effort, the agency's Customer and Mobility Analytics and Insights (CMAI) team plays a central role in delivering the analytics that inform decisions across its transport network. From daily performance metrics to long-term infrastructure planning, the team supports service leaders across rail, ferry, metro, bus, and road operations with timely, trusted analytics.

To meet growing analytics demands, CMAI used a third-party Extract, Transform, and Load (ETL) platform—a data integration solution—to manage more than 400 ingestion workflows. Over time, this approach introduced two critical risks—cost in ongoing licensing and operating constraints with inhibited ability to innovate and deliver new features and services. The platform also required deep expertise that, as it grew in scale and complexity, became concentrated within a small group, creating a dependency risk for operations and change management. When the team needed to make updates or enhancements, there was increasing reliance on a limited pool of specialists and in some instances, need for external support. Both situations introduced unpredictable delays and increased operational risk.

The team set out to modernize its mobility data integration strategy. This would help optimize costs, mitigate risk from concentrated expertise, and give its integrated data and analytics team more control over day-to-day development. The shift would also help advance the agency's long-term goals for real-time insights, agile service planning, and modern transport infrastructure across New South Wales.

To reduce operational risk and lower integration costs, the team migrated its ETL workloads to AWS Glue, a serverless data integration service. Through this initiative and in using AWS Glue in conjunction with additional serverless compute and orchestration services, TfNSW were able to cut CMAI annual data integration costs by ~80 percent. Because AWS Glue supports Python—a widely understood programming language—the team was also able to distribute expertise across several existing team members. These members now manage data workflows independently, eliminating reliance on specialized knowledge or development partners and accelerating delivery.

The team created reusable transformation patterns using key AWS data services which significantly streamlined the development process for new data ingestion jobs—reducing build times from several months to a few days and predictable deployment outcomes. This enhanced the sustainability and scalability of mobility data workflows, supporting more integrations and faster data-driven decisions for both daily operations and long-term planning.

The team also used AWS Step Functions, to improve workflow orchestration and monitoring. This enhanced error handling added retry logic, and increased visibility into processing status, resulting in more fault-tolerant data pipelines. For scalable storage, the team adopted Amazon Simple Storage Service (Amazon S3) as its data lake foundation. The team utilized Amazon S3 to store both raw and processed data, supporting long-term trend analysis across the state's transport network. Standard RDS instances were also used to support backward compatibility as legacy and modern solutions converged within the evolving mobility data ecosystem

During the migration phase, AWS specialists delivered comprehensive training sessions to over 30 team members on modern data organizations, covering not only AWS Glue but also broader data tooling, strategies, and architecture as a whole. TfNSW opened these sessions to anyone interested in the data space across the organization, strengthening internal expertise in modern data practices. As a result, TfNSW's Customer and Mobility Analytics and Insights (CMAI) team accelerated adoption of modern data approaches beyond just the ETL migration.

With its modernized data ecosystem in place, TfNSW delivers near real-time insights across its transport network. Business stakeholders—from operations to planning—rely on trusted, predictable data outputs to inform timely decisions.

For instance, TfNSW's media and communications team can now quickly prepare accurate public updates when service changes impact commuters. This responsiveness helps improve customer experience and supports more informed, proactive service management across New South Wales.

The CMAI team continues to enhance its analytics capabilities, building on a scalable AWS foundation