Transport for NSW Uses AWS and Machine Learning to Make Real-Time Decisions

Transport for NSW has been using AWS for its customer-facing assets since 2014, choosing AWS for the breadth and depth of managed services available. “A lot of our improvements are done by small, four to five people teams who are skilled across different technical disciplines. That’s only possible because AWS has invested in opening up these services as part of their infrastructure offering,” says Chris Bennetts, executive director of Digital Products Delivery at Transport for NSW.

In October 2018, Transport for NSW began experimenting with machine learning (ML) to transition from historically based analytics to a forward-looking model with predictive capability. One of its small teams had been using the Tableau data visualization tool and homegrown algorithms to deliver analytics reports, but results were always two or three days behind. “Executives didn’t have the pulse of the network, and the analytics team lacked the latest technology tools to help our operators make better decisions today for tomorrow,” explains Mr. Bennetts.

Multiple data sources, including Opal data, feeds from a third-party vendor and takes many hours to ingest—making it difficult to access, process, and analyze datasets at speed. Through the accelerator, the agency worked with Contino, a Premium Consulting Partner in the AWS Partner Network (APN), and Vizalytics Technology, an APN Select Technology Partner, to develop dashboards that deliver near real-time insights and data-driven recommendations. Transport for NSW is now able to predict patronage numbers across the entire transport network, enabling the agency to better plan workforce and asset utilization, and improve customer satisfaction.

Transport for NSW developed its analytics platform with Contino as a highly scalable, serverless solution built using DevOps practices to maximize business agility. To quickly and easily train a ML model that could predict customer patronage across the network, Transport for NSW ran one year of Opal and weather data through Amazon SageMaker to get predictions for the coming year. ML model training now takes just 35 minutes using the Amazon SageMaker DeepAR forecasting algorithm and Amazon Elastic Compute Cloud (Amazon EC2) C5 Instances, with each transport mode consisting of 168,000 data points. Transport for NSW stores weather data on Amazon DynamoDB, which works in conjunction with Amazon Simple Storage Service (Amazon S3) and Amazon Kinesis Data Firehose to enable the ingestion and transformation of high-velocity data feeds. The agency takes advantage of AWS Lambda to minimize operational requirements while reducing cost.

“The advantage of applications like Amazon SageMaker and Lambda is the ability to quickly spin up a dashboard and build it in a couple of weeks to show our executives,” Mr. Bennetts says. “By using Amazon SageMaker, teams effectively save half the time required to build and train ML models, by not having to code neural networks.” Mr. Bennetts has even built a simple ML model on Amazon SageMaker himself, demonstrating the ease of using the tool.

Thanks to additional dashboards developed with Vizalytics, train controllers benefit from data-driven recommendations on how to mitigate and efficiently recover from service interruptions. “The most important impact of this technology is the confidence it gives our operational teams to make better and more timely decisions that affect customers,” Mr. Bennetts says. For example, operators can react immediately to delays on a train line and deploy a supplementary bus service or add a train when usage spikes in a particular location. Such actions have led to improved customer satisfaction.

Because they can predict patronage figures up to 48 hours in advance, and now have access to real-time contextual insight on impending weather events, management can provide more transport services to customers when they need them. The dashboard solution for NSW TrainLink, for example, has eliminated the previous 24 to 48-hour lag in receiving KPI reports. Management can predict how weather fluctuations will impact utilization across each mode of transport and enable better long and short-term planning. In addition, data is easily shared across the enterprise, which facilitates agility and wider involvement in decision making.

Overall, this ML initiative has made analytics a central part of the agency’s efforts. “This whole process has given us a lot of confidence around how we need to procure and build tools going forward,” Mr. Bennetts says. With ML tools now at their fingertips, he adds, teams may choose to build in-house or together with the startup community through the accelerator, instead of issuing big requests for proposals to consortiums of companies.

“We want to shift our way of working to partnering and incrementally delivering capability, using a test-and-learn mindset,” Mr. Bennetts says. “In transport, this change has been well received. We have really turbocharged the transport sector globally by using real-time data to improve customer satisfaction.”

Reflecting on Transport for NSW’s ML journey, Mr. Bennetts explains how this step-by-step approach has come to define the agency’s path to innovation. “When you’re using these emerging technologies, sometimes it’s hard to envision what’s possible,” he says. “You have to iteratively build by capability, and from there comes more questions and you begin to build a backlog of what comes next. This was very true for the evolution of our dashboards.”

The agency has since built the Vizalytics Interactive Passenger Report to give a real-time replay of passenger flow through the system, similar to a weather radar loop. Executives are using the report as a tool to help improve customer service. Data feeds from this line will be particularly important because this is the first service of its kind in Australia. Mr. Bennetts concludes, “Our ML use cases are fairly straightforward, but even the most basic use cases can be completely transformative to operational and customer teams.”