Transport for NSW Uses AWS and Machine Learning to Make Real-Time Decisions
Public-Private Partnerships Yield Digital Innovation
Travelers to Sydney, Australia, have many choices when it comes to public transportation. From ferries to metro lines to buses and light rail, the state of New South Wales (NSW) maintains one of the world’s largest public transportation systems in terms of geography, in addition to regional roads. Transport for NSW, the government agency responsible for managing this system, issues Opal RFID transport cards that are used by most passengers for public transport services. Passengers can also tap onto the Opal network using their credit card, debit card, or linked device. From 2018 to 2019, Transport for NSW’s train patronage alone comprised 424 million passengers, followed by bus patronage at 349 million.
To improve the customer experience and better connect communities, Transport for NSW continually invests in technology, especially big data processing. Amazon Web Services (AWS) is a partner of the Future Transport Digital Accelerator in Sydney. The Accelerator facilitates direct collaboration between the public and private sectors by connecting teams from the NSW transport cluster with industry partners and startups in the digital space. Several third-party mobile apps have launched from accelerator projects, allowing passengers to check the status of trains, capacity of buses, and more.
“By using Amazon SageMaker, teams effectively save half the time required to build and train ML models.”
Chris Bennetts, Executive Director, Digital Product Delivery, Transport for NSW
About Transport for NSW
Benefits of AWS
AWS Services Used
About Transport for NSW
Transport for NSW is the government agency responsible for the Australian state’s extensive public transport network, including rail lines, buses, ferries, and more. Its role is to lead the development of a safe, efficient, and integrated transport system to make New South Wales a better place to live, work, and visit.
Benefits of AWS
- Improves customer satisfaction with more efficient transport services
- Facilitates development in small, cross-skilled teams with managed services
- Performs ML model training in just 35 minutes with no special skill set required
- Enables operators to make quick decisions using real-time dashboards
- Predicts patronage figures 48 hours in advance with ML-driven analytics
AWS Services Used
Checking the Pulse of the Network
Transport for NSW has been using AWS for its customer-facing assets since 2014, choosing AWS for the breadth and depth of managed services available. “A lot of our improvements are done by small, four to five people teams who are skilled across different technical disciplines. That’s only possible because AWS has invested in opening up these services as part of their infrastructure offering,” says Chris Bennetts, executive director of Digital Products Delivery at Transport for NSW.
In October 2018, Transport for NSW began experimenting with machine learning (ML) to transition from historically based analytics to a forward-looking model with predictive capability. One of its small teams had been using the Tableau data visualization tool and homegrown algorithms to deliver analytics reports, but results were always two or three days behind. “Executives didn’t have the pulse of the network, and the analytics team lacked the latest technology tools to help our operators make better decisions today for tomorrow,” explains Mr. Bennetts.
Multiple data sources, including Opal data, feeds from a third-party vendor and takes many hours to ingest—making it difficult to access, process, and analyze datasets at speed. Through the accelerator, the agency worked with Contino, a Premium Consulting Partner in the AWS Partner Network (APN), and Vizalytics Technology, an APN Select Technology Partner, to develop dashboards that deliver near real-time insights and data-driven recommendations. Transport for NSW is now able to predict patronage numbers across the entire transport network, enabling the agency to better plan workforce and asset utilization, and improve customer satisfaction.
Training ML Models Using AWS Services
Transport for NSW developed its analytics platform with Contino as a highly scalable, serverless solution built using DevOps practices to maximize business agility. To quickly and easily train a ML model that could predict customer patronage across the network, Transport for NSW ran one year of Opal and weather data through Amazon SageMaker to get predictions for the coming year. ML model training now takes just 35 minutes using the Amazon SageMaker DeepAR forecasting algorithm and Amazon Elastic Compute Cloud (Amazon EC2) C5 Instances, with each transport mode consisting of 168,000 data points. Transport for NSW stores weather data on Amazon DynamoDB, which works in conjunction with Amazon Simple Storage Service (Amazon S3) and Amazon Kinesis Data Firehose to enable the ingestion and transformation of high-velocity data feeds. The agency takes advantage of AWS Lambda to minimize operational requirements while reducing cost.
“The advantage of applications like Amazon SageMaker and Lambda is the ability to quickly spin up a dashboard and build it in a couple of weeks to show our executives,” Mr. Bennetts says. “By using Amazon SageMaker, teams effectively save half the time required to build and train ML models, by not having to code neural networks.” Mr. Bennetts has even built a simple ML model on Amazon SageMaker himself, demonstrating the ease of using the tool.
Supporting Timely Decisions with Near Real-Time Dashboards
Thanks to additional dashboards developed with Vizalytics, train controllers benefit from data-driven recommendations on how to mitigate and efficiently recover from service interruptions. “The most important impact of this technology is the confidence it gives our operational teams to make better and more timely decisions that affect customers,” Mr. Bennetts says. For example, operators can react immediately to delays on a train line and deploy a supplementary bus service or add a train when usage spikes in a particular location. Such actions have led to improved customer satisfaction.
Because they can predict patronage figures up to 48 hours in advance, and now have access to real-time contextual insight on impending weather events, management can provide more transport services to customers when they need them. The dashboard solution for NSW TrainLink, for example, has eliminated the previous 24 to 48-hour lag in receiving KPI reports. Management can predict how weather fluctuations will impact utilization across each mode of transport and enable better long and short-term planning. In addition, data is easily shared across the enterprise, which facilitates agility and wider involvement in decision making.
Putting Analytics at the Core
Overall, this ML initiative has made analytics a central part of the agency’s efforts. “This whole process has given us a lot of confidence around how we need to procure and build tools going forward,” Mr. Bennetts says. With ML tools now at their fingertips, he adds, teams may choose to build in-house or together with the startup community through the accelerator, instead of issuing big requests for proposals to consortiums of companies.
“We want to shift our way of working to partnering and incrementally delivering capability, using a test-and-learn mindset,” Mr. Bennetts says. “In transport, this change has been well received. We have really turbocharged the transport sector globally by using real-time data to improve customer satisfaction.”
Taking an Incremental Approach to ML
Reflecting on Transport for NSW’s ML journey, Mr. Bennetts explains how this step-by-step approach has come to define the agency’s path to innovation. “When you’re using these emerging technologies, sometimes it’s hard to envision what’s possible,” he says. “You have to iteratively build by capability, and from there comes more questions and you begin to build a backlog of what comes next. This was very true for the evolution of our dashboards.”
The agency has since built the Vizalytics Interactive Passenger Report to give a real-time replay of passenger flow through the system, similar to a weather radar loop. Executives are using the report as a tool to help improve customer service. Data feeds from this line will be particularly important because this is the first service of its kind in Australia. Mr. Bennetts concludes, “Our ML use cases are fairly straightforward, but even the most basic use cases can be completely transformative to operational and customer teams.”