How Swimming Australia uses machine learning for real-time training pool analytics with AWS Panorama

Swimming Australia is the national governing body for swimming in Australia. The not-for-profit organization is responsible for the leadership, promotion and development of swimming from the grassroots to high-performance elite level swimmers across Australia. Swimming Australia’s vision is to create a nation of swimmers admired by the world, with its sights firmly set on achieving this in the water at the upcoming Olympic and Paralympic Games. Out of the water, its Performance Insights team is supporting this vision with a new way of thinking about athlete training data and how to harness it to drive a global competitive advantage.

Swimming Australia’s Performance Insights team stands to enhance its training environment by implementing a new, state-of-the-art performance analysis system. This technology uses pool footage from training sessions to extract meaningful data insights, empowering coaches to provide targeted feedback to athletes. Coaching feedback is most effective during or immediately after a training drill, which means the system needs to operate in real-time.

In this blog post, we explore how Swimming Australia uses AWS Panorama from Amazon Web Services (AWS) to perform real-time computer vision (CV) inference at the edge with state-of-the-art computer vision models. We describe how to integrate an end user application with AWS Panorama along with the rationale for design strategies and their impact. This solution features services such as AWS Panorama for machine learning (ML) at the edge, Amazon Simple Queue Service (Amazon SQS) for messaging, Amazon Kinesis Video Stream for video streaming, and AWS App Runner for the end user application.

Customer challenge

In the training pool, athlete performance measurements are primarily collected manually using stopwatches. If long-term data storage is required, support staff manually record data with pen and paper and enter it into a database post session. This process doesn’t scale, limits the volume of data insights, and leads to inaccuracies and delays in coaches accessing athlete metrics by several hours. A new, digitized performance system equips coaches with automatic and accurate real-time training insights.

Swimming Australia’s existing end-user application uses on-premises infrastructure provisioning with manual deployments, which results in long lead time to market and increased effort. This challenge presents an opportunity to migrate and modernize the application to the cloud to increase resiliency, availability and accelerate feature developments in the new Training Insights application. This let’s end users like coaches and performance support personnel see improvements and updates in the system faster, rather than waiting for days to update.

Swimming Australia’s Olympic and Paralympic athletes train in multiple pool locations across the country, all of which are open to the public. Consequently, the cameras installed to capture training sessions must be strategically positioned at elevated angles to ensure compliance with Australian personal identifiable information (PII) video protections while still providing birds-eye coverage over the entire pool, which at some locations requires using multiple cameras. Additionally, the wireless internet connectivity within these training pool environments is unreliable and has limited bandwidth, which makes streaming a high-resolution, RTSP camera feed to the cloud a challenge. This necessitates a hybrid solution to combine the benefits of the cloud with an edge device operating with limited internet bandwidth.

Customer solution

Swimming Australia’s new training analysis system combines the flexibility and scalability of the cloud with the low latency and bandwidth efficiency of the edge. Coaches and performance support personnel interact with computer vision models at the edge via an end-user application hosted in the cloud. Before a training session, athlete metadata and training drill details are input and the live analysis initiated.

The multi-step machine learning pipeline, hosted at the edge, starts by sending a rolling batch of video frames to a deep learning model that detects swimmers’ heads. Detected swimmers are tracked throughout their training session, generating time, distance, and velocity metrics. Additional machine learning models generate novel streams of insights from these metrics, such as the efficiency of a swimming turn. These final insights are uploaded as lightweight payloads and made immediately available to the user in the application for real-time and actionable athlete feedback.

Following the completion of a training session, data insights, along with attached metadata, can be reviewed against training video footage and edited as needed before upload to Swimming Australia’s existing data lake for long-term storage. The video footage of the athletes’ training session is stored in the data lake for future model development and retraining. Analyzing training insights over months or years unlocks unique opportunities for Swimming Australia, not only in tracking individual athlete development but in retrospective analysis of the training that led to successful competition races.

Architecture overview

To unleash the full potential of Swimming Australia’s Training Insights application, the solution architecture combines multiple services and tools from AWS. Deployment is automated with AWS Cloud Development Kit (AWS CDK) and Continuous Integration/Continuous Deployment (CI/CD) pipelines.

AWS App Runner hosts the end-user application and integrates with Amazon DynamoDB and Amazon S3 data lake to persist the training insights data and video footage. An intelligent video frame retrieval process, using Amazon Kinesis Video Stream, enables the quick review of generated insights before they are persisted.

The frames are streamed using the Amazon KVS GStreamer plugin with a network-constrained configuration to conserve bandwidth and prevent dropped video frames. For technical details, please refer to this Panorama Sample App for streaming to Amazon KVS.

Figure 1: Solution overview for Training Insights, a new digitized performance system for real-time training pool analytics.

AWS Panorama runs real-time inference and automates the management of edge applications and models. AWS Panorama is an ML appliance and software development kit (SDK) that allows users to bring computer vision to their on-premises cameras and make automated predictions with high accuracy and low latency. Key differentiators of AWS Panorama as a computer vision edge solution include optimized hardware for efficient hosting of machine learning models, support for video stream ingestion from multiple cameras, and ease of maintenance thanks to full device management through over-the-air (OTA) software updates and automated OS patching.

Figure 2: Machine learning inference pipeline for analyzing a swimmer’s turn in real-time.

The machine learning inference pipeline and the Amazon KVS Producer client are deployed onto the AWS Panorama Appliance as two separate AWS Panorama Applications. To control the AWS Panorama applications from AWS App Runner, Amazon SQS is used as the messaging bus of choice. Amazon SQS ensures reliability, low latency, and scalability when multiple AWS Panorama applications are deployed on the same AWS Panorama appliance.

When initialized, the AWS Panorama applications automatically subscribe to the Amazon SQS topic that is associated with the camera and will listen for a ‘start’ or ‘stop’ message sent from the AWS App Runner application. Similarly, the AWS Panorama applications send messages back to the AWS App Runner application, such as training insight payloads or application health checks, via dedicated Amazon SQS topics. Data encryption protects the data, including video footage, while it is in transit and at rest through encryption support with Amazon SQS, Amazon S3, and Amazon Kinesis Video Streams. The AWS Glue workflow implements row-level security in existing Amazon Quicksight dashboards, enabling coaches to access only authorized athlete data.

Business impact

This new ML- and computer vision-powered system assists athlete performance in training with faster coach-to-athlete feedback (from hours to seconds), more accurate and novel data insights, and allows collection of personalized, long-term data trends. Manual efforts on the pool deck are streamlined with digitized systems providing 90% faster access to training insights and 91% faster footage review.

Conclusion

Swimming Australia has set its vision on fundamentally transforming the way it captures and uses data in order to train its athletes to be the world’s best. In this blog post, we highlighted the key architecture components of a brand-new performance analysis system for the training environment, replacing manual methods with computer vision models deployed at the edge.

The solution operates in limited bandwidth using AWS Panorama to host the ML- and computer vision-powered inference pipeline that generates training session insights. These insights are sent, via Amazon SQS, to an application hosted on AWS App Runner and persisted in an Amazon S3 data lake. The solution applies to use cases that need to run computer vision models in near-real time at the edge with limited or unstable internet bandwidth.

This lays the foundation for future innovations in athlete training and positions Swimming Australia well on its path to realize the goal of setting a global industry standard in high performance athletic training.