Bat Conservation Trust gets Better Bat Counts Using Lambert Labs to Build Scalable Solution on AWS

Monitoring and research programs are central to the BCT’s work. Volunteers across the UK help with bat counts and set up monitors to supplement the work done by its core staff. The BCT’s bespoke processing tools—called the BCT Sound Classification System (SCS)—were developed in-house and used machine learning (ML) algorithms that were developed jointly by the University of Edinburgh and University College London (UCL) to analyze bat recordings. As a proof of concept, it worked well, but the system couldn’t scale to handle large amounts of input data—an estimated 100 TBs or more—and was constraining BCT’s research.

Bats eat insects almost exclusively and that prevents billions of dollars per year globally in damage to agricultural crops. The health of bat populations can indicate issues with the larger environment. Getting a clear understanding of where bats live and hunt is core to BCT’s work. “We generate a huge amount of data,” says Dr Lia Gilmour, research manager at Bat Conservation Trust. “Passive acoustic monitoring of a site—putting out a sensor and leaving it at the site—can easily generate a large amount of data in a night. Our tools were very good at processing and classifying the data, but not being able to process the data quickly enough was holding us back.” The SCS had shown what was possible. Using technology to analyze survey recordings reduced the need for people to monitor and classify the data. The BCT needed a way to scale its system so it wouldn’t be limited in the number of sites it could survey. As a nonprofit organization, it also wanted to find a way to cut costs so it could use the funding it had to do more.

The BCT looked for ways to expand its capabilities using technology. It was introduced to AWS Partner Lambert Labs by its Amazon Web Services (AWS) account manager and the two organizations worked together to give BCT greater reach and power. The first step was to rebuild BCT’s data processing pipeline so that it could scale to whatever level was needed. “The team at BCT are really smart,” says George Lambert, chief executive officer and founder of Lambert Labs. “They knew what they wanted their tools to do, which was to scale up to let them do more. We applied our data processing expertise, our knowledge of AWS services, and our human skills to take on that challenge and rebuild their AWS infrastructure, launching the first iteration in just 4 weeks.”

This wasn’t just a simple lift and shift, though. The original version of the SCS was running on Amazon Elastic Compute Cloud (Amazon EC2), but the resources weren’t optimized. Lambert Labs, with its deep understanding of AWS, rearchitected BCT to run using AWS Batch, which lets developers, scientists, and engineers efficiently run hundreds of thousands of batch computing jobs while optimizing compute resources. This provides the ability to focus on analyzing results and solving problems, as well as faster ingestion of the data that BCT was generating. AWS Fargate, a serverless, pay-as-you-go compute engine that lets users focus on building applications without managing servers, gave BCT access to the compute resources it needed—as well as the ability to spin down those resources when not needed. Lambert Labs also containerized the SCS to add scalability and flexibility. AWS Lambda—a service that runs a user’s code in response to events and automatically manages the compute resources—works to keep things moving through the pipeline. The system also uses Amazon Relational Database Service (Amazon RDS) and Amazon Simple Storage Service (Amazon S3) for storage.

Using multiple AWS accounts for policy-based management on AWS Organizations helps more easily manage policies for groups of accounts and automate account creation. Lambert Labs also worked with BCT to develop a staging environment, enabling BCT to safely work on improving its capabilities. “Lambert Labs are great to work with,” says Gilmour. “They took the time to understand what we needed to do and then worked with us to give us something that makes our job easier. We don’t have to spend time worrying about managing AWS or waiting for data to process. We can focus on our mission: bats.” Lambert Labs also helped BCT apply for an AWS IMAGINE Grant, which it received. These are public grant opportunities open to registered nonprofit charities that are using cloud technology to accelerate their mission. “We couldn’t guarantee that BCT would receive the grant,” says Lambert. “However, we wanted to make them aware of it and help with the process.” That grant, says Gilmour, is a welcome bonus to working with Lambert Labs. “It really was a collaborative effort,” she says. “We appreciate the help. It’s very exciting. It’ll help us get more done.”