DataFarming Processes Data from 35 Million Acres Using Amazon ECS

Australian precision agriculture company DataFarming wanted to improve its ability to deliver high-quality data from satellite imagery to farmers quickly and effectively, helping them optimize their crop growth. The company wanted farmers to be able to view the data on mobile devices out in the field, where very limited bandwidth is common. To achieve these goals without increasing costs for the business or its customers, DataFarming turned to solutions from Amazon Web Services (AWS). Using Amazon Elastic Container Service (Amazon ECS)—a fully managed container orchestration service that makes it simple to deploy, manage, and scale containerized applications—and other services from AWS, DataFarming has achieved a 900 percent increase in satellite imagery use in the Australian grains market in just over 4 years.

Established in 2017 and based in Queensland, Australia, DataFarming provides digital solutions that empower farmers to react to changes in crops and soil for more productive farming. Having decided at the beginning that cloud services would be superior to legacy software and databases for its digital solutions, DataFarming has always used AWS services. “AWS empowers us to rapidly implement solutions and react to changing requirements from people in the field and respond to them quickly,” says Paul Grambauer, senior software engineer at DataFarming.

The company’s digital solutions use satellite imagery to help farmers address challenges, such as pest and disease incidence and increasing fertilizer costs. Furthermore, this imagery helps farmers gain insight into factors, such as crop moisture content and maturity, to determine the ideal harvest timing. All of these variables affect farmers’ profitability and productivity. The DataFarming founders brought the first high-resolution remote sensing satellites into the Australian agriculture industry back in 2003. In doing so, the company has played a key role in the agriculture industry’s growing access to lower cost, better quality, and more frequent satellite imagery.

Originally, DataFarming processed high-resolution satellite imagery data on a scheduled basis, with limitations on the number of hours it could process in 1 day. Wanting to improve agility and deliver data to farmers faster, the company decided in 2021 to change to event-driven processing. At the same time, the company wanted to be able to deliver its data in the smallest package possible for farmers’ ease of access, even in areas with limited bandwidth. With its long history of using AWS, DataFarming once again looked to AWS solutions to accomplish its goals.

DataFarming began developing proofs of concept in a containerized setting in AWS Lambda, a serverless, event-driven compute service that lets users run code for virtually any type of application or backend service without provisioning or managing servers. But it found that Amazon ECS better suited its specific needs. Because the company was already using containers on AWS, the switch to Amazon ECS was smooth. “The environment for running containers in AWS makes it super simple to pick a solution,” says Grambauer. “Amazon ECS works best for us, but AWS Lambda does a similar thing, and the ability to pick the right service for our needs is my favorite part of developing a solution using AWS.”

The company now uses Amazon ECS as its primary compute environment, and it uses the service to provide functions for its backend APIs to support the user interface for its digital solutions. To run processes for this containerized environment, DataFarming uses Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances, which companies can use for various stateless, fault-tolerant, or flexible applications, such as big data, containerized workloads, and test and development workloads. “Amazon EC2 Spot Instances has delivered a lot of outcomes for us in cost optimization. We can scale up as an influx of orders comes through and have instances to compute on,” says Grambauer. “We’re seeing a 70 percent savings using Amazon EC2 Spot Instances compared to On-Demand Amazon EC2 prices.”

While using Spot Instances to scale cost effectively, DataFarming also optimizes its scalability in data storage. The company stores most of its image-based data in Amazon Simple Storage Service (Amazon S3), an object storage service that offers industry-leading scalability, data availability, security, and performance. “If we capture data over a farm, we have the capability to store that data for the user forever, so they can look back several years and compare performance,” says Tim Neale, managing director at DataFarming. “It’s important that they have access to that data, so it must be stored effectively, and that is what Amazon S3 does for us.” DataFarming uses Amazon S3 alongside Amazon ElastiCache, a fully managed, in-memory caching service that supports flexible, real-time use cases. It uses Amazon ElastiCache as a temporary data storage to improve delivery and optimize data in a mobile-friendly and low-bandwidth way for fast and efficient delivery to farmers.

Using AWS services, DataFarming built a solution that includes 28,000 farms and processes data for 35 million acres of farmland. “We’ve reached 40 percent of the Australian grains market in 4 years, and before we came along, only 4 percent of farmers used satellite imagery,” says Neale. “This is the biggest uptake of technology that I’ve seen in my lifetime.” The company’s progress is spurred by scalable solutions that provide a positive user experience, which DataFarming achieves using AWS services.

Going forward, DataFarming has plans to continue expanding its user base internationally. Seeking to increase farmers’ use of satellite imagery across the globe, the company is currently targeting countries with large numbers of farmers in Western Europe, Africa, North and South America, and Southeast Asia.

Additionally, because farmers are facing increasing climate variability, DataFarming is working on machine learning solutions to predict potential climate-related problems and improve the early detection of crop stresses, pests, and disease. “Early detection is going to need machine learning,” says Neale. “Working with spatial information adds challenges that normal machine learning won’t be able to tackle. We are excited about the future and keen to explore more machine learning tools that AWS has to offer.”