DataFarming Processes Data from 35 Million Acres Using Amazon ECS
Australian precision agriculture company DataFarming wanted to improve its ability to deliver high-quality data from satellite imagery to farmers quickly and effectively, helping them optimize their crop growth. The company wanted farmers to be able to view the data on mobile devices out in the field, where very limited bandwidth is common. To achieve these goals without increasing costs for the business or its customers, DataFarming turned to solutions from Amazon Web Services (AWS). Using Amazon Elastic Container Service (Amazon ECS)—a fully managed container orchestration service that makes it simple to deploy, manage, and scale containerized applications—and other services from AWS, DataFarming has achieved a 900 percent increase in satellite imagery use in the Australian grains market in just over 4 years.
Amazon EC2 Spot Instances has delivered a lot of outcomes for us in cost optimization. We can scale up as an influx of orders comes through and have instances to compute on.”
Senior Software Engineer, DataFarming
Building with Scalability in Mind
Established in 2017 and based in Queensland, Australia, DataFarming provides digital solutions that empower farmers to react to changes in crops and soil for more productive farming. Having decided at the beginning that cloud services would be superior to legacy software and databases for its digital solutions, DataFarming has always used AWS services. “AWS empowers us to rapidly implement solutions and react to changing requirements from people in the field and respond to them quickly,” says Paul Grambauer, senior software engineer at DataFarming.
The company’s digital solutions use satellite imagery to help farmers address challenges, such as pest and disease incidence and increasing fertilizer costs. Furthermore, this imagery helps farmers gain insight into factors, such as crop moisture content and maturity, to determine the ideal harvest timing. All of these variables affect farmers’ profitability and productivity. The DataFarming founders brought the first high-resolution remote sensing satellites into the Australian agriculture industry back in 2003. In doing so, the company has played a key role in the agriculture industry’s growing access to lower cost, better quality, and more frequent satellite imagery.
Originally, DataFarming processed high-resolution satellite imagery data on a scheduled basis, with limitations on the number of hours it could process in 1 day. Wanting to improve agility and deliver data to farmers faster, the company decided in 2021 to change to event-driven processing. At the same time, the company wanted to be able to deliver its data in the smallest package possible for farmers’ ease of access, even in areas with limited bandwidth. With its long history of using AWS, DataFarming once again looked to AWS solutions to accomplish its goals.
Using Amazon ECS to Meet Customer Needs
DataFarming began developing proofs of concept in a containerized setting in AWS Lambda, a serverless, event-driven compute service that lets users run code for virtually any type of application or backend service without provisioning or managing servers. But it found that Amazon ECS better suited its specific needs. Because the company was already using containers on AWS, the switch to Amazon ECS was smooth. “The environment for running containers in AWS makes it super simple to pick a solution,” says Grambauer. “Amazon ECS works best for us, but AWS Lambda does a similar thing, and the ability to pick the right service for our needs is my favorite part of developing a solution using AWS.”
The company now uses Amazon ECS as its primary compute environment, and it uses the service to provide functions for its backend APIs to support the user interface for its digital solutions. To run processes for this containerized environment, DataFarming uses Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances, which companies can use for various stateless, fault-tolerant, or flexible applications, such as big data, containerized workloads, and test and development workloads. “Amazon EC2 Spot Instances has delivered a lot of outcomes for us in cost optimization. We can scale up as an influx of orders comes through and have instances to compute on,” says Grambauer. “We’re seeing a 70 percent savings using Amazon EC2 Spot Instances compared to On-Demand Amazon EC2 prices.”
While using Spot Instances to scale cost effectively, DataFarming also optimizes its scalability in data storage. The company stores most of its image-based data in Amazon Simple Storage Service (Amazon S3), an object storage service that offers industry-leading scalability, data availability, security, and performance. “If we capture data over a farm, we have the capability to store that data for the user forever, so they can look back several years and compare performance,” says Tim Neale, managing director at DataFarming. “It’s important that they have access to that data, so it must be stored effectively, and that is what Amazon S3 does for us.” DataFarming uses Amazon S3 alongside Amazon ElastiCache, a fully managed, in-memory caching service that supports flexible, real-time use cases. It uses Amazon ElastiCache as a temporary data storage to improve delivery and optimize data in a mobile-friendly and low-bandwidth way for fast and efficient delivery to farmers.
Using AWS services, DataFarming built a solution that includes 28,000 farms and processes data for 35 million acres of farmland. “We’ve reached 40 percent of the Australian grains market in 4 years, and before we came along, only 4 percent of farmers used satellite imagery,” says Neale. “This is the biggest uptake of technology that I’ve seen in my lifetime.” The company’s progress is spurred by scalable solutions that provide a positive user experience, which DataFarming achieves using AWS services.
Embracing the Future of Agriculture
Going forward, DataFarming has plans to continue expanding its user base internationally. Seeking to increase farmers’ use of satellite imagery across the globe, the company is currently targeting countries with large numbers of farmers in Western Europe, Africa, North and South America, and Southeast Asia.
Additionally, because farmers are facing increasing climate variability, DataFarming is working on machine learning solutions to predict potential climate-related problems and improve the early detection of crop stresses, pests, and disease. “Early detection is going to need machine learning,” says Neale. “Working with spatial information adds challenges that normal machine learning won’t be able to tackle. We are excited about the future and keen to explore more machine learning tools that AWS has to offer.”
DataFarming is a precision agriculture company based in Toowoomba, Queensland, Australia. It delivers digital solutions and satellite imagery data to farmers around the world to improve precision agriculture products and farm data.
Benefits of AWS
- Reached 40% of the Australian grains market in 4 years
- Achieved 70% savings using Amazon EC2 Spot Instances compared to On-Demand pricing
- Processes data from 35 million acres of farmland
- Increased scalability for the international market
- Provides mobile-friendly solutions to meet customer needs
- Achieved a 900% increase in farmer adoption of satellite imagery
AWS Services Used
Amazon Elastic Container Service (Amazon ECS)
Amazon ECS is a fully managed container orchestration service that helps you easily deploy, manage, and scale containerized applications. It deeply integrates with the rest of the AWS platform to provide a secure and easy-to-use solution for running container workloads in the cloud and now on your infrastructure with Amazon ECS Anywhere.
AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers. You can trigger Lambda from over 200 AWS services and software as a service (SaaS) applications, and only pay for what you use.
Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances
Amazon EC2 Spot Instances let you take advantage of unused EC2 capacity in the AWS cloud at up to a 90% discount compared to On-Demand prices. You can use Spot Instances for various stateless, fault-tolerant, or flexible applications such as big data, containerized workloads, CI/CD, web servers, high-performance computing (HPC), and test & development workloads.
Amazon Simple Storage Service (Amazon S3)
Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading scalability, data availability, security, and performance. Customers of all sizes and industries can store and protect any amount of data for virtually any use case, such as data lakes, cloud-native applications, and mobile apps.
Organizations of all sizes across all industries are transforming their businesses and delivering on their missions every day using AWS. Contact our experts and start your own AWS journey today.