EagleView Saves 80% Using Amazon EC2 Spot Instances for Image Processing
EagleView, a technology leader in aerial imagery and property data analytics, uses aerial imagery combined with machine learning, computer vision, and data analytics tools to provide insights to customers in construction, real estate, insurance, emergency services, energy, among many other fields.
To collect the imagery that it analyzes, EagleView operates a nationwide network of fixed-wing aircraft. Flying almost daily, these planes take aerial photos across the United States at ultra-high resolution, capturing a daily average of 300,000 images representing about 64 TB of data. The company's image-processing system then uses proprietary EagleView algorithms to identify and extract a huge variety of information, such as building measurements or which structures are most likely to be damaged in a hurricane.
EagleView ran this system and others like it on premises but was hitting the scaling limits of a traditional data center. "Certain jobs were taking days to complete, even for our lower-resolution legacy images," says Josh Leaverton, director of cloud operations at EagleView. “It would have taken weeks for the new higher-resolution images. Cloud computing offered an obvious answer to this challenge, but we had to find the right path to the cloud. We wanted to get the scale we needed without breaking the bank."
That's why—instead of simply lifting and shifting the system onto Amazon Web Services (AWS)—EagleView decided to re-architect its image-processing system to make maximal use of Amazon Elastic Compute Cloud Spot Instances (Amazon EC2 Spot Instances). EagleView now has all the scalability it needs, at a monthly cost that averages 80 percent less than if it were using only Amazon EC2 On-Demand instances.
“Amazon EC2 Spot Instances are helping us retain our current customers and win new business because we can cost-effectively expand our data processing and extraction pipelines."
– Josh Leaverton, Director of Cloud Operations, EagleView
AWS Services Used
EagleView uses aerial imagery combined with machine learning, computer vision, and data analytics tools to provide insights to customers in construction, real estate, insurance, emergency services, energy, and many other fields.
- Saves 80% over On-Demand instances
- Architecting for Spot Instances increases fault tolerance
- Expanded data extraction and processing pipelines
- Reduced cost of test environments
- Eases entering international markets
- Unlimited scalability
AWS Services Used
Architecting for Interruption
Amazon EC2 Spot Instances offer unused AWS compute capacity that costs substantially less than Amazon EC2 On Demand instances, but—because this unused capacity fluctuates—Spot Instances can be pre-empted, or interrupted, with two minutes’ notice. EagleView re-architected its image-processing system with this in mind.
"To be able to use as many Spot Instances as possible, we constructed a distributed system based on stateless functions," says Leaverton. "It's an entirely event-driven system running complex workflows, but it's built so that the code running on one Spot Instance can execute and return its result regardless of what any other instance is doing."
EagleView's proprietary machine learning functions listen for inputs from Amazon Simple Queue Service (Amazon SQS) and then initiate processing jobs. When jobs have been completed, the system calls an AWS Lambda function to orchestrate next steps. If a function's Spot Instance is pre-empted before it can signal job completion, Amazon SQS redelivers the prompt to a function running on another instance. This architecture enables EagleView to run its image-processing system on an Amazon EC2 fleet that usually comprises at least 90 percent Spot Instances.
"We re-architected with Spot Instances in mind to ensure we can run the vast majority of the system on the lowest cost compute resources available," says Leaverton. "Because we were coding for an environment where instances can randomly go down, our system is more robust as we were forced to create an architecture resistant to failures.”
Expanded Pipelines on Amazon EC2 Spot Instances
For EagleView, the most important benefit of using Amazon EC2 Spot Instances is low-cost scalability. "By running mostly on Amazon EC2 Spot Instances, each dollar budgeted for data extraction goes much further," says Leaverton. "We’re getting data to our customers more efficiently while still saving money."
The cost savings help EagleView add more new features to attract and retain customers. "In our industry, adding new features means figuring out how to extract new categories of data from our imagery," says Leaverton. "New features are good for our existing customers and help attract new customers who may not have previously seen the value in aerial imagery. Amazon EC2 Spot Instances are helping us retain our current customers and win new business because we can cost-effectively expand our data processing and extraction pipelines."
One way that using Amazon EC2 Spot Instances improves the process of innovating and creating new features is by reducing the cost of test environments. "With Spot Instances, our developers can stand up complete copies of our systems for sandbox testing without breaking the bank," says Leaverton. "One of our medium-term goals is to shift as much of our testing and staging as possible onto Spot Instances."
Spot Instances for Scalability and Cost Savings
Another goal for EagleView is international expansion. The company has started offering its services in England and Australia, and the flexibility of Spot Instances is helping there, too. "It's vital to internal stakeholders that our system does its processing close to the data origin and the end customer," says Leaverton. "One of the benefits of running everything on AWS is that the entire system can be encapsulated as code and spun up in a new AWS Region."
One possible challenge when entering new international market is that Spot Instances availability can vary by region. That's why—even though the EagleView image-processing system is architected to minimize the need for On-Demand instances—Leaverton is happy there is a feature that makes it easy to incorporate them when necessary. "Because of the variable availability of Spot Instances worldwide, it is a big plus that we can use the same API to spin up both Spot Instances and On-Demand instances."
Leaverton says that the savings available from Spot Instances helped EagleView make the case for its new AWS-first strategy. "The cloud value proposition is usually either scalability or cost savings," says Leaverton. "Running on Amazon EC2 Spot Instances enables us to scale massively when we need to and still spend less than we were spending on our on-premises data center. As we considered the business case for the cloud, that was a helpful point to make internally, and it’s a big part of why our strategy now relies so heavily on AWS."