Nearmap Builds AWS-Powered ML Pipeline to Help Others Analyze the World

As cities build new roads, bridges, and skyscrapers, it’s easy to forget what came before. Memory and even photographs can be incomplete or unreliable: meticulously analyzing a city’s past, what exactly changed, and how the change has made an impact requires sophisticated tools. Australian aerial imaging company Nearmap has developed one such tool.

Founded in 2007, Nearmap has made a name for itself by compiling and producing high-resolution 2D and 3D images of populated areas around the world. After years of providing clear and up-to-date imaging products, the company decided to go a step further and incorporate sophisticated image classification powered by artificial intelligence (AI) and machine learning (ML). Presented with a significant challenge in terms of cost and compute capacity, Nearmap found a fast and affordable solution with Amazon Web Services (AWS)—specifically, Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances.

To collect its data, Nearmap installs its proprietary cameras in a fleet of planes that capture aerial images of about 1,000 square km per average flight, resulting in 2D and 3D visualization that offers far better clarity than high-resolution satellite images.

In June 2019, Nearmap introduced an AI beta product that would use image classification to get even more information out of its 2D top-down images. On June 1, 2020, it launched Nearmap AI as a generally available product for organizations, adding a heads-up display-style view of layers such as solar panels and building footprints to its web-based MapBrowser product as well as the ability to export the data to use for other workflows. This product ingests images from the Amazon Simple Storage Service (Amazon S3)—an object storage service that customers can use to store and protect any amount of data for a range of use cases—and passes them through an ML pipeline that first assigns individual pixels of meaning to each image to identify layers: one layer for swimming pools, one for solar panels, and so on. The AI then postprocesses those pixels to generate information. It identifies shapes and objects, performs geospatial calculations on them, and compares those calculations to property datasets—all of which builds super-rich datasets loaded with details about the sites in the imagery.

“We’re effectively mapping the evolution of cities at incredible scale and detail,” says Michael Bewley, senior director of AI systems for Nearmap. “With that granular view of the world, you suddenly have more control in understanding how you’re planning out cities and how you want your cities to develop, because it can be backed by much more quantitative information.”

From a cost standpoint, however, Nearmap had compute-heavy AI and ML workloads that would be virtually impossible to run using an on-premises solution—not to mention that an on-premises solution simply couldn’t handle the need to rapidly scale from a handful to thousands of GPUs in a matter of hours to reach peak load. Even a typical cloud solution seemed cost prohibitive: to provide enough compute capacity for its sophisticated classification feature, Nearmap would have to reserve thousands of Amazon EC2 GPUs to handle its max computing needs. “If we’ve got reserved instances, we’d have to commit to a particular generation of GPU over a long period of time,” says Bewley. “That’s the nice thing about Amazon EC2 Spot Instances.”

Amazon EC2 Spot Instances enable customers to access spare Amazon EC2 capacity at up to a 90 percent discount, compared to Amazon On-Demand Instances pricing. The Nearmap business model relies on imaging regions on specification and having customers—from small traders to large enterprises and the government—pay a subscription for access to a vast library of 2D and 3D images and, most recently, AI datasets. The scalability of Amazon EC2 Spot Instances makes them a perfect fit for Nearmap, enabling it to run algorithms on 50 nodes one day and 1,000 the next. The company could effectively run substantial workloads without worrying about losing money from unused reserved compute capacity. “As a company, we’ve been using Amazon EC2 Spot Instances since they started in Australia,” says Bewley. “It’s simply the way you do things at Nearmap.”

Nearmap started out by using Amazon EC2 Spot Instances to access Amazon EC2 P2 Instances, which are powerful, scalable instances that provide GPU-based parallel compute capabilities. But when it saw an opportunity to upgrade from Amazon EC2 P2 Instances to AWS’s next-generation Amazon EC2 G4 Instances—instances specifically designed to deliver the most cost-effective and versatile GPU instance for deploying ML models in production and graphics-intensive applications—the company went for it.

Because of the flexible nature of Amazon EC2 Spot Instances, making the switch to Amazon EC2 G4 Instances was relatively easy. With some reconfiguring of its Kubernetes clusters, Nearmap was able to make the switch to Amazon EC2 G4 Instances in a few weeks, and the increased memory and compute capacity immediately allowed Nearmap to run larger, more complex models with additional optimizations that were not possible on the older-generation hardware. Amazon EC2 G4 Instances also significantly reduced the company’s costs. “The GPU-based component, when you tie it all up, is about three times cheaper to run on an Amazon EC2 G4 Instance compared to an Amazon EC2 P2 Instance based on current Amazon EC2 Spot Instances pricing,” says Bewley.

With Amazon EC2 Spot Instances and the high availability of inexpensive compute capacity, Nearmap has been able to reroute some of its resources to developing a cutting-edge image classification product. Insurance companies use the tool to assess property risk and note and quantify damage in the wake of a disaster. Telecommunications and utility companies use the tool to pinpoint power poles and solar panels, helping them determine where to install 5G antennae and how to manage their power grids. Local governments use Nearmap AI to do property appraisals and understand how their built environment and green spaces are changing over time.

Ultimately, innovation at AWS dovetails with innovation at Nearmap, and Nearmap uses that to build viable products with profound ramifications. “I think there’s going to be a fascinating future of helping people keep track of how the built environment is changing, tracking how cities evolve at scale with nuance and hopefully making them more livable places as a result,” says Bewley. “That’s what I’d love to see—by empowering everyone with knowledge about how our cities are actually changing, they can have much more informed discussions about not just how they are evolving, but how they should evolve.”