How Nearmap built continental-scale aerial search using Amazon S3 Vectors

Nearmap captures high-resolution aerial imagery across populated areas of the United States, Canada, Australia, and New Zealand several times a year, at resolutions as fine as 1.5 inches. More than 10,000 customers globally use these images to assess insurance portfolios, size solar arrays, and track construction sites. Since 2007, Nearmap has completed more than 35,000 aerial surveys, creating an archive of more than 100 petabytes that continues to grow.

Nearmap AI uses purpose-trained models to catalog more than 130 features, such as roof condition, solar panels, and overhanging vegetation. The challenge was the long tail of questions those models did not anticipate. A customer query like “show me every property in this county with a caravan in the backyard” required collecting examples, labeling them, and training and deploying a new detector, just to answer one question.

Zero-shot Search is Nearmap’s semantic search technology for its continental-scale aerial imagery archive. It allows users to find objects and patterns using a text description, an uploaded image, or a map location without training a detector for each new question. It converts aerial images and search terms into vector embeddings, compact numerical representations that place similar inputs near one another in the same vector space. When a phrase and an image represent the same concept, their embeddings are closer in this space, allowing a query to match directly against the archive.

Embeddings place visually similar aerial scenes near one another, forming groups such as swimming pools, solar panels, rooftops, fields, animals, and vehicles.

Nearmap built Zero-shot Search using Amazon S3 Vectors—a cost-efficient cloud object store with support to store and query vectors natively in Amazon Simple Storage Service (Amazon S3)—to index more than 1 billion embeddings generated from 107 aerial surveys. In this post, we explain how the implementation organizes compatible embeddings across S3 Vectors indexes, stores the metadata needed to render results, and scales ingestion and retrieval through batching and parallelism. You will learn how to apply these patterns to build semantic similarity search across your own large-scale datasets.

The challenge: A continental-scale archive with no semantic index

Nearmap serves the archive as map tiles addressed by (zoom, x, y) coordinates. Tiles are a natural unit for embedding because they already exist in Amazon S3 and each tile represents a coherent scene. Zero-shot Search uses several embedding models and separate generation and search paths for map tiles, top-down property imagery, and oblique property imagery. This allows each representation to capture visual patterns at the relevant scale and viewing angle.

Each aerial survey can yield tens of millions of vectors, and Nearmap retains historical embeddings so that users can compare locations over time. This required permanent storage for a continuously growing corpus without running an always-on search cluster sized to hold every vector in memory. Nearmap also needed to absorb bursty interactive queries from customers and AI agents without provisioning always-on capacity for peak demand that goes mostly unused.

Evaluating Amazon S3 Vectors

Nearmap evaluated storage options based on how Zero-shot Search retrieves imagery. Users can find matching tiles with a text description, an uploaded image, or a point selected on the map. The service turns that input into a query embedding and compares it with stored tile embeddings.

For similarity heatmaps, Zero-shot Search uses a selected map area to define the tiles in scope, scores them against the text description, and shades the map by similarity. Each stored vector also carries metadata fields, such as its survey identifier, that allow the service to limit searches to relevant imagery.

These retrieval workflows, combined with Nearmap’s growing corpus, required permanent retention at billion-vector scale, similarity search across the corpus, exact retrieval by key for selected map areas, and metadata filtering by survey. Nearmap selected Amazon S3 Vectors because it provides purpose-built, cost-optimized vector storage in Amazon S3, along with the similarity search, direct key lookup, and metadata filtering that these workflows require.

Zero-shot Search scores approximately 79,000 tiles in Hervey Bay, Queensland, Australia, against the query “caravans.” Warm colors indicate stronger matches, and cool colors indicate weaker matches.

Zero-shot Search can return either individual map tiles or properties. A map tile result is a fixed-size scene identified by (zoom, x, y) coordinates. A property result combines one top-down image with up to eight oblique views of a parcel, the mapped area of land associated with the property. Nearmap stores the top-down and oblique embeddings in separate sets of S3 Vectors indexes and links them with a shared property identifier, allowing the query layer to assemble the matching views into a single result. Oblique imagery exposes facade details that are not visible from above, so users can search for those features without a task-specific detector.

Zero-shot Search returns the top oblique property match for each text query, with the matched parcel outlined in blue.

Cost and operational simplicity also supported the choice. S3 Vectors pricing is based on vector storage, uploads, and queries rather than provisioned query capacity, which suits a permanent corpus with unpredictable and bursty demand. As the archive grows, Nearmap does not need to provision, resize, or rebalance a search cluster.

Nearmap also benchmarked the S3 Vectors API requests used by Zero-shot Search. Depending on the query path and the number of S3 Vectors indexes involved, Zero-shot Search sends from 10 to 180 requests in parallel. Across all benchmark scenarios, mean latency per API request remained consistently in the hundreds of milliseconds.

Architecture

Zero-shot Search has an interactive serving path for text, image, and map-location requests, and a separate batch path for ingesting embeddings. The batch path runs on Amazon Elastic Kubernetes Service (Amazon EKS) using Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances. Its stages have different resource profiles: GPU nodes generate embeddings and write them as Parquet files to Amazon S3, then CPU nodes load the files into S3 Vectors. This design allows Nearmap to scale GPU and CPU resources independently. Each stage is idempotent and resumes from checkpoints, so Spot Instance interruptions do not require a full rerun.

The storage design distinguishes an embedding’s definition from how its vectors are grouped, divided for scale, and identified. An embedding specification is an immutable definition of the model, vector dimension, input preparation, index-name prefix, and metadata schema. An index family is the complete set of S3 Vectors indexes that share one embedding specification.

Each shard in that family is one S3 Vectors index. Sharding divides the vectors across indexes for parallel ingestion and retrieval. A deterministic key is a vector identifier computed from the source data, so a caller can reproduce it without consulting another lookup system.

A single ingest run targets one embedding specification and one survey, as shown in the following figure. The run resolves the corresponding index family from the specification in the source manifest, computes every vector key and shard assignment, and writes pre-sharded Parquet files. Each worker reads only the files assigned to its shard and writes to the corresponding S3 Vectors index.

Ingestion: logical architecture with CPU-bound, GPU-bound, and throughput-bound fan-outs.

Each ingestion worker sends batches of up to 500 vectors in a single PutVectors API call to its assigned S3 Vectors index. Each index supports up to 2,500 vector insertions and deletions per second, combined. During heavy ingestion loads, the AWS SDK automatically retries requests. All steps checkpoint their outputs, so an interrupted run resumes where it stopped instead of re-ingesting hundreds of millions of vectors. Retries and restarts are safe because vector writes are idempotent.

The query service routes each query vector only to the index family for the specification used to create it. Retrieval follows one of two paths, as shown in the following figure. Text, image, and map-point searches issue QueryVectors requests to every shard in the selected family, then merge the candidates by cosine similarity. A property search can create query vectors for the top-down and oblique families, query each family, and combine their matches by property identifier. Tile and property vectors are not mixed in one family.

For a heatmap, the selected map area determines the exact tiles to score, so Zero-shot Search does not search the corpus for nearest matches. Instead, it computes each tile’s deterministic key, routes the key with the same hash used during ingestion, retrieves the vectors with GetVectors, and scores them against the query embedding in the query service.

Retrieval: similarity searches fan out with QueryVectors and merge by cosine similarity. Heatmaps route deterministic keys to parallel GetVectors reads and score them in the query service.

Index design for Zero-Shot Search

Four decisions shape how Zero-shot Search lays out its indexes: what makes vectors comparable, how each vector is named, how the corpus is divided, and what each vector carries alongside its embedding.

One index family per embedding specification

Embeddings are comparable only when they are produced in the same way. Changing the crop, resolution, or padding can produce a vector with the expected dimension but incompatible semantics. Dimension validation alone does not detect this mismatch.

Zero-shot Search prevents this mismatch by assigning each complete embedding specification to its own index family. Two specifications can have the same dimension but use different models, input preparation, or metadata, so their vectors are not mixed. Depending on the specification, each embedding vector has 768, 1,024, or 1,152 dimensions. Tile, top-down property, and oblique property vectors use separate index families, even when they have the same dimension. Within each shard, S3 Vectors fixes the vector dimension and distance metric and supports similarity search, retrieval by key, and metadata filtering.

Generation pipelines use the shared registry to select the correct model and reject shape mismatches before writing vectors. The service creates each query vector from one specification and routes it only to that specification’s index family, so a query vector never spans dimensions. A property search that uses both view types creates one query vector per specification, queries both families, and combines the results at the property layer.

Deterministic keys

A heatmap needs to retrieve the known tiles inside a selected map area rather than search the entire corpus for similar scenes. To support this direct lookup, a tile vector key combines the survey identifier, zoom level, and tile coordinates: {survey}_{z}_{x}_{y}. Any caller that knows the tile can compute the key without a lookup. Zero-shot Search enumerates the tiles covering an area, computes their keys, and fetches them directly with GetVectors.

An alternative implementation using opaque keys would require a separate geography-to-key index that Nearmap would need to write, scale, and keep consistent. Deterministic keys remove that component from the architecture.

Hash sharding

Each shard is one S3 Vectors index within an index family. Zero-shot Search encodes the shard count and shard number in each index name, so <model>-tiles-vert-512-n10-3 identifies the S3 Vectors index used as shard three of ten. In that name, 512 refers to the input preparation used to produce the embeddings, not the vector dimension. The service discovers the topology with ListIndexes instead of reading it from configuration.

The routing formula is MD5(key) modulo shard count. Writers and readers import the same function, so they use the same shard assignment. Sharding provides parallelism on both paths. During ingestion, each worker owns one shard and reads its pre-sharded file instead of scanning the whole dataset. During a query, Zero-shot Search searches the smaller S3 Vectors indexes concurrently and merges the results.

This fixed topology simplifies the system by eliminating the need for routing state at runtime. If a different shard count is required, Zero-shot Search must reload all embeddings from Parquet files in Amazon S3. The process is relatively inexpensive and does not require GPU workers.

Non-filterable metadata

Each vector in S3 Vectors can include application-defined metadata as key-value pairs that describe the embedding’s source and context. Searches can filter on selected fields, and responses can return metadata with each match. At index creation, up to 10 keys can be designated as non-filterable through nonFilterableMetadataKeys. Each vector remains subject to the documented metadata size limits.

Zero-shot Search keeps fields used to narrow searches filterable: the survey identifier, capture date, address, area, and view direction. It stores parcel geometry and image references as non-filterable metadata. Each parcel boundary is stored as base64-encoded Well-Known Binary (WKB), a standard format for geometric shapes, alongside the image references needed to reproduce the view.

For property searches, the initial QueryVectors fan-out ranks candidates without returning metadata. The service then uses GetVectors to retrieve the selected vectors and their metadata. Storing this context in S3 Vectors allows the service to outline the property and render the associated images without looking up the original Parquet files.

Non-filterable keys are fixed when an index is created. Changing the classification requires new indexes and re-ingestion.

Conclusion

In this post, we showed how Nearmap uses Amazon S3 Vectors to retain and search more than 1 billion embeddings generated from 107 aerial surveys. S3 Vectors allows Nearmap to retain this growing corpus without provisioning, resizing, or rebalancing a search cluster. Zero-shot Search enables Nearmap to help customers answer a broader range of questions using current and historical imagery without training and deploying new detectors.

To get started, see the Amazon S3 Vectors documentation and use the AWS Pricing Calculator to estimate costs for your scenario. For more customer stories and best practices, visit the AWS Storage Blog.