How NASA JPL Processes 70 TB of Satellite Data Products a Day Using Amazon EC2 Auto Scaling with Spot Instances

JPL is a federally funded research and development center and is managed by the California Institute of Technology. The center has more than 40 active space and airborne missions. When it launches in early 2025, NISAR will be one such mission. The mission will observe and measure some of the most complex changes on Earth, using space-borne synthetic aperture radar (SAR) satellite observations to understand how changes in land and ice surfaces contribute to climate change impacts, natural disasters, and gradual Earth surface changes. SAR is an active instrument and can see through clouds, which is especially useful during natural disasters such as hurricanes and floods.

NISAR is a dual-frequency satellite that will provide radar data of nearly every land and ice surface on the planet. That data will help scientists forecast, compare, and assess changes such as the rate of ice melts, deforestation, or changes in land before and after natural disasters such as earthquakes, leading to new insights into the planet’s dynamics.

JPL uses on-premises infrastructure to process the satellite data from other missions, but the volume and frequency of the data generated from the NISAR mission is expected to be around 70 TB per day. To process that data, NISAR anticipated storage and compute needs that exceeded on-premises capacity, along with the need for flexibility to scale up workloads for lower-latency response needs. JPL had used AWS for an earlier mission, so instead of investing in more on-premises infrastructure—which would have been costlier, more time-consuming, and not as flexible—the center decided to adapt the previously used AWS solution to process NISAR data (see figure 1).

NISAR’s satellite data will be stored in Amazon Simple Storage Service (Amazon S3), which provides object storage built to retrieve any amount of data from anywhere. To trigger workflows to start processing and transforming the data, JPL will use Amazon Simple Queue Service (Amazon SQS)—which offers fully managed queuing for microservices, distributed systems, and serverless applications—and Amazon Simple Notification Service (Amazon SNS), which is a fully managed Pub/Sub service for A2A and A2P messaging. The data transformations will trigger production rules that help determine whether all the complex conditions and requirements to further process the data have been met. Those rules will run in a combination of AWS Lambda—which helps run code without thinking about servers or clusters—for lightweight rules and containers for the more complex production rules.

The rules will be checked and the data processed in distributed compute nodes on Amazon EC2, which provides secure and resizable compute capacity for virtually any workload. Since most of the processing system is fault tolerant, JPL will take advantage of Spot Instances for scaling. Using Spot Instances, JPL has the flexibility of paying for usage per hour or per second without needing to commit to a specific amount of compute usage, making it even more cost-effective. For critical processing, JPL will also use Amazon EC2 On-Demand pricing, paying for compute capacity by the hour or second with no long-term commitments and On-Demand Capacity Reservations to create and manage reserved capacity on Amazon EC2. The data processing across Spot Instances and On-Demand will be done through automatic scaling fleets that react to the dynamic workload demands.

The data will be processed using algorithms packaged in Docker containers, and the output data will be stored in Amazon S3 buckets. Amazon EC2 provides a variety of compute offerings, such as multi-core CPUs and GPUs, which many of NISAR’s algorithms need for science data processing. The transformed data will be processed to higher and higher levels of derived data products. For example, raw radar signals are decoded, reformatted, and then focused into high-resolution, complex-valued images. These precursor data products are used to generate higher-level data products, such as displacement maps. Once the final data products are generated and stored in Amazon S3, a notification will be sent to the Alaska Satellite Facility Distributed Active Archive Center (ASF DAAC) to ingest the data products into its archive, which is also in the us-west-2 AWS Region. Since the Near Earth Network’s ground stations, the ASF DAAC, and JPL Science Data System are collocated in the same AWS Region, no egress cost will be incurred. After the products reach the DAAC, the relevant JPL teams will be notified through Amazon SNS that the data has been processed (see figure 2). The cycle will then repeat with each downlink pass.

This system processes about 4.4 TB of downlinked data daily, generating up to 70 TB of final data products. JPL uses Spot Instances best practices to make sure the algorithms run more optimally on best-sized multi-core Amazon EC2 instances, which helps access more diversified Spot Instances capacity fleets. The system can scale automatically to process data at low latency without significantly increasing compute costs. Automatic scaling provides the required processing capacity when JPL needs it, including for urgent-response scenarios such as natural disasters. Using Spot Instances also increases resiliency, providing JPL with fault-tolerant computing at scale.

JPL is preparing for NISAR’s launch and is looking forward to making all standard data products generated in AWS available to the scientific community. After launch, NISAR will undergo a 90-day commissioning period before commencing with its main science operational phase, during which it will capture SAR acquisitions of nearly all of Earth’s land and ice surfaces every 6–12 days. The large, scalable, and cost-effective compute capacity provided by Spot Instances will help JPL make this data available quickly as final data products at optimized costs.

Those data products will be archived and distributed from NASA’s Earthdata Cloud, a data lake being built in AWS by DAACs under NASA’s Earth Science Data and Information System Project. NASA’s open science policy will make the archived data—from NISAR and other missions—available to the open scientific community. The consistent and growing collection of data will help scientists develop multiple time series to understand big and subtle changes on the planet, building a better understanding of Earth over time.