The China-Brazil Earth Resources Satellite Mission

Guest post by Frederico Liporace, AMSKepler, Diego Magalhaes, WWPS Sr SA, LATAM, AWS and Melissa Ravanini, WWPS Sr SA, Brazil, AWS

The China-Brazil Earth Resources Satellite (CBERS) satellites are the result of a cooperation agreement between Brazilian and Chinese space agencies (INPE and CAST, respectively) started in 1988. Since then, five satellites have launched (CBERS-1/2/2A/3/4).

The mission generates images from Earth with characteristics similar to USGS’ Landsat and ESA’s Sentinel-2 missions. In 2004, INPE announced that all CBERS-2 images would be available at no charge for the public. It was the first time this distribution model was used for medium-resolution satellite imagery. Now, this model is used for all CBERS satellites.

The cloud-based computing environment enables a wide range of new applications that make use of satellite imagery, but for this to be effective, you must ‘bring’ the processing as close to the data as possible. The conventional model of distributing images through a catalog where the user browses and selects images for downloading and local processing works well if your application requires just tens of images, but it does not scale to support the more data-hungry applications that are now possible, such as continuous monitoring of large areas.

“The free data policy adopted by INPE in 2004 enabled new business and increased the number of users of earth observation data in Brazil, and also encouraged USGS and ESA to open Landsat and Sentinel data access, respectively.  Certainly, the initiative, CBERS on AWS, will contribute to create new users and new applications for CBERS (China-Brazil Resources Earth Satellite),” said Leila Fonseca, Head of Earth Observation General Coordination, National Institute for Space Research – INPE.

AMS Kepler has been involved in the development of the system used to record, process, and distribute CBERS image data since the start of the mission. AWS Public Datasets, Landsat on AWS and Sentinel on AWS, are used for some remote sensing applications, and we wanted to do the same with CBERS data. Through an Earth on AWS Cloud Credits for Research Program grant, we were able to start building something similar for CBERS data.

Image 1. CBERS on AWS current architecture

Architecture

We use the distribution channel already available from INPE for the general public to obtain the images. Right now, this process is done manually, as there is no API to retrieve images from the catalog. We have created a robot (scripted in Selenium, an open source tool) that runs and simulates image requests to INPE’s catalog.

Once the images are available for download, INPE sends an e-mail with the download links. This e-mail is automatically processed by an Amazon Simple Email Service (SES) ruleset, which invokes an AWS Lambda function to extract the download URL from the text.

We then create State Machines to download the files in parallel. Amazon Step Functions are great for doing this, since they include support for error detection and retry logic when downloading the files.

Once the files are stored in Amazon Simple Storage Service (Amazon S3), we start AWS Batch jobs to process the files. This processing step consists of: uncompressing the files; converting the GeoTIFF to Cloud Optimized GeoTIFFs (COG), a format more suitable to serve the image content through a network; generating a subsampled image for browsing; and processing the image metadata. This processing is done on top of Docker images made available by Development Seed.

AWS Batch priority queues enable us to define different priorities for this processing step, which is useful when you are ingesting a lot of archive data, and it may be interesting to prioritize images with less cloud coverage, for instance, when processing.

The metadata files are generated in a way to make indexing through Amazon Athena faster by using partitions based on the image acquisition date. Amazon Athena is then used directly on the metadata to provide serverless indexing of the entire archive.

Next steps

We started the project ingesting archive data from CBERS-4 MUX, a 20m Red, Green, Blue and NIR camera, with new images being included one day after they are available in INPE’s catalog. Once the operation for MUX archive is stable, it is possible to extend the archive with other CBERS cameras currently flying. It is also possible to add all past CBERS missions into the archive.

We believe that this will be a major step to widen the usage of CBERS data and hope to continue the work for the future CBERS-4A and Amazonia-1, a Brazilian mission planned to be launched in 2019.