This Guidance shows how build an automated geospatial insights engine on AWS to improve supply and demand forecasting and risk management. Many industries depend upon intelligence and insights gained from processed geospatial and Earth-observation data from sources like satellite and aerial imagery and remote sensing. This Guidance couples this data with mechanistic and AI-powered models, so you can enhance forecasting, automate risk mitigation, and meet regulations.
Please note: [Disclaimer]
Architecture Diagram
-
Overview
-
UI Module
-
Region Module
-
Scheduler Module
-
Executor Module
-
Results Module
-
Notification Module
-
Overview
-
This architecture diagram shows an overview of the automated geospatial analysis engine, specifically the key modules and their interactions.
Click to enlarge
Step 1
The UI module is responsible for transforming analysis results into displayable visual assets. It uses a geospatial mapping service to obtain base map data, and it renders interactive maps on the frontend using a map-rendering library.
Step 2
The region module manages the hierarchical structures of groups, regions, polygons, and states.
Step 3
The scheduler module manages the scheduling of engine processing tasks based on each region’s processing configuration.
Step 4
The scheduler module also subscribes to new satellite images using an Amazon Simple Notification Service (Amazon SNS) topic hosted in Open Data on AWS.
Step 5
The executor module manages the execution of the engine analysis, invoked by the schedule module or by new sentinel images in Amazon Simple Storage Service (Amazon S3) that match the created regions.
Step 6
The results module acts as an intermediary between the region and executor modules, transforming and publishing region data to a SpatioTemporal Asset Catalog (STAC) server and mapping execution results to their corresponding region.
Step 7
The notification module manages your subscriptions to notifications generated by other modules.
Step 8
Note: The STAC server module is an AWS Cloud Development Kit (AWS CDK) port of stac-server, an execution of the STAC API specification for searching and serving metadata for geospatial data, including but not limited to satellite imagery.
Step 1
The UI module is responsible for transforming analysis results into displayable visual assets. It uses a geospatial mapping service to obtain base map data, and it renders interactive maps on the frontend using a map-rendering library.
Step 2
The region module manages the hierarchical structures of groups, regions, polygons, and states.
Step 3
The scheduler module manages the scheduling of engine processing tasks based on each region’s processing configuration.
Step 4
The scheduler module also subscribes to new satellite images using an Amazon Simple Notification Service (Amazon SNS) topic hosted in Open Data on AWS.
Step 5
The executor module manages the execution of the engine analysis, invoked by the schedule module or by new sentinel images in Amazon Simple Storage Service (Amazon S3) that match the created regions.
Step 6
The results module acts as an intermediary between the region and executor modules, transforming and publishing region data to a SpatioTemporal Asset Catalog (STAC) server and mapping execution results to their corresponding region.
Step 7
The notification module manages your subscriptions to notifications generated by other modules.
Step 8
Note: The STAC server module is an AWS Cloud Development Kit (AWS CDK) port of stac-server, an execution of the STAC API specification for searching and serving metadata for geospatial data, including but not limited to satellite imagery.
-
UI Module
-
This architecture diagram shows how to set up the user interface (UI), manages authentication and authorization, and query analysis results.
Click to enlarge
Step 1
Load the sample React application from the S3 bucket, using Amazon CloudFront to maintain low latency and high performance.Step 2
The React application renders maps using Amazon Location Service.Step 3
The React application overlays the base map with the processed satellite images by querying the tiler API on Amazon API Gateway.Step 4
Amazon Cognito delivers a token for your authentication.Step 5
Amazon Verified Permissions uses the role encoded in your token to perform fine-grained authorization.Step 6
Once you have been authenticated, the AWS Lambda tiler-API function queries the STAC server’s metadata of the polygon execution result (based on the current view port of the map).
Step 7
The Lambda tiler-API function generates signed URLs for all image assets specified in the STAC items and returns them to the React application for rendering.
Step 1
Load the sample React application from the S3 bucket, using Amazon CloudFront to maintain low latency and high performance.Step 2
The React application renders maps using Amazon Location Service.Step 3
The React application overlays the base map with the processed satellite images by querying the tiler API on Amazon API Gateway.Step 4
Amazon Cognito delivers a token for your authentication.Step 5
Amazon Verified Permissions uses the role encoded in your token to perform fine-grained authorization.Step 6
Once you have been authenticated, the AWS Lambda tiler-API function queries the STAC server’s metadata of the polygon execution result (based on the current view port of the map).
Step 7
The Lambda tiler-API function generates signed URLs for all image assets specified in the STAC items and returns them to the React application for rendering.
-
Region Module
-
This architecture diagram demonstrates the hierarchical structure of groups, regions, polygons, and states.
Click to enlarge
Step 1
Create group, region, and polygon resources using the region API through API Gateway. During the process of setting up a new region, you’ll be able to define the schedule and prioritization for the geospatial data processing tasks associated with it.
Step 2
Amazon Cognito delivers a token for your authentication.
Step 3
Verified Permissions uses the role encoded in your token to perform fine-grained authorization. Your role is stored in custom:role, a custom attribute of Amazon Cognito.
Step 4
Once you have been authenticated, API Gateway forwards the request to Lambda.
Step 5
The group and region data is stored in an Amazon DynamoDB table.
Step 6
A region’s resource changes emit events to an Amazon EventBridge event bus. Events can be tracked to update other components of the framework.
Step 7
Lambda records a segment with details about invoking and running the function and sends it to AWS X-Ray.
Step 1
Create group, region, and polygon resources using the region API through API Gateway. During the process of setting up a new region, you’ll be able to define the schedule and prioritization for the geospatial data processing tasks associated with it.
Step 2
Amazon Cognito delivers a token for your authentication.
Step 3
Verified Permissions uses the role encoded in your token to perform fine-grained authorization. Your role is stored in custom:role, a custom attribute of Amazon Cognito.
Step 4
Once you have been authenticated, API Gateway forwards the request to Lambda.
Step 5
The group and region data is stored in an Amazon DynamoDB table.
Step 6
A region’s resource changes emit events to an Amazon EventBridge event bus. Events can be tracked to update other components of the framework.
Step 7
Lambda records a segment with details about invoking and running the function and sends it to AWS X-Ray.
-
Scheduler Module
-
This architecture diagram shows how to schedule the engine's processing tasks, based on each region's processing configuration.
Click to enlarge
Step 1
The scheduler module subscribes to region events published by the region module.
Step 2
If a region is configured with a recurrent schedule, it creates a processing schedule for that region using an EventBridge scheduler.
Step 3
When a schedule is invoked, the EventBridge scheduler puts messages into the job queue for the region to be processed.
Step 4
The scheduler module also subscribes to new-satellite-image notifications through an Amazon SNS topic hosted in Open Data on AWS. Amazon Simple Queue Service (Amazon SQS) queues messages.Step 5
When a new satellite image becomes available in the Open Data on AWS bucket, Lambda queries the STAC server module to check if the image includes the geographic region where the regions are located.
Step 6
In the event of spatial intersection, Lambda puts a processing-task message in the job queue.
Step 7
Lambda records a segment with details about invoking and running the function and sends it to X-Ray.
Step 1
The scheduler module subscribes to region events published by the region module.
Step 2
If a region is configured with a recurrent schedule, it creates a processing schedule for that region using an EventBridge scheduler.
Step 3
When a schedule is invoked, the EventBridge scheduler puts messages into the job queue for the region to be processed.
Step 4
The scheduler module also subscribes to new-satellite-image notifications through an Amazon SNS topic hosted in Open Data on AWS. Amazon Simple Queue Service (Amazon SQS) queues messages.Step 5
When a new satellite image becomes available in the Open Data on AWS bucket, Lambda queries the STAC server module to check if the image includes the geographic region where the regions are located.
Step 6
In the event of spatial intersection, Lambda puts a processing-task message in the job queue.
Step 7
Lambda records a segment with details about invoking and running the function and sends it to X-Ray.
-
Executor Module
-
This architecture diagram demonstrates the execution of the region analysis invoked by the schedule module.
Click to enlarge
Step 1
The executor module subscribes to the job queue of the scheduler module, which queues messages when a region is scheduled for processing.
Step 2
The queue invokes the executor module using Lambda, which then places the job in the right-priority AWS Batch queue based on the region configuration.
Step 3
AWS Batch will pick up a job from these queues and launch tasks using AWS Fargate for Amazon Elastic Container Service (Amazon ECS) to process the polygons associated with the region in parallel.
Step 4
The Amazon ECS tasks retrieve the satellite images from the S3 bucket hosted in Open Data on AWS.
Step 5
These tasks then process the images and store the results in an S3 bucket.
Step 6
Once the execution finishes, the Amazon ECS tasks publish the results’ metadata to an EventBridge event bus.
Step 1
The executor module subscribes to the job queue of the scheduler module, which queues messages when a region is scheduled for processing.
Step 2
The queue invokes the executor module using Lambda, which then places the job in the right-priority AWS Batch queue based on the region configuration.
Step 3
AWS Batch will pick up a job from these queues and launch tasks using AWS Fargate for Amazon Elastic Container Service (Amazon ECS) to process the polygons associated with the region in parallel.
Step 4
The Amazon ECS tasks retrieve the satellite images from the S3 bucket hosted in Open Data on AWS.
Step 5
These tasks then process the images and store the results in an S3 bucket.
Step 6
Once the execution finishes, the Amazon ECS tasks publish the results’ metadata to an EventBridge event bus.
-
Results Module
-
This architecture diagram shows how to transform data into STAC items, publish them to the STAC server, and map execution results to corresponding regions.
Click to enlarge
Step 1
Interact with the results module through API Gateway.Step 2
Amazon Cognito delivers a token for your authentication.
Step 3
Verified Permissions uses the role encoded in your token to perform fine-grained authorization.Step 4
Once you have been authenticated, API Gateway forwards the request to Lambda.
Step 5
The data containing the execution result details—such as the unique ID, status, last updated time, and execution specifics—is stored in a DynamoDB table.
Step 6
The results module subscribes to groups- and regions-change events from the region module and region-analysis-result events from the executor module.
Step 7
The results module transforms these events into a STAC item and publishes it to the Amazon SNS STAC server module topic.
Step 8
Lambda records a segment with details about invoking and running the function and sends it to X-Ray.
Step 1
Interact with the results module through API Gateway.Step 2
Amazon Cognito delivers a token for your authentication.
Step 3
Verified Permissions uses the role encoded in your token to perform fine-grained authorization.Step 4
Once you have been authenticated, API Gateway forwards the request to Lambda.
Step 5
The data containing the execution result details—such as the unique ID, status, last updated time, and execution specifics—is stored in a DynamoDB table.
Step 6
The results module subscribes to groups- and regions-change events from the region module and region-analysis-result events from the executor module.
Step 7
The results module transforms these events into a STAC item and publishes it to the Amazon SNS STAC server module topic.
Step 8
Lambda records a segment with details about invoking and running the function and sends it to X-Ray.
-
Notification Module
-
This module shows how to manage subscriptions to notifications generated by other modules.
Click to enlarge
Step 1
Create a subscription for region processing events through API Gateway.
Step 2
Amazon Cognito delivers a token for your authentication.
Step 3
Verified Permissions uses the role encoded in your token to perform fine-grained authorization.
Step 4
Once you have been authenticated, API Gateway forwards the request to Lambda.
Step 5
Your list of subscription data is stored in a DynamoDB table.Step 6
Lambda subscribes you to an Amazon SNS region topic. (It will create the topic if it does not already exist.)Step 7
Lambda subscribes you to an EventBridge event bus for region processing events published by the execution module.
Step 8
Lambda transforms the events into notification messages and publishes them to you (or other subscribed users) through Amazon SNS.
Step 9
Lambda records a segment with details about invoking and running the function and sends it to X-Ray.
Step 1
Create a subscription for region processing events through API Gateway.
Step 2
Amazon Cognito delivers a token for your authentication.
Step 3
Verified Permissions uses the role encoded in your token to perform fine-grained authorization.
Step 4
Once you have been authenticated, API Gateway forwards the request to Lambda.
Step 5
Your list of subscription data is stored in a DynamoDB table.Step 6
Lambda subscribes you to an Amazon SNS region topic. (It will create the topic if it does not already exist.)Step 7
Lambda subscribes you to an EventBridge event bus for region processing events published by the execution module.
Step 8
Lambda transforms the events into notification messages and publishes them to you (or other subscribed users) through Amazon SNS.
Step 9
Lambda records a segment with details about invoking and running the function and sends it to X-Ray.
Get Started
Deploy this Guidance
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaper
X-Ray provides complete tracing and monitoring capabilities to help you identify performance bottlenecks and troubleshoot issues. It enables you to visualize and analyze the components of this Guidance, such as API calls, Lambda functions, and AWS Step Functions workflows. Finally, AWS CloudFormation provisions and manages the required resources, enabling automated deployments and changes.
-
SecurityRead the Security whitepaper
Amazon Cognito provides secure user authentication mechanisms so that only authorized users can access your applications and resources. AWS Identity and Access Management (IAM) policies and roles let you control access to resources according to the principle of least privilege. Additionally, Verified Permissions provides a centralized and efficient system for managing access to your custom applications. Through a policy-based approach that uses the Cedar policy language for defining fine-grained permissions, it promotes consistency and maintainability in the authorization process. It also separates the authorization process from the application code so that you can easily update and manage permissions.
-
ReliabilityRead the Reliability whitepaper
AWS Batch automatically scales the number of compute resources based on the number of jobs in the queue. As a result, this Guidance provisions the right amount of resources for satellite image processing, reducing the risk of job failures due to resource constraints. Amazon SQS and EventBridge maintain durability and persistence by decoupling components through asynchronous messaging and by storing messages redundantly across multiple Availability Zones. This makes applications more resilient: if an individual component fails, messages are temporarily stored in the queue and processed when the component recovers.
-
Performance EfficiencyRead the Performance Efficiency whitepaper
DynamoDB lets you define a partition key (and an optional sort key) for your table to enhance performance efficiency. By using defined keys to distribute your data across servers and partitions in addition to using appropriate indexes, you can optimize data access patterns. Additionally, Amazon S3 offers seamless scalability, enabling your applications to handle high request rates effortlessly. For example, each partitioned Amazon S3 prefix can sustain at least 3,500 requests per second for PUT, COPY, POST, and DELETE operations. Each prefix can also sustain up to 5,500 requests per second for GET and HEAD operations. Because there are no restrictions on the number of prefixes you can create within a bucket, you can partition your data effectively to achieve optimal performance and scalability. And by encoding the region, polygon, or results ID in the file prefix when storing satellite analysis images, you can enable parallel processing and listing.
-
Cost OptimizationRead the Cost Optimization whitepaper
The serverless and event-driven nature of Lambda, EventBridge, Amazon SQS, and AWS Batch means that they automatically scale based on demand and don’t overprovision capacity. This helps you optimize costs, because you only pay for the resources you use. Additionally, EventBridge and Amazon SQS remove the need for you to set up polling-based architectures, helping you further reduce costs.
-
SustainabilityRead the Sustainability whitepaper
This Guidance uses serverless services for compute and storage—including Lambda, AWS Batch, and DynamoDB—so they automatically scale to zero when not in use. This removes the need for always-on infrastructure and reduces the overall environmental impact of your workloads. Additionally, Lambda uses AWS Graviton2 processors, which are designed to be more energy-efficient than traditional x86-based processors. They consume less power while delivering comparable performance, reducing your carbon emissions.
Related Content
[Title]
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.
References to third-party services or organizations in this Guidance do not imply an endorsement, sponsorship, or affiliation between Amazon or AWS and the third party. Guidance from AWS is a technical starting point, and you can customize your integration with third-party services when you deploy the architecture.