How to detect wildfire smoke using Amazon Rekognition

According to the National Oceanic and Atmospheric Administration (NOAA), the effects of climate change, such as increased heat, extended drought, and more, have been a key driver in the increasing risk and extent of wildfires in the United States. Wildfires, then, contribute to climate change through greenhouse gas emissions, destruction of ecosystems, and lost carbon storage in the destruction of forests. In addition to the risk to life and biodiversity, wildfires can destroy and disrupt property, essential infrastructure, communications, air quality, and more.

Since wildfires can double in size and intensity every three to five minutes, early detection and reduced response times are essential to help reduce and prevent catastrophic loss of life and property. Cloud technologies, including artificial intelligence (AI) and machine learning (ML), can help with this. Disaster response teams can use the cloud to create wildfire detection workflows that automatically deploy rapid notifications to fire incident response teams, so disaster response teams are alerted to the first signs of wildfire formation.

In this post, learn a high-level architecture to create a solution with Amazon Web Services (AWS) that uses AI and ML to identify and classify wildfire smoke imagery and then rapidly alert and inform first responders about the location and condition of a fire incident. This walkthrough also explores the results of training and testing an ML model with a sample dataset to identify wildfire smoke.

Solution overview: Using AI to detect and locate wildfire smoke

Figure 1 features a high-level architecture for use cases designed to identify and classify wildfire smoke from fixed tower sensors, and then send alerts to first responders with detailed information, such as the location of the incident, images, and other data to assist wildfire response teams in evaluating a potential wildfire before arrival on the incident scene. Similar use cases may also use a mobile unmanned aerial vehicle (UAV) such as a drone or a network of fixed sensor and mobile, low-altitude, remote sensing devices, including multispectral bands for thermal imaging, and more. For example, San Diego Gas & Electric is using AWS to prevent wildfires with drones and AI and ML.

Figure 1. Wildfire smoke detection architecture diagram, using imagery from a fixed field tower. The major components shown in this figure are Amazon Kinesis Video Streams, Amazon Rekognition Custom Labels, AWS Lambda, and Amazon Simple Notification Service (Amazon SNS).

The architecture features the following components:

1. Remotely-sensed data collection

Remote towers with attached sensors for collecting optical imagery collect images or videos. Adding sensors that capture thermal imagery would increase the accuracy of positively labeling wildfires and wildfire smoke.

2. Amazon Kinesis Video Streams

The imagery collected at the tower location is securely streamed into Amazon Kinesis Video Streams. Kinesis Video Streams extracts images to an Amazon Simple Storage Service (Amazon S3) bucket every 30 seconds.

3. Amazon Rekognition Custom Label machine learning model

When the video stream images are posted in the Amazon S3 bucket, an AWS Lambda function sends the images to an Amazon Rekognition Custom Label service, which helps identify the objects in images based on a customized, trained model. The Amazon Rekognition Custom Labels response includes whether an object—in this case, wildfire smoke—is detected in the image, and if detected, the bounding box of the object detected in the image.

3a. Training

Organizations can train a machine learning model in Amazon Rekognition Custom Label derived from the wildfire data. The example use case in this blog post uses the High Performance Wireless Research and Education Network (HPWREN) dataset from the University of California San Diego to train and test the sample model, but organizations should make sure to use the appropriate vegetative and contextual training data for their needs. This model in action would benefit and improve from additional data and relabeling and retraining, including incorporating future observations with Amazon SageMaker Ground Truth.

Deployed models should be updated and contextually represented by the type of wildfire smoke imagery that best captures regional differences – meaning that models deployed in specific vegetative landscapes should include training and retraining data for regional models with similar landscapes, as geographic differences in landscape may lessen accurate object detection.

3b. Model deployment hosting

When a suitable model with satisfactory accuracy is trained and tested in Amazon Rekognition Custom Label, Amazon Rekognition hosts the model and a classification endpoint. The classifier is accessed via the AWS Command Line Interface (AWS CLI) or the SDK for Python library. Image file location and information, such as the minimum confidence score, serve as input parameters for the Amazon Rekognition Custom Labels service detect_custom_labels() method.

4. Notification for fire response and management teams

When a positive wildfire smoke object label is detected, services like Amazon Simple Notification Service (Amazon SNS) and Amazon Simple Email Service (Amazon SES) can be used to build alerting mechanisms to alert the fire response teams and, in this case, include imagery, location, and other metadata associated with the fire incident, possibly including existing environmental conditions like wind, weather, vegetation metrics, and wildfire risk metrics. In this use case, an AWS Lambda function triggers the notification services when an object is positively detected and included in the response from the Amazon Rekognition Custom Label service. These notifications can be sent to the nearest fire companies in a database or by creating a geofence for fire company service areas using Amazon Location Service.

5. Additional contextual data regarding wildfire fuel from First Street Foundation Wildfire Model

First Street Foundation, a nonprofit research and technology group, supplies 30 meter resolution data on wildfire hazards. Integrating this data with the Amazon Location Service Maps feature and sending the data to fire response teams can assist with incident response. More AWS services can be integrated into the architecture to help organizations and scientists generate important contextual data, like vegetative conditions and more.

Training the model: Using Amazon Rekognition Custom Labels for wildfire smoke detection with HPWREN observations

To train the ML model for this example use case, this example walkthrough uses the HPWREN, which includes data that is best described as mountaintop observations of fire-prone areas and bounding box-labeled data.

I used 592 images (80% of 740 total) to train the model and 148 images (20% of 740 total) to test the model. Some data needed to be transformed from the source .xml data for the bounding boxes to adhere to the .json specifications defined by the object manifest file schema in Amazon Rekognition.

Although the initial Amazon Rekognition automatic label detection identified “smoke” in some of the training data, it was often at lower confidence (around 60% in some tests). The model identified more prominent landscape features such as “mountain” or “nature” in the training images with higher confidence. As a result, I developed the new purpose-trained model using Amazon Rekognition Custom Labels to better assist in creating a more wildfire smoke-focused object detection model. While Amazon Rekognition Image recognition service models are helpful for quick, wide use case classification implementations, generating object-specific models for bespoke use cases may increase the confidence in specific object detection.

Figure 2. Example training data with Amazon Rekognition Custom object label as wildfire “smoke.”

Figure 2. Example training data with Amazon Rekognition Custom object label as wildfire “smoke.”

After I transformed the bounding box .xml data to the manifest.json format for the Amazon Rekognition Custom Labels service, I added the collection of labeled images to an Amazon S3 bucket and uploaded the manifest file via the SDK for Python.

Figure 3. Images from the sample dataset in the Amazon Rekognition Custom Labels dashboard in the AWS Management Console.

Figure 3. Images from the sample dataset in the Amazon Rekognition Custom Labels dashboard in the AWS Management Console.

Evaluating model performance

After training the model on the 592 images (80% of 740 total) from the HPWREN dataset, I evaluated the model performance by testing it against the test dataset of 148 images (20% of 740 total). The model performance evaluation metrics produced an F1 score of 0.971, a precision score of 0.973, and a recall score of 0.970.

Figure 4. Example application running Amazon Rekognition Custom Labels object detection and response with detected object bounding box and confidence score.

Figure 4. Example application running Amazon Rekognition Custom Labels object detection and response with detected object bounding box and confidence score.

Assumptions and limitations

The use case outlined in this blog post uses data sourced from Southern California with a collection of highly similar image contexts that includes angles, elevations, and sensor conditions in relation to the local topography. For those looking to deploy this type of use case, make sure to incorporate additional data in your model training that accounts for the context in how the imagery is generated. This is also applicable for use cases that source data from thermal detection and remote sensing. If thermal imagery is available, consider implementing an ensemble approach to improve model performance and confidence. An ensemble approach would include multiple models, for example, having positive object detections from thermal sensor bands for heat signature associated with wildfire incidents as well as optical sensor data detecting wildfire smoke in the vicinity increases confidence of a true positive wildfire incident.

Additional use cases for wildfire response support

As average global temperatures rise and some regions experience greater risk and exposure to wildfires, detection makes up an important component of reducing risk associated with wildfires. Here are follow-up use cases that could also use the breadth and depth of AWS services:

Identifying fire lookout blind spots (viewshed white space analysis)

Although many fire lookouts have been decommissioned and/or retired, these assets can be useful in fixed, ground-based Earth observation (EO): fire lookouts can act as infrastructure on which to add EO cameras, and any historical data from these lookouts can help train ML models. Operating fire lookout towers can serve as a pilot for more rapid wildfire detection. If the results of these pilot studies prove helpful in reducing incident response times, a large-scale implementation may benefit from a country-wide viewshed analysis with high resolution and/or Light Detection and Ranging (LiDAR)-derived digital elevation models (DEM) to identify regions for intervention of placing additional fire lookouts or rehabbing/repurposing retired fire lookout towers. Additionally, advanced geospatial analysis could identify sensor viewshed service areas, identifying gaps in covered areas as well as optimal placement to build the most effective network of wildfire detection devices.

Wildfire smoke detection location triangulation to increase location precision

If a network of sensors is deployed with some viewshed overlap, it’s possible to generate triangulated positioning of smoke detection that renders on a map in messaging or an HTML interactive map endpoint where the area of interest (AOI) or incident location is updated in near real time. Providing fire response teams with more precise location information would improve incident response time and help with strategies for deploying response teams.

Figure 5. Example notification from Amazon Location Service Maps depicted on an Amazon S3 pre-signed URL HTML page.

Routing and response resource allocation with Amazon Location

When a wildfire is detected, response teams may benefit from additional features from Amazon Location Service, such as optimal routing to the identified site, trackers, basemaps, geofencing, and more to best deploy wildfire response resources and personnel.

Conclusion

This post presents an example architecture for detecting smoke associated with terrestrial wildfires from fixed remote sensing devices using Amazon Rekognition Custom Labels. As with most machine learning and object detection, it’s important to include retraining and redeploying models based on more up-to-date data to keep improving models.

Learn more about Amazon Rekognition Custom Labels object detection and Amazon Location Service.