AWS Public Sector Blog

Enhance the citizen experience with deep learning-powered suggestions

Every interaction a citizen has with government is an opportunity to enhance the customer experience with rich citizen engagement and direct lines of feedback. Recent federal priorities have been set to improve the citizen experience when interacting with government.  Deep learning image analysis and high-performant yet cost-effective architectures on Amazon Web Services (AWS) can modernize constituent experiences and improve digital interactions with government.

For example, citizens want to report issues to their local governments in a fast and simple manner and not have to worry about identifying the right government agency or phone number—for instance, if a fire hydrant is broken, or a road sign has fallen over. Cloud technologies can help government agencies develop a seamless solution, accessible via smartphones, through which citizens can take a picture of an issue like damaged public property on a mobile website and then route the appropriate identifying information and location data to the right government officials that can solve the issue.

In this blog post, learn how to set up a solution with AWS deep learning services that creates a fluid experience for reporting and addressing these issues.

How to set up a deep learning-powered solution to log and resolve citizen reports: Solution overview

This solution is an example of a government service in which citizens can submit images of damaged government or public property through a mobile website. When citizens upload images to the website, the solution uses machine learning (ML) to identify the type of damaged property, and then routes that information along with location data to the appropriate government agency, where a government official can triage the issue for remediation. As such, this solution has two workflows, one for the citizen, and one for the government worker:

  • Citizen workflow: Citizens load a mobile web page and submit a report of damaged government property by taking a picture of the damaged area.
  • Government workflow: Government workers load a desktop web page and view a map of reported damage.

Architecture overview of the citizen workflow

Figure 1. The architecture overview of AWS services involved in the citizen workflow, described in detail in the following section.

Figure 1. The architecture overview of AWS services involved in the citizen workflow, described in detail in the following section.

1. Citizen loads a mobile web page from Amazon CloudFront and Amazon Simple Storage Service (Amazon S3).

2. After page load, the mobile browser silently retrieves temporary credentials from Amazon Cognito.

3. The interface prompts the user to take a picture, which is uploaded to Amazon S3 using the credentials from Amazon Cognito.

4. The upload triggers an Amazon S3 Event Notification to run Python code in AWS Lambda. The Lambda function extracts location data from the image metadata and requests Amazon Rekognition to identify labels in the image.

5. The Lambda function retrieves pre-built report options from the Amazon DynamoDB table and combines this with data from the previous step to provide suggested reports, which are stored in the report table.

6. The citizen’s mobile browser polls for suggestions and available report options via Amazon API Gateway and AWS Lambda. The citizen is prompted to make a selection, either one of the suggested reports or any other report.

7. The citizen makes a selection and submits the report, which includes the GPS coordinates of the user’s mobile browser. This action updates the record in the report table.

Architecture overview  of the government workflow

Figure 2. The architecture overview of AWS services involved in the government workflow, described in detail in the following section.

Figure 2. The architecture overview of AWS services involved in the government workflow, described in detail in the following section.

1. A government worker loads a desktop web page from Amazon CloudFront and Amazon S3.

2. The browser retrieves temporary credentials from Amazon Cognito.

3. The browser renders an interactive map via the Amazon Location Service using the credentials from the previous step.

4. The browser retrieves all currently open reports via Amazon API Gateway, AWS Lambda, and Amazon DynamoDB.

5. The worker browses reports by clicking on pins on the map, which loads the image from Amazon S3 and displays the metadata from the report table.

6. The worker then resolves the report using a button on the interface that updates the report table, marking the record as resolved.

Walkthrough for how to set up a deep learning-powered solution to log and resolve citizen reports

The solution is packaged as an automated deployment via the AWS Serverless Application Model (AWS SAM) command line interface (CLI).

Prerequisites

For this walkthrough, you must have the following prerequisites:

  • An AWS account
  • AWS SAM CLI installed and set up with credentials
  • Python version 3.9 installed and in your PATH variable
  • A smartphone with location services enabled for testing
  • Select one of the supported AWS Regions:
    • US East (Ohio)
    • US East (N. Virginia)
    • US West (Oregon)
    • Europe (Frankfurt)
    • Europe (Ireland)
    • Europe (Stockholm)
    • Asia Pacific (Singapore)
    • Asia Pacific (Sydney)
    • Asia Pacific (Tokyo)

Deployment

The AWS SAM CLI is an open-source command line tool used to locally build, test, debug, and deploy serverless applications defined with AWS SAM templates.

Download the source code and extract locally: AWS Samples – AWS Serverless Deep Learning Suggestions

Build and deploy the stack

1. In a terminal, navigate to the ./sam/ directory

2. Run the following command to build and package the project for deployment:
sam build

3. Deploy the SAM template to your account. The wizard will guide you through the process of deploying the SAM CloudFormation stack. Details on this process are found in the sam build documentation.

a. Run the following command:

sam deploy --guided --capabilities CAPABILITY_NAMED_IAM

b. Select the supported AWS Region you chose in the prerequisites section.

c. The default parameters are suggested.

d. Choose “Y” for all Yes or No items.

4. Wait for the deployment to complete. This process takes approximately five minutes.

5. Once the build completes, note the entries in the Outputs section. These are the URLs for the two pages of the Deep Learning Suggestion application. Note these for the next sections:

a. SubmissionURL – Citizen UI for submitting reports of damage

b. MapViewURL – Government UI for viewing the submitted damage reports on a map

This concludes the deployment of the Deep Learning Suggestion application. AWS SAM CLI uses AWS CloudFormation to orchestrate the deployment of both the backend API, image processing infrastructure, and the front-end static website. The entire application is deployed.

Optionally, you can view the deployed resources by logging into the AWS Management Console, navigating to the CloudFormation service, selecting the relevant CloudFormation stack, and choosing the Resources tab.

Test the solution: Create a report

Figure 3. The mobile website interface showing the three steps of the submission process citizens use to report issues, described in detail in the following section.

Figure 3. The mobile website interface showing the three steps of the submission process citizens use to report issues, described in detail in the following section.

In this testing phase, you will act as a citizen, reporting damaged government property. Decide whether you will take a live picture of an object in the world or find an image online. Suggested objects include fire hydrants, traffic signs, traffic signals, and postal mailboxes. These objects align with the sample reports included with the solution, but can be expanded upon later.

Finding and visiting an actual object in the world in Step 4 below will yield the most accurate results. If you decide to find and use an image online, it is important you make the image capture appear as close to real life as possible. The online picture must encompass the entire camera image without borders, cursors, or other web content.

1. On your smartphone, make sure you have location services enabled.

2. Load the citizen UI on your smartphone by navigating to the Amazon CloudFront static website. This URL is the SubmissionURL value you noted earlier, found in the Outputs section of the SAM deploy command. You will see a page titled Maintenance Reporting, which prompts for an image upload.

3. Tap Choose File halfway down on the left side. Take a picture of the object and confirm the preview.

4. Tap Upload to upload and process the image. Wait to be prompted before continuing to the next step.

5. After the image is processed, suggested and most relevant report types appear at the bottom under Looks like you’re reporting.

a. If you do not see the Looks like you’re reporting section, refresh the page and try again. This means Amazon Rekognition was not able to recognize a reportable item. Confirm you’re taking a picture of a suggested object from the first paragraph of this section. If you’re taking a picture of an image on a computer screen, you must capture it without borders, cursors, or other content you would not see in the real world.

6. Select the most appropriate report type from the list or Other if none exists and you would like to continue.

7. Tap Submit Report to complete the submission. If prompted for the website to access the location of your mobile device, allow the access.

8. Confirm that the Submission Info is complete, showing Received! with a timestamp and the submission identified.

The uploaded image now exists in the Amazon S3 bucket for image storage, and submission metadata, including image filename, report type, timestamps, and GPS coordinates, as an entry in the Amazon DynamoDB report table. Submissions do not store personally identifiable information that make it possible to link a submission back to an individual.

Test the solution: View all reports

Figure 4. The desktop web interface government officials use to triage issues, described in detail in the following section.

Figure 4. The desktop web interface government officials use to triage issues, described in detail in the following section.

In this section, you will act as a government employee, viewing the citizen-submitted reports of damaged government property.

1. Load the government UI in your desktop browser by navigating to the Amazon CloudFront static website. This URL is the ViewingURL value found in the Outputs section of the SAM deploy command. Find a page with a satellite map view and an Instruction

2. Wait a moment for the pin to appear on the map. If no pin appears, move the map window to show your current location. Navigate and zoom until you find the submission from the previous Create a Report section above.

a. Note: The map pins prefer GPS coordinates from the image metadata over the citizen’s mobile browser location. GPS coordinates at the time the picture was taken are preferred in case the citizen is in a moving vehicle or takes a picture and submits it later.

3. Place your cursor over the pin. You will see a tooltip indicating the report type, submission identifier, and a thumbnail of the submitted image.

4. Click the pin to view the details of the submission:

a. The header shows the report type(s).

b. The Resolve button marks the submission as resolved in the database and dismisses it from view.

c. Location (Image) shows the GPS coordinates from the image metadata (EXIF).

d. Location (Mobile) shows the GPS coordinates from the mobile browser location.

e. Identified Labels (ML) shows the labels returned from the Amazon Rekognition service, with confidence level in parentheses.

f. Relevant Reports (ML) shows the most relevant reports as decided by the ProcessUpload Lambda function based on the number of Amazon Rekognition labels that match the prebuilt reports in the report table. These are the same entries the citizen sees in the Looks like you’re reporting section of their submission interface.

5. When ready, click Resolve to dismiss the report.

The uploaded image remains in the Amazon S3 Bucket for image storage, and submission metadata, including image filename, report type, timestamps, and GPS coordinates still exist as an entry in the Amazon DynamoDB report table, but is marked as resolved.

This solution, as deployed, is a publicly-accessible API protected by an API key generated during deployment and stored as part of the website source code. Utilizing the Usage Plan feature of the API Gateway, usage is throttled to protect from abuse. However, to support reliable service, these settings and protections should be reviewed against expected usage patterns before releasing for production use. As an additional step to deter abuse, any invalid data (nonimage, too large, etc.) is logged, then immediately discarded after upload.

Additional considerations in security

As a best practice, the solution enables the following access logging features:

While the solution is designed with security best practices, certain items have been omitted as they are highly customer specific. Consider the following opportunities for additional security controls:

Clean up

This is an entirely serverless solution. Therefore, costs are directly related to usage — only a few dollars to receive, process, store, and view 1,000 submissions for one month. The persistent cost to maintaining this application when not in use is in the submission data stored in Amazon DynamoDB and the images and website content in the Amazon S3 buckets.

To avoid incurring future charges, delete the resources.

To deprovision all resources:

1. In the AWS Management Console, navigate to AWS CloudFormation and select the stack specified in the deployment section of this post.

2. Select Delete near the top and click in the popup confirmation to begin deleting the stack. This process will take approximately five minutes.

a. WARNING: This will delete the contents of the Amazon S3 bucket and Amazon DynamoDB table, including all submitted images and metadata.

3. Delete the logging bucket that stores access logs for the application. These are preserved by default for audit purposes. This is prefixed with dl-suggest-blog-logging-.

Conclusion

In this post, I presented an interactive mobile website with real-time feedback combined with location data capture to create a seamless experience for citizens to report issues with public and government property in their area. Governments can use and build on this solution to improve the citizen experience and deliver on federal priorities with a high-performant yet cost-effective solution. This solution also provides highly accurate location reporting and damage context with images to increase efficiency in triaging reports, saving governments time and money.

Each component of this application includes a Free Tier. Use the AWS Free Tier to learn and experiment with new technologies.

Learn more about how governments use AWS to innovate for their constituents, design engaging constituent experiences, and more at the AWS Cloud for State and Local Governments hub.

More related resources from AWS:

Subscribe to the AWS Public Sector Blog newsletter to get the latest in AWS tools, solutions, and innovations from the public sector delivered to your inbox, or contact us.

Please take a few minutes to share insights regarding your experience with the AWS Public Sector Blog in this survey, and we’ll use feedback from the survey to create more content aligned with the preferences of our readers.