Combating illicit activity by tracking flight data via the cloud

Many organizations including the intelligence community, security organizations, law enforcement, regulatory bodies, news organizations, and non-governmental organizations work together to disrupt transnational crime networks. Their missions include combating illicit trade; disrupting human, animal, and narcotics trafficking; detecting money laundering; and exposing political corruption. This community needs rapid analysis of large, diverse streams of information about air transportation networks, because air transportation is the fastest way to conduct illicit trade internationally. The nonprofit Center for Advanced Defense Studies (C4ADS) built the Icarus Flights application to meet this need.

The Icarus Flights application is built on Amazon Web Services (AWS). By using managed cloud services, C4ADS spends less time and energy managing infrastructure, which frees them to focus on building innovative analytics and alerting services that their user community needs.

Ingesting and analyzing flight paths

The Icarus Flights application ingests and analyzes flight paths, provides queries and alerts of new flights or aircraft matches, and lets users search current and historical airframe registration data aggregated from public sources. The application delivers information about suspicious activity to expert analysts, who can then provide law enforcement with accurate, precise, timely, and actionable information about possible criminal activity.

Typically, Icarus Flights processes about 2 billion aircraft position pings per month, and this number is steadily rising as ADS-B Exchange’s network of receivers expands. More than 5,000 ground-based receivers receive information from upwards of 200,000 unique aircraft transponders worldwide.

Icarus Flights enables investigators to monitor conflict zones, discover surveillance flights, and query aircraft ownership data. The following sections describe the Icarus Flights architecture in two parts:

• The Data Processing Architecture describes how Icarus Flights ingests a variety of data sources in real-time and batch modes, blends the data, and generates the data that are used in the Icarus Flights applications.
• The Application Architecture describes how Icarus Flights provides end-user interfaces and Application Program Interfaces (APIs) for search queries and analysis.

Icarus Flights data processing architecture

Figure 1: Icarus flights data processing architecture

Icarus Flights ingests live data streams from aircraft transponder and flight plans, and incorporates scheduled updates of airline routes and aircraft registrations. C4ADS is developing Apache Kafka (Kafka) pipelines using Amazon Managed Streaming for Apache Kafka (Amazon MSK) to feed data-to-data processors such as event inference and image analysis. Data processors in turn feed processed data to Kafka, which writes the data to two destinations: a PostgreSQL database for queries and analysis, and Amazon Simple Storage Service (Amazon S3) for durable, long-term storage.

C4ADS uses Amazon Relational Database Service (Amazon RDS) PostgreSQL to support data processing and application services. Jake Glass, product and engineering manager of Icarus Flights, says, “Amazon RDS is simple to deploy and provides impeccable reliability and failover capability.”

Glass adds, “We’re using Amazon Elastic Kubernetes Service (Amazon EKS) to orchestrate our live streaming pipeline for ingesting aircraft position data from ADS-B Exchange and process it before it’s made available to search within the app.” The pipeline includes a number of application containers performing modular, stateless tasks on the incoming data, so it needs to be highly available and straightforward to maintain for the various workers. The C4ADS team was able to stand up their services quickly by learning and using the simplest Kubernetes features that enabled them to meet their reliability, performance, scalability, and security requirements. He writes: “Our simple deployment only scratches the surface of Amazon EKS and Kubernetes capabilities, but the `eksctl` command line interface (CLI) has been a pleasure to work with thus far.”

Icarus Flights application architecture

Figure 2: Icarus Flights Application Architecture

The Icarus Flights application supports both a web application client and a Developer API Client through a common REST API. C4ADS implemented the API using Golang and Echo on AWS Elastic Beanstalk to deploy application containers in a highly available environment without manually provisioning and tuning load balancers.

Amazon ElasticCache Redis in-memory cache to support complex queries at a large scale with responsive near real-time performance. The Redis cache stores the application state, which allows the app to scale up smoothly while maintaining performance. C4ADS uses React/Redux (Typescript) for their web client served via Amazon S3 as a content origin and Amazon CloudFront as their content delivery network.

Icarus Flights ingests a large, fast-moving, and diverse data set using a combination of streaming and batch processing. It provides rich search capability for data exploration—taking a large volume of data and quickly identifying events of interest for experts to analyze. AWS managed services enabled them to develop this application quickly and operate it efficiently and securely. The use of managed services frees them to build innovative solutions to meet the needs of their user community.

C4ADS is making the beta version of Icarus Flights available to investigative journalists and NGOs, and sends out free ADS-B receivers globally to improve aircraft coverage. To register your interest in access to Icarus Flights or to host a free receiver, visit https://icarus.flights.

Read more analytics and nonprofit stories on the AWS Public Sector Blog.