Given the continued growth of PAI, managing the volume and speed of data is a major challenge for intelligence analysts and decision-makers. BAE Systems designed OSINT DSP to ingest, process, and extract meaningful knowledge from the firehose of variegated PAI data sources. The OSINT DSP delivers a comprehensive interactive development environment (IDE) with data science and machine learning tools for better managing OSINT and PAI. It provides the ability to curate and visualize high velocity OSINT data across a variety of sources including news websites, social media, and deep and dark web.
The OSINT DSP utilizes a range of scientific computing libraries such as Pandas, Matplotlib, Scipy, Sci-kit Learn, and others. It integrates a Jupyter-based IDE, including Anaconda, and facilitates collaboration across users. Jupyter offers a robust report writing capability and includes inline visualization. The OSINT DSP also includes an example notebook that illustrates practical implementation of Apache Drill data connections, as well as a complementary OSINT notebook to perform automated text summarization. The Apache Drill search engine, packaged within the offering, provides example connectors that can be used to interface with a variety of storage technologies (e.g., MySQL and Elasticsearch). Apache Drill also has connectors that facilitate access to proprietary databases commonly used within the Intelligence Community (IC).