AWS Case Study: Recorded Future

While the applications for this technology are quite extensive, Recorded Future currently focuses on three areas: 1) the quantitative finance industry, 2) the commercial competitive intelligence industry, and 3) the national security and intelligence area. Within these markets, the company’s technology assists customers in identifying and understanding historical events based on the patterns that emerge. By organizing events around dates from the text, they are able to empower customer to predict what might happen in the future.

The Business Challenges

Recorded Future empowers analysts with forecasting tools via two channels. First, it allows analysts to ask temporal questions like “Who’s traveling to Venezuela next week?” or “What events are supposed take place over the next 10 days in Iran?” by quickly aggregating all future-oriented observations from the source materials and building a timeline of weighted opinions about the likely timing of future events. Second, Recorded Future provides curated Web intelligence signals for predictive mathematical models. A massive historical archive with 2+ years of more than 5 billion structured time-tagged facts from the web, along with past and real-time quantifiable signals are an invaluable assets for predictive statistical models.

The Recorded Future Premium service, an analytic environment hosted in the Amazon Elastic Compute Cloud (Amazon EC2) cloud at RecordedFuture.com, mines unstructured text from hundreds of thousands of sources on the open Web – everything from government filings, to blogs, to Twitter feeds – and organizes and aggregates the data to provide a picture of what the world knows about the future. With no downloads or plug-ins needed, users can interact with the data immediately via a true Web 2.0 interface or a web service API. The data and time exploration tools feature momentum and sentiment measurements across multiple languages. These calculations contextualize buzz on the web and measure how positive and negative that discussion is.

Why Amazon Web Services

A small team with diverse needs, Recorded Future contemplated the right solution for their company. Jason Hines, Federal Director at Recorded Future, explains the decision to go with AWS: “AWS is a great environment for startups looking to develop and test new technology without having to front hefty investment costs in hardware.” Recorded Future's index contains 5 billion time-tagged facts, in the past and in the future, all algorithmically extracted from the Web. Hines explains the data intensity: "We harvest and process hundreds of thousands of documents every hour, in real-time from tens of thousands of sources so our index is always growing. Within about 10 or 15 minutes of news breaking, it’s harvested, analyzed, and available in our system."

Recorded Future uses several solutions from AWS. Amazon Elastic Block Store (Amazon EBS) allows the company to store and move data between instances. Recorded Future also uses Amazon Route 53 to manage their DNS, elastic load balancing for load balancing of web servers, and Amazon Simple Storage Service (Amazon S3) to store backups.

Today, Recorded Future is running on 40-70 instances depending on need and load, 1-1.5TB of RAM, and about 100TB of storage. With AWS, the company can adjust this as needed, based on load. A diagram of Recorded Future's architecture is shown below:

The Business Benefits

Looking ahead, Recorded Future – a team made up mostly of software engineers - doesn’t see the need to procure or deploy their own hardware. With AWS, they find their needs are met. When asked why they chose AWS, Ulf Mansson, Recorded Future's Chief of Operations, concludes: “We simply haven’t found any other provider that can meet all of our diverse computing needs.”

They credit AWS for managing their whole platform which they engineered specifically for the cloud. Knowing that their small team didn’t have the resources to take on such a large challenge within a typical IT infrastructure, they turned to AWS and Amazon EC2. “AWS is our architecture," says Mansson. "Literally every component of the Recorded Future machinery utilizes AWS. The only thing we don’t use AWS for is our email and CRM.”

Now the Recorded Future engineers can focus on what they do best – applying their energy and efforts on organizing the Web to predict the future.

To learn more, visit recordedfuture.com .

Added March 15, 2012