Lowering the Cost of Curiosity

In his post on the AWS Public Sector blog, John Brady, the CISO of the Financial Industry Regulatory Authority (FINRA), talks about building a data lake in the cloud to reduce the cost of curiosity. The concept is brilliant and consistent with the way I like to think about agility and innovation: that reducing the cost of experimentation with the cloud and DevOps gives enterprises the key to encouraging innovation. The cost of curiosity is essentially this same idea translated into the world of data.

FINRA regulates one critical part of the securities industry — brokerage firms doing business with the public in the United States. Its mission is to protect investors and maintain market integrity by looking for cases of fraud, abuse, and insider trading. Every day, FINRA receives and processes 6 terabytes of data, representing an average of 37 billion new records, although, on peak days, it can receive over 75 billion transactions. FINRA analysts run analytics on this data and also run interactive queries on what is often more than 600 terabytes of data. They can also query years of historical data — petabyte scale — in minutes or hours, rather than weeks or months.

Because they are looking for suspicious patterns — where “suspicious” is not always well-defined in advance — it’s important that FINRA analysts be able to be … well, curious. And the speed and low cost that FINRA is able to achieve with its data lake in the cloud make this curiosity actionable. That, in a sense, is data agility — a state where companies can explore possibilities without defining all their requirements in advance; where they can get fast feedback on results and use that feedback to modify their approach; and where they can adapt rapidly to change and work to confirm or refute hypotheses they may generate. Lowering the cost of curiosity is crucial to achieving this state. It’s a necessary enabler that permits organizations to use data in an Agile way.

Of course, security is an important consideration when allowing analysts the freedom to be curious. For most companies, keeping personal information private is a key consideration in analytics; for FINRA, the integrity of its data and compliance with financial industry regulations are especially critical. That’s why — consistent with good Agile development practices — FINRA brought security engineering into its process at the very beginning. Indeed, FINRA’s DevOps process gives it consistency in deployments to ensure fully compliant environments, and AWS tools help it oversee systems in production by monitoring for continued security. According to Brady:

In the last four years as we transitioned to the cloud, I have come to realize that as a relatively small organization, we can be far more secure in the cloud and achieve a higher level of assurance at a much lower cost, in terms of effort and dollars invested. We determined that security in AWS is superior to our on-premises data center across several dimensions, including patching, encryption, auditing and logging, entitlements, and compliance.

I agree with Brady. In my earlier role as the CIO of US Citizenship and Immigration Services (USCIS), I often made the case that, even as a relatively large organization, we were more secure in the cloud than in the Department of Homeland Security (DHS) data centers, especially when comparing across similar dimensions to those Brady mentions.

The specifics of FINRA’s solution, as Brady explains, included the involvement of security, audit, and compliance groups early in the process; micro-segmenting servers with security groups; administering keys with AWS Key Management Service (KMS); using the controls it could inherit from Amazon EC2 and AWS Lambda; and setting up a DevOps automation process to ensure testing and compliance during its development and deployment processes.

With these security controls, FINRA has been able to reduce the cost, as well as the risk, of curiosity. For a look at what it means to make big data Agile, please take a look at Part One and Part Two of Brady’s posts. I think you’ll agree that this represents a new dimension in enterprise agility.

–Mark
@schwartz_cio
A Seat at the Table: IT Leadership in the Age of Agility
The Art of Business Value