Harvest.ai: Changing How Companies Share Content Securely

Guest post by Paul Radulovic, Director of Engineering and owner/architect of harvest.ai’s AWS environment

At harvest.ai, we are changing how companies prevent theft of their information from cyber-attacks. We do this by creating a smarter security infrastructure that utilizes artificial intelligence (AI) to learn to identify critical assets and how they are being accessed.

In this post, I’ll share the story behind harvest.ai, discuss how we built a secure, trusted production environment, and provide some lessons learned along the way.

Security Is in Our DNA

Our company’s DNA is rich in security. Our team leaders include former NSA, FBI, DoD, FireEye, and Websense experts who have used their collective intelligence and experience to build a product that has a unique ability to prevent leaks of an organization’s intellectual property, hard work, and information.

The modern cloud environment provides global access to shared content at speeds that are faster than ever. We provide a safe platform for content collaboration and content sharing without the fear of a data breach.

How People Share Content and How We Can Protect Them

Right from the start, our focus was to partner with Fortune 1000 organizations that were migrating to cloud-based platforms for collaboration, but needed enhanced security to protect their critical assets. When widespread data breaches at Sony and Anthem became public, we saw other security firms focusing on malware, border security, and traditional detection methods. Although those methods can be useful, we’ve always focused first on what is most important to companies: the content itself.

Sharing data is an integral part of today’s business world. By assigning risk to content and monitoring the content that is leaving your network, you can prevent significant loss to your company. At harvest.ai, we made the decision to couple data-loss prevention practices and user-behavior analysis as a way to gain the following key insights:

• Understand the value of content within a network
• Learn how users interface with that content
• Detect, alert, and mitigate when a user’s behavior indicates either malicious or unintended data loss

Call it our light-bulb moment. By applying machine-based learning and understanding natural language processing, we created MACIE, an analytic platform that protects user accounts from compromise and helps to prevent the theft of data and intellectual property.

Our Launch and Our First Fortune 100 Customer

We were met with two major opportunities concurrently — officially launch harvest.ai and deploy MACIE for our first paying customer. In December of last year, we successfully managed both! Our first customer is a multinational Fortune 100 company that wanted to leverage our core product features — such as data privacy, support for hundreds of different data types in multiple languages, and a global customer success model — as soon as possible. With these urgent priorities, there was no room for slow development lead times or complicated processes. Losing two days while waiting for hardware to be delivered wasn’t an option.

Providing a Secure, Trusted Environment

In the early planning stages, Alex Watson (our founder and CEO) and I architected the back-end environment to run on a set of very powerful, bare-metal servers inside a local data center. As we continued to clarify the scale, the number of documents that we needed to manage, and the types of data that we needed to handle, we knew that we had to identify future challenges before they became roadblocks. For that reason, we designed our infrastructure according to the following considerations:

Scalability:

Processing 10+ million documents requires a different approach than processing 10,000. The initial approach of using a small number of powerful machines would have taken us weeks, even when implementing significant multiprocessing. By distributing the work across a large number of less powerful virtual machines (specifically, the t2 and m3 families of Amazon EC2 instances), we were able to spin up orders of magnitude much faster (in hours, not days) and at a reasonable cost.

AWS allows us to easily scale vertically by upgrading the virtual hardware within instance families according to the needs of the system, and horizontally by adding additional cost-effective instances (t2, m3, and m4 families) to distribute the processing load.

Security:

Protecting our customers’ data is our first priority. AWS enables us to provide our customers with the security their data requires and deserves, without requiring the development and maintenance of our own non-standard security practices. We accomplish this by using separate VPCs for each customer, individual encryption keys for all data volumes, strict security group rules, and limited exposure to the public Internet.

Because customer data, security needs, and data types vary (HIPAA, other regulatory documents, credit card numbers, PII, etc.), providing certified operational environments for our hardware independently is cost prohibitive. AWS takes care of that by providing certifications on the hardware layer and by continuing to improve, modernize, and upgrade their hardware environment without increased costs.

Cost:

Cost is always an important consideration for a startup, but lower costs also allow us to pass the savings on to the customer. Instead of spending engineering cycles and startup funds on hardware selection and testing, AWS allows us to pay only for the resources that are necessary for the size of each customer. This is especially important as the demand for resources change through the lifetime of the product.

When the data pipeline is architected in a fail-safe manner, you also can use EC2 Spot Instances to add cost-effective processing power only when needed.

Redundancy and Repeatability:

Because our product works 24/7 behind the scenes, redundancy is critical. By utilizing launch configurations, auto-scaling groups, and snapshots, we can spend less time worrying about uptime and more time developing the product. Our instances are designed so that they automatically rebuild themselves if they fail and are recreated by auto-scaling groups.

By utilizing Ansible playbooks stored in Git repositories, we automated the configuration and controlled our environments, allowing us to launch our service for new customers in a matter of minutes instead of hours or days. Additionally, by removing the human element from the configuration progress, we guaranteed that security measures are in place every time.

Takeaways

• With a team of around 20 people, hours count. Using AWS has increased our productivity tenfold because we never worry about hardware maintenance, purchasing, 24×7 uptime, or compliance.
• As AWS changes and adds services, it encourages our teams to evolve our product and service offerings on a global scale.
• When developing new techniques to increase scalability, we bypass speed bumps by utilizing existing AWS services. Take the time early on to learn about all the services available, as it allows you to consider more implementation avenues during development down the road. For example, Amazon SQS allowed us to implement real-time queues in 10 minutes with 20 lines of code. This saved us hours, if not days, of installing, configuring, debugging, and maintaining our own solution. It also allows our analytics to stop threats in near real-time, which is a significant advantage to protecting against data breaches.
• Think long term when developing your infrastructure’s backend. Shortcuts you take today will hurt you in the long run. Design for 200,000 users, but be able to scale down the resources and cost for 1,000 users.
• Document. Document. Document. Plan for business continuity, new employee integration, and for that 2:00 AM debugging session when the energy drink starts to wear off.

Onward and Upward

Today we are supporting organizations that are passionate about utilizing cloud services to provide cost efficient, flexible work environments for their employees to create, collaborate, and share. We view data protection as a necessary layer within the workspace, but we will never interrupt the flow of business and teamwork.

As we continue to scale and grow, we see AWS products like CloudWatch, CloudTrail, Elastic File System, and S3 as important tools to keep our development process smooth. With the scalability and availability to evolve our product, and leveraging AWS for best-in-breed hardware, support, and technology, harvest.ai will continue to help companies create, collaborate, and share with confidence that their data is secure.