Forage Improves Data Analytics Speeds and Gains Real-Time Metrics with AWS

As offices worldwide increasingly embrace flexible work-from-home arrangements, the nature of internships is undergoing a profound shift, with many companies moving their programs online.

Forage, a startup technology company that connects students with Fortune 500 companies and other organizations offering virtual internships, experienced rapid growth in 2020 as this trend accelerated. Enrollments on its platform surged from 80,000 interns in December 2019 to more than 1 million in December 2020. Over the same period, companies providing internships on Forage tripled, from 20 to more than 70. 

When Forage first launched in 2017, then known as InsideSherpa, it was set up on MongoDB Atlas, a managed hosting provider for MongoDB which used servers and databases that ran on Amazon Web Services (AWS). As the business grew, Forage has expanded its use of AWS to create more value for its customers by enabling them to execute faster, high-quality data queries, such as analyses of student retention rates and comparisons of survey responses across countries.

“In the past, some queries would take 30−40 seconds. Now, it’s gotten down to 1 second or half a second. For us, the faster analytics speeds with AWS represent a huge improvement on product quality and usability for our clients,” says Pasha Rayan, cofounder and chief technology officer at Forage.

Forage’s customers can access a dashboard that allows them to sort and analyze information about interns, including their backgrounds and performance in the program. With faster query speeds on AWS, Forage structures its data so its clients can quickly identify reporting and analytics for diversity. This enables its clients to reach underrepresented groups, especially in the technology industry. Recruiters could previously visit only 20–30 university campuses a year in the US, but with Forage, they can reach thousands of international campuses via a single online platform.

Forage worked with AWS to build a data analytics solution that takes advantage of AWS Database Migration Service (AWS DMS) to replicate data from a legacy database into a serverless data lake on AWS Lambda. And Forage used AWS Glue to transform, process, and publish the data to Amazon Relational Database Service (Amazon RDS) for the dashboards that its applications display.

This analytics solution provides real-time quantitative metrics for use on client dashboards, including student completion rates for programs, user demographic and geographic breakdowns, usage patterns, and more. It has also been useful for Forage’s internal data science, with ad hoc data queries in Postgres being much faster and easier to implement than MongoDB equivalents.

From its roots in Australia and the US, Forage has quickly expanded internationally to the UK, Europe, Singapore, India, and South Africa, with more opportunities on the horizon. This growth into new geographic regions has driven the company’s need for infrastructure that is truly global. Data sovereignty and data privacy are vital to Forage’s clients. Forage is using multiple AWS Infrastructure Regions to support its global footprint and store client data in local servers across the UK, US, Europe, and Asia. “The security across multiple AWS Regions means we can grow into different countries and areas that we wouldn’t have been able to without AWS,” says Rayan.

Additionally, some of Forage’s customers require security and compliance certifications—including General Data Protection Regulation (GDPR), California Consumer Privacy Act (CCPA), and SOC 2—all of which AWS has been able to support. “We often get asked if our infrastructure meets the highest standards. With AWS, we have documentation that allows us to say, ‘Yes, we have the highest standards,’” Rayan says.

With Forage’s technology solution running on AWS, the company scaled with relative ease into a period of rapid growth in 2020. AWS also enabled Forage the flexibility to continue to innovate at a fast pace, including creating customizable graphs and analytics for corporate partners and providing better feedback by surfacing the right users faster. “Because AWS is both cost-driven and rich in its tool set, we can experiment a lot faster than most other infrastructure providers. We’re quite a high experimentation team. For us, having the freedom to play around with our infrastructure, and set it up to do what we need to do, has been invaluable,” Rayan says.

As its business has grown, so too has Forage’s IT team. When the company was formed, it had just two IT specialists. By the end of 2020, the company had 30 employees, including more than a dozen IT specialists. According to Rayan, AWS Support has been “incredibly important” in helping facilitate growth alongside new additions to Forage’s internal tech team. “We wouldn’t have been able to roll out key features, such as our enterprise dashboard and our reporting and emailing features, with our previous data infrastructure. With our data analytics solution running on the AWS Cloud, plus 24/7 support from the AWS team, we’ve been able to release key features, get a leg up, and move much faster. It would’ve been very hard for us to build a company if not for AWS.”