Lensa Builds a Fast, Smart, and Scalable Job Search Platform Using AWS

Lensa was founded in 2015 with the ambition to be more than just another job search board. With more than 20 million job seekers using its database and billions of job openings posted each day, the company deals with vast quantities of information and its usage requirements vary hour to hour. For instance, it sees traffic spikes during the morning, soon after it sends out job alerts to customers. Along with processing job openings, it also analyzes relational data on customer interactions, changes, and logs.

Lensa’s platform compares databases of candidates and job openings and, when it finds matches, sends out batches of emails and SMS alerts—more than 6 million alerts in total—every day, which requires significant compute power. So Lensa needed an elastic infrastructure that could easily scale up and then scale down after the alerts are sent.

Lensa runs ML models on the data to extract high-quality job matches. To accomplish this, the company required a cost-effective way to handle both its large MySQL database and its growing data lake.

The Lensa technical team also wanted to give users a good experience, which was required to succeed in a highly competitive marketplace. This meant optimizing website and database performance so search results could be returned without delay. In addition, it wanted an agile environment for developers so they could quickly create and deploy new features.

Lensa uses a range of services to provide the best possible job matching service while minimizing its infrastructure spending. For its jobs databases, Lensa uses Amazon DynamoDB, a fast, flexible NoSQL database service for single-digit-millisecond performance. For complicated queries, it uses Amazon Redshift, the fastest, easiest, and most widely used cloud data warehouse.

Pál Megyeri, chief technology officer (CTO) at Lensa, explains that the two databases store information in different ways. Amazon DynamoDB is a high-performance, key-value NoSQL database. Amazon Redshift also provides high performance, but it’s better at compressing information and it’s able to perform much faster when dealing with extremely complex queries because it uses columnar storage. After migrating data from MySQL to Amazon Redshift, for instance, Lensa’s storage requirements fell from 490 GB to 180 GB. This is because Redshift stores data and its indexes more effectively than MySQL, so Lensa can separate out infrequently used data and store it as ‘cold data’, while still maintaining access to it as needed.

The company used Amazon Redshift to improve complex query times by a factor of 10 compared to the same query using its MySQL database. “A complicated query in MySQL used to take about 8 minutes—in Amazon Redshift, we get the same results in 22 seconds,” says Megyeri.

Most of the company’s 1 PB of data is stored in Amazon Simple Storage Service (Amazon S3), which provides object storage built to retrieve any amount of data from anywhere. It also makes constant use of Amazon Elastic Compute Cloud (Amazon EC2) instances, which provide secure and resizable compute capacity for virtually any workload. “We can now explore and analyze our huge data lake using different AWS solutions, depending on the depth and scale of the project,” says Megyeri. “Using AWS, we have virtually unlimited resources on tap, without any upfront investment. This is a significant business benefit for us.”

Lensa uses Amazon Athena, an interactive query service, to analyze, clean, and aggregate the data. It then uses AWS Glue to discover, prepare, and integrate data. When its developers work on extremely large projects that can reach tens of PBs, they use Amazon EMR, which can run big data workloads easily.

For a cost-effective way of handling background services to process one-off events and some specific messaging and feedback services, Lensa uses AWS Lambda, which runs code without Lensa developers having to think about servers or clusters. “Using AWS Lambda, we don’t need hardware,” says Megyeri. “If the project evolves into something more demanding, we can fire up an Amazon EC2 instance. This is the flexibility we have using AWS—it makes life easy.”

Lensa has years of experience using AWS for various projects, and Megyeri says the company’s deep knowledge of AWS gives it a competitive edge. Lensa has about 120 employees, about 84 of whom are in technical positions and receive regular AWS training. “We’ve used AWS for many years and are very happy,” he says. “We train our developers to know all the services inside out because then they can use the most appropriate tool to innovate and solve any problem.”

Using AWS, Lensa’s developers are encouraged to innovate and explore new ideas for features and functionality. This is possible because they don’t have to spend time on infrastructure maintenance. In fact, they have the resources to create environments and test features at their convenience in a continuous integration and delivery (CI/CD) environment, which has increased their productivity. “We send out 4,000 releases a year—our platform is continuously changing, minute by minute, without any outages,” says Megyeri. “From a developer’s perspective, it’s a dream place to work.”

Lensa’s developers are now free to experiment and have the resources to do so on demand. “Using AWS, if we want to create a big ML model, we can do it right away,” says Megyeri. “We just choose lots of GPUs and 1 TB of memory, and play. It’s really cool.”

Lensa has grown quickly since its 2018 launch—at that time, it had 7 million job seekers in its database and that figure has now almost tripled to 20 million. In the past 12 months, the company’s data lake has increased in size from 470 TB to 970 TB. This data is being used to continuously improve Lensa’s job-matching algorithms—so as the amount of data grows, its product produces better results.

Megyeri says the depth of the AWS service portfolio means his developers are able to pick a service that provides an instant solution. He had no hesitation about using AWS. “We use the most appropriate AWS service to innovate, grow, and succeed,” he says. “If a problem arises, there’s always another service that provides the solution.”