We regularly run Amazon Elastic MapReduce jobs that may require hundreds or even thousands of servers at once, depending on the client. Processing this big data is extremely easy and cost-effective for us compared to doing it on a physical infrastructure.
Ismail Arslan Product Manager

It’s tough to stay ahead of the competition in the online retail game. Increasingly, retailers find themselves having to drive down costs in order to make their products and services attractive to customers. Turkey-based startup Convertale set up its real-time recommendation engine in an effort to help its e-commerce clients deliver added value to their customers. Through analysis of user behavior in online stores, the engine provides suggestions of related products that shoppers are likely to be interested in. This helps retailers personalize the online experience and maximize revenue in a highly competitive market. Founded in 2008, it is one of the leading recommendation engines in Turkey.

At first, Convertale ran its recommendation engine on dedicated on-premises servers, but it soon encountered limitations in terms of scale. “We build code into our clients’ websites in order to collect data from the source,” says Ismail Arslan, Product Manager at Convertale. “So when a client’s website receives a lot of traffic from potential customers, our infrastructure receives all that traffic too. This means we need to scale out rapidly. The main problem with our dedicated setup was that we weren’t able to cope with this traffic.”

In addition to being unable to meet these traffic demands, the on-premises hardware was also unresponsive for testing and experimenting purposes. Arslan continues, “To find patterns in buying behavior, our data scientists need to experiment with big data, which requires significant compute power. With our on-premises setup, not only was it difficult to predict how many servers we would need for a given experiment, but we didn’t have an infinite amount of servers so we were limited in what we could achieve.”

Convertale needed a cost-effective infrastructure on which to run its recommendation engine that didn’t require upfront payment. It was also vital that any system it chose allowed seamless integration with other technologies, especially open source tools.

Building a recommendation engine requires running huge amounts of data, which is where the capacity provided by AWS proved to be a perfect fit. “AWS was the obvious choice, and after we did some testing, we knew it would provide us with the scale we needed at the right price,” says Arslan.
  
For the data collection part of the process, Convertale uses Elastic Load Balancing with Amazon Elastic Compute Cloud (Amazon EC2) instances. Auto Scaling automates the scaling of resources across the Amazon EC2 instances and Amazon Kinesis crunches all the data in real-time before it moves into Amazon DynamoDB and Amazon Simple Storage Service (Amazon S3).

The next step, which involves processing the data in order to find meaningful connections, makes use of Amazon Elastic MapReduce (Amazon EMR). Elastic Load Balancing is used again in order to deliver the recommendations. In addition, Convertale employs Amazon Relational Database Service (Amazon RDS) for managing relational data as well as AWS Elastic Beanstalk for deploying services.

The recommendation engine relies on a range of other technologies—for example, the Jenkins integration tool, RabbitMQ messaging, and other open source tools—that integrate with the firm’s AWS cloud infrastructure.

“It took us just two weeks to get fully live on AWS,” says Arslan. And the company now processes more than half a billion user behaviors a month for its e-commerce clients. When it has required assistance from AWS Support, Arslan says, “The response has always been knowledgeable. The AWS Support team knows its stuff.”

Using AWS, Convertale has access to the scale it requires to deliver a reliable and insightful service to its customers. With its on-premises setup, the availability of its system ran to 98 percent, but on its cloud infrastructure, this has risen to 99.965 percent. In addition, working in the cloud means the company no longer has to plan downtime and compromise service continuity when it needs to deploy new features or services. “Before, if we wanted to upload a new feature, we would have to inform clients and shut everything down for up to 10 minutes. Now, we can launch a new service without any downtime at all.” This means a smoother service and less disruption to clients. Unplanned downtime is also a thing of the past. “With our on-premises system we sometimes had issues that would cause unexpected downtime—on some occasions for as long as 20 minutes. This doesn’t happen with AWS.”

But, above all, it is the ability to cope with huge amounts of data that sees the most significant improvement from the startup’s previous infrastructure. “When we first migrated to AWS, we were working on a significant deal with a new client. If we’d been using our own data center we just wouldn’t have had the money to buy the additional servers to cope with the demand that this client would generate. Using AWS we were able to onboard that client and know we could deliver our service with confidence because we had the scale of the AWS cloud. And the great thing about AWS is that you don’t pay for anything upfront. It’s a great model for us,” says Arslan.

He continues, “We regularly run Amazon Elastic MapReduce jobs that may require hundreds or even thousands of servers at once, depending on the client. Processing this big data is extremely easy and cost-effective for us compared to doing it on a physical infrastructure. We just set them up, process the data, and shut them down as required.” This also benefits Convertale data scientists exploring connections in big data, allowing them to experiment and test. They can easily launch and shut down servers without worrying about the time and cost of provisioning physical resources. Arslan says, “When developers create new features for the recommendation engine, they need to test it in an environment similar to the production environment. If you’re working on dedicated hardware, you can’t do that. Using AWS, we can create multiple environments at once. This way, testing is simple, quick, and cost-effective. We test the new feature and once we’re done with this process; we simply terminate the AWS service and stop paying for it.”

The AWS pricing model has additional benefits in that it allows Convertale to price its own services more competitively. As Arslan says, “We have better control of our costs with AWS. This gives us an edge in the market because we can be competitive among global players.” In fact, since deploying AWS technologies, Convertale has not only eliminated its upfront costs, but it’s achieved savings in its own operating costs too. “We’ve seen a reduction in cost of around 50 percent since we started using AWS technologies such as Amazon DynamoDB, Amazon EMR, and Amazon Kinesis,” says Arslan.

When asked about his overall experience using AWS, Arslan concludes, “AWS is fantastic. Rubber wheels instead of stone wheels. It’s faster, more comfortable, needs much-less maintenance, and goes further.”

To learn more about how AWS can help you with your big data needs, visit our Big Data details page.