Sprout Social Reduces Costs by 40% and Improves Performance by 50% Using Amazon EMR

Because Amazon EMR is a managed service that works using Apache Hadoop, it was a natural fit for the needs of Sprout Social. As a result, the company had an almost-seamless migration to Amazon EMR. In fact, the Sprout Social team could quickly import a snapshot it had taken of its existing Apache Hadoop cluster, and the service was up and running in a matter of hours. After migrating its first cluster in August 2021, Sprout Social completed the migration of two additional clusters by January 2022. The AWS team provided support for Sprout Social through the migration process, both with technical issues, like specific cluster-level settings to maximize performance, and cost-related issues, like testing without going over budget. “Because Amazon EMR is very easy to stand up, it was trivial for us to test this process a few times in advance,” says Johnson. “We had full confidence going into it that we knew what the actual migration window would be and could communicate that with the rest of engineering and support.”

Sprout Social saw the benefits of migrating to Amazon EMR almost immediately. The biggest benefit was that Sprout Social saw reduced costs using Amazon S3 storage over Amazon EBS volumes. “Amazon EMR is orders of magnitude cheaper for the large dataset we have,” says Johnson. “What that means is that we have more predictability around our cost as our company and our dataset expands.” Using Amazon EMR, scaling clusters is now significantly more straightforward than its self-managed solution, which saves many hours of Sprout Social engineers’ time. Also, the Sprout Social team estimates that it saved roughly 40 percent in total costs over its previous data storage solution.

Sprout Social has also seen improvements of 30–50 percent in overall batch job performance, traditionally its biggest bottleneck given how much data must be processed in any given job. “Amazon EMR has been an absolute game changer because of our ability to scale compute independently from storage,” Johnson says. “And we’ve seen less instability due to disk input/output and overall better and more predictable job run times on Amazon EMR, as opposed to our old traditional Apache Hadoop stack.”

Going forward, Sprout Social is planning to further optimize its use of Amazon EMR. Specifically, the team wants to explore how it could reduce the size of its main cluster and start using ephemeral clusters to handle batch jobs on a more as-needed basis. By doing so, it hopes to reduce costs associated with operational overhead and provide new features to its customers that wouldn’t have been possible before it migrated to Amazon EMR.

“Tools like Amazon EMR are critical to our ability to invest our money wisely and in areas other than data storage,” says Johnson. “AWS offerings make it possible for us to continue investing heavily in research and development and developing customer features as opposed to fighting a battle to keep costs under control.”