GrowByData Transforms Operations to Gain Scalability, Reduce Costs with Genese Solution on AWS

GrowByData, based in the US and Nepal, enables its clients to collect and process data from multiple sources to generate insights used for growth strategy. It offers improved market visibility and performance to brands through its share-of-voice reporting, competitive price intelligence, and brand defense solutions. Customers include companies in retail and financial services and in large media holding companies. The company provides its clients with customized market insights as condensed and actionable strategy reports. Previously, these reports were based on a vast, and growing, volume of data that was ingested, analyzed, and processed by the company’s business intelligence platform as ETL jobs. The outputs were then turned into customer-specific insights by the company’s data scientists and market strategy professionals.

As GrowByData grew, its customers increasingly wanted more insights, and they wanted them faster. The volume of both structured and unstructured data increased, and that data was also being captured more quickly. This stressed the company’s infrastructure—a hybrid of on-site systems, AWS, and other cloud providers—and increased costs. The company had to overprovision its compute resources to cope with the demand on its servers. The number of faults that it reported on its ETL jobs was unacceptable. It needed greater scalability to economically cope with its concurrent ETL processing loads. 

GrowByData had initially built its data pipeline and ETL infrastructure on standalone Amazon Elastic Compute Cloud (Amazon EC2) instances. Over time, the system evolved significantly, growing from a few job types to dozens, each requiring execution for hundreds of clients. Configuring these jobs became increasingly cumbersome, because each client had unique requirements and parameters, leading to a complex and time-intensive setup process. The initial standalone architecture, which had previously been sufficient, now struggled to handle the growing scale, variability, and operational demands of the pipeline.

To address these challenges, GrowByData conceptualized and built a new, scalable architecture that automated job execution and introduced a configuration management application. Using this application, delivery analysts could control the behavior of ETL jobs directly through a user-friendly web interface, eliminating the need for software or data engineers to manually configure jobs. The new architecture was designed to be fully dynamic, with Apache Airflow serving as the background orchestrator engine. Apache Airflow was custom-architected to integrate seamlessly with the configuration management system, enabling job definitions to be dynamically managed and updated through the web interface.

GrowByData partnered with Genese Solution to deliver this vision as a cloud-native solution. The company had worked with Genese for around 7 years and, because both companies have offices in Nepal, they were able to work closely to define, design, and build a solution that could address GrowByData’s pain points. “We needed a partner who was an AWS specialist,” says Prasanna Dhungel, co-founder and managing partner at GrowByData. “Genese can guide us in terms of cost management and act as our partner if those costs start to get out of control.”

Through collaborative problem-solving, Genese helped design and implement a dynamic ETL orchestration framework, integrating Apache Airflow with GrowByData’s configuration management system. Together, they built a fully scalable data pipeline using Amazon Elastic Container Service (Amazon ECS) for container orchestration, Amazon Simple Storage Service (Amazon S3) for staging, Amazon ElastiCache for caching, and Amazon Redshift for data warehousing.

The architecture was designed to auto scale on demand, run jobs horizontally and in parallel, and ensure fault tolerance and observability. The integration of the configuration management system with Apache Airflow further enhanced maintainability, enabling delivery analysts to manage job configurations easily. Genese helped validate key architectural decisions by carrying out a structured proof of concept (PoC) phase.

GrowByData’s new, scalable infrastructure has transformed the way it manages its workflows and delivers actionable marketing intelligence. The company can now support 4 times as many concurrent data workflows as it could before and reports a 70 percent reduction in ETL process failures. It estimates that it is 20 times more scalable than previously, with a 3 times improvement in data observability. 

With auto scaling of AWS services, resources are now dynamically allocated based on demand, meaning that GrowByData pays only for the compute power it uses. This has eliminated overprovisioning and helps to optimize costs during peak processing times. Additionally, costs have been cut by 70 percent. The enhanced architecture provides GrowByData with better fault tolerance and reduced maintenance overhead. By combining GrowByData’s innovative design with Genese’s AWS expertise, the solution not only met the current demands of the data pipeline but also positioned the company for future growth and adaptability.

The web-based workflow manager has reduced the heavy dependence on the engineering team. “Defining new workflows is very much simpler now,” says Bimarsh Pokharel, director of product engineering at GrowByData. “If we have a new data workflow requirement, instead of getting it done in a couple of weeks, we can just do it in a couple of days.” 

For co-founder Dhungel, the project enables the company to consistently and reliably deliver actionable intelligence to its clients at scale. “The collaboration between AWS, Genese, and GrowByData allows us to provide needle-moving strategic insights quickly to leadership teams at some of the largest brands in the world,” he says. “Our new capabilities allow us to delight our clients at these times of unprecedented changes as digital platforms face significant competitive pressures.”