Deputy offers Enterprise-Grade Data Analytics and Increases Reporting Responsiveness by 99% Using Amazon Redshift

Deputy aimed to improve and enhance the data analytics capabilities of its platform. Since Deputy serves as the system of record for labor costs, which are the largest controllable expense in hourly workplaces, these enhancements were especially crucial for its customers.

However, as a multi-tenant, micro-service architecture SaaS platform, Deputy’s customer data was spread across multiple siloed databases. This led to inefficiencies in data analytics, slow report generation of up to 10 minutes, and untapped new revenue streams through advanced reporting.

As Deputy had begun to focus on medium- to enterprise-sized businesses (beyond 100,000 employees), the data volume grew by an order of magnitude. This increase required a more scalable and cost-efficient solution. Additionally, the company had to handle large amounts of Personal Identifiable Information (PII) across multiple databases. This posed challenges in maintaining consistent access controls and governance policies.

To address these issues, Deputy turned to AWS in 2023 to create a hub-and-spoke architecture with a unified data ingestion, governance, and query framework.

Deputy built a modern data architecture on AWS using Amazon Redshift, a fully managed cloud data warehousing service, as the central component of a hub-and-spoke data platform. By using Amazon Redshift, the company can now generate sophisticated reports in 6 seconds, a 99 percent improvement in speed. With faster reporting, Deputy gains access to deeper, data-driven insights, leading to reduced operational inefficiencies and enhanced agility and performance. 

The company also implemented Amazon Managed Streaming for Apache Kafka (Amazon MSK), a fully managed streaming data service for Apache Kafka. This was deployed to consolidate, stream, and ingest data from multiple sources in real time.

Meanwhile, Amazon Simple Storage Service (Amazon S3), an object storage service, served as an intermediary staging layer for future artificial intelligence and machine learning use cases.

By consolidating multiple siloed databases into a single, scalable data warehouse, the company optimized IT costs and maintained a predictable cost structure. Amazon Redshift's multi-tenancy support ensured multiple enterprises can now use Deputy’s platform without data interference, enhancing scalability. Additionally, Amazon Redshift’s data sharing and federated query features improved data governance and security.

Additionally, Deputy employed RA3 clusters to separate storage from compute. This increased flexibility for the company's "hub", added several serverless cluster endpoints as "spokes" for specific use cases, and unlocked a distributed, self-service consumption architecture for serving insights.

Deputy also consolidated and replicated 80 terabytes of workforce management data from the previous infrastructure to Amazon Aurora, a fully managed relational database, for analytical workloads. With Amazon Aurora’s high throughput rate, Deputy maintains over 200,000 MySQL databases distributed across 20 clusters, each containing over 2 million tables, with minimal lag and disruptions. With this migration, the company improved query speed and latency by 30 percent.

The new platform allows Deputy’s platform team to focus on developing features that enhance customer experience and improve internal efficiency. This setup fosters innovation, allowing the company to create new products using AWS data services such as Amazon OpenSearch Service. The company now accesses real-time insights, eliminating data silos and empowering teams to make decisions based on data.

By leveraging AWS data analytics services, we rapidly ingested millions of tables into Amazon Redshift and created reusable templates for data pipelines. This democratized data access and simplified governance by empowering domain owners and feature squads to create their own pipelines, significantly reducing the time needed for data replication.” said Kevin Tenggara, Senior Manager of Engineering, Deputy.

Deputy is currently working with AWS on several generative artificial intelligence (AI) proof of concepts. The company has established an intermediary staging layer within Amazon S3 to support AI, machine learning, and other data-related use cases in the future. These efforts aim to implement more intelligent features that can automate mundane tasks and provide quicker access to information, further improving customer experiences and operational efficiency. Eventually, Deputy plans to integrate generative AI with its platform to make quality-of-life improvements for customers.

“Building on AWS has revolutionized our data architecture, reduced report generation times, and provided near real-time insights. This transformation has enhanced our operational efficiency and empowered our teams to innovate rapidly. With Amazon Redshift and our ongoing AI initiatives, we are well-positioned to continuously deliver smarter, data-driven solutions that elevate our customer experience and drive business growth,” said Qamal Kosim-Satyaputra, Senior Director of Engineering, Deputy.