Razor Group’s journey to a modern data lakehouse on AWS

Razor Group is one of Europe’s leading ecommerce aggregators, operating 250+ brands across multiple global marketplaces. With a portfolio exceeding $400M in revenue, the company relies on data to power every critical business decision, from dynamic pricing and inventory optimization to advertising spend and supply chain orchestration.

At the heart of this operation sits the Razor Operating System (ROS), a proprietary platform that processes 370M+ API calls monthly through 9,300+ data pipelines, transforming marketplace signals into automated actions at scale.

In this post, we share how Razor Group optimized their data platform by implementing a lakehouse architecture on AWS. We cover the architectural decisions, the phased migration approach, and the measurable business outcomes. Whether you’re looking to optimize workload performance, reduce infrastructure costs, or unlock multi-engine flexibility for your analytics, this blueprint provides actionable insights you can adapt for your organization.

The business challenge: Scaling data infrastructure for hypergrowth

As Razor Group’s brand portfolio expanded rapidly, the demands on their data platform grew significantly. The company needed their analytics infrastructure to keep pace with the speed of ecommerce, where pricing decisions, stock replenishment, and advertising bids happen in near real time.

Their existing architecture, built on Amazon Redshift provisioned clusters, had served them well during earlier growth stages. As workloads diversified and data volumes surged, several optimization opportunities emerged:

Figure 1: The Razor Operating System data architecture before the migration

• Workload contention: Over 1,000 SQL models for ETL, transformation, and analytics competed for the same compute resources, creating resource contention during peak processing windows.
• Cost-to-utilization mismatch: Always-on clusters ran 24/7, but workload analysis revealed that 98% of compute demand came from batch ETL rather than interactive analytics, which resulted in significant idle capacity during off-peak hours.
• Data freshness gaps: Batch-oriented pipelines delivered data with 4–6 hour latency, limiting the team’s ability to react to fast-moving marketplace dynamics.
• Scaling constraints: As concurrent users and pipeline complexity grew, vertical scaling alone couldn’t address the need for workload isolation and elastic capacity.

These weren’t failures of any single service. They were signals that the architecture needed to evolve to match the scale and diversity of Razor Group’s workloads.

Why a lakehouse architecture?

Rather than replacing their existing investments, Razor Group recognized the opportunity to optimize workload placement by adopting a modern lakehouse architecture. The core principles driving this decision:

• Open table formats: Apache Iceberg provides ACID transactions, time travel, and schema evolution. Data is stored once and accessed by any compatible engine without duplication.
• Elastic, per-workload scaling: With data persisted on Amazon Simple Storage Service (Amazon S3), each engine independently scales compute to match its workload. Each engine spins up for peak processing and scales to zero when idle, without over-provisioning shared infrastructure.
• Multi-engine flexibility: Different workloads have different requirements. Heavy ETL benefits from distributed Spark processing, ad hoc exploration from serverless queries, and business intelligence (BI) dashboards from high-performance warehouse engines, each optimized for its purpose.

This approach allowed Razor Group to right-size each workload to the best-fit engine while maintaining a single, governed copy of data accessible across the entire platform.

Solution overview

Razor Group partnered with AWS to implement a comprehensive lakehouse architecture that brings together multiple AWS services, each playing a complementary role:

Figure 2: End-to-end lakehouse architecture on AWS

Designing for scale: The lakehouse vision

The core insight driving Razor Group’s new architecture was simple: build a single, open format data lake that any engine can query. In the old model, each tool maintained its own copy of the data. In the new model, a single open-format data lake on Amazon S3 serves as the source of truth, and multiple purpose-built compute engines read from it based on the workload at hand.

This shift, commonly called a lakehouse architecture, combines the cost economics and scalability of a data lake with the query performance and governance of a data warehouse. Its open table format, Apache Iceberg, provides ACID transactions, schema evolution, time travel, and no vendor lock-in.

Storage and governance: The open data foundation

• Amazon S3 Tables (a capability of Amazon S3) with Apache Iceberg — The primary storage layer, providing open-format tables with ACID transactions, partition evolution, and time travel. Data is stored once and accessible by any Iceberg-compatible engine.
• AWS Glue Data Catalog — A unified metadata repository for consistent data discovery across all compute engines.
• AWS Lake Formation — Fine-grained access control with column-level and row-level security so that governance scales with the platform.

Compute: Right engine for the right workload

• Apache Spark on Amazon Elastic Compute Cloud (Amazon EC2) — Elastic, distributed compute for heavy ETL and transformation workloads. It uses AWS Graviton instances and Amazon EC2 Spot Instances for cost optimization.
• Amazon Athena — Serverless SQL for ad hoc exploration and lightweight queries directly on Iceberg tables, with no infrastructure to manage.
• Amazon Redshift Serverless — High-performance serving layer for BI dashboards, Tableau workloads, and interactive analytics. Amazon Redshift Serverless automatically scales to meet demand and pauses when idle, so it stays cost-efficient for the analytics workloads it serves best.

Orchestration and observability

• Apache Airflow — Pipeline orchestration that manages 9,300+ data pipelines with dependency tracking and service level agreement (SLA) monitoring.
• Comprehensive observability stack — Cost attribution, pipeline health monitoring, and data quality checks across all layers.

Note: When the architecture was originally designed, Amazon Redshift lacked Iceberg write support, making self-managed Spark the only viable ingestion path. This constraint has since been removed. Amazon Redshift now supports full Apache Iceberg DML (UPDATE, DELETE, MERGE), complementing its earlier CREATE/INSERT capabilities and AWS Glue Iceberg materialized views. This makes it a complete read/write Iceberg engine.

Migration approach

Rather than a risky big-bang cutover, Razor Group adopted a phased migration of five stages, each delivering standalone value while building the foundation for the next. Both Amazon Redshift and Spark pipelines ran in parallel during the transition, which maintained business continuity and let the team compare outputs with confidence. At no point was a production pipeline paused or a dashboard unavailable.

The migration journey: Five phases

The migration unfolded across five structured phases, each building on the previous one and delivering incremental value before the next began.

Phase 1: Establish the lakehouse foundation

Before migrating a single query, Razor Group needed to answer three questions: where does the data live, how is it managed, and how do we query it?

Why S3 Tables over self-managed Iceberg

Razor Group had already committed to Apache Iceberg as the table format: open, engine-agnostic, and equipped with ACID transactions and time travel. The question was whether to self-manage Iceberg on standard S3 buckets or use Amazon S3 Tables.

Self-managed Iceberg is powerful but operationally expensive. Someone has to run compaction jobs to prevent small-file proliferation. Someone has to expire old snapshots before metadata bloat degrades query planning. Someone has to clean up orphaned data files after interrupted writes. With 700+ models running across 40+ schemas, many of them materializing multiple times per day, that maintenance burden would scale with the platform rather than shrink.

S3 Tables eliminated this entire category of work. Compaction, snapshot management, and unreferenced file removal run continuously and automatically. The integrated Iceberg REST Catalog API means any compatible engine, such as Spark, Trino, Athena, Amazon Redshift, and Flink, can discover and query tables without maintaining a separate metastore. Discovery is unified through AWS Glue Data Catalog, which now exposes the Iceberg REST Catalog protocol as its access interface. Because tables are first-class AWS resources, access control, encryption, and lifecycle policies operate at the table level rather than through complex S3 bucket policies layered on top of file-path conventions.

For a company that didn’t want the operational burden of self-managing open table format maintenance, this was the deciding factor.

AWS Glue Data Catalog provides unified metadata discovery across all tiers. Lake Formation handles column- and table-level access control, with AWS Identity and Access Management (IAM) roles that follow least-privilege principles and AWS CloudTrail turned on for a full audit trail.

Choosing the query protocol

Prior to the rearchitecture, the Amazon Redshift cluster was 98% ETL, and only a fraction of compute hours were analyst SELECT queries. The replacement engine needed to handle both heavy batch transformations and interactive ad hoc queries.

Traditional Spark (spark-submit) handles batch ETL well, but couples clients to the cluster. Every job requires packaging driver JARs, managing classpaths, and submitting from within the cluster. For a platform running 200+ production directed acyclic graphs (DAGs) that process massive data volumes daily, this operational friction was a non-starter.

Spark Connect is the gRPC-based client-server protocol introduced in Spark 3.4, and it solved the coupling problem entirely. The cluster runs a persistent gRPC endpoint. Clients connect remotely and submit queries over the wire. Airflow operators become thin clients: they open a session, submit SQL, and get results, with success and failure mapping directly to task states. There are no driver JARs and no polling. Multiple consumers, including pipeline orchestrators, the web application, and developer notebooks, share one cluster without any of them needing Spark installed locally.

Deploying Spark Connect

Razor Group deployed a self-hosted Spark cluster on Amazon EC2: an on-demand AWS Graviton leader node, Spot workers at about 70% cost savings, and the Spark Connect endpoint exposed through an internal Network Load Balancer. Custom Amazon Machine Images (AMIs) bake in the full Spark, Iceberg, and S3 Tables stack, so private-subnet nodes have everything they need without internet access at runtime.

This phase produced no immediate business value, but it made everything that followed possible.

Phase 2: Migrate data ingestion

Razor Group’s ingestion layer pulls data from Amazon Selling Partner API, Seller Central portals, NetSuite ERP, and custom web scrapers. In the previous architecture, all of this landed in Amazon Redshift through COPY commands, which meant data freshness was dictated by batch job schedules and competed for resources on the same cluster that served analytical queries.

Razor Group migrated these pipelines to AWS Lambda functions orchestrated by Apache Airflow, writing data directly to S3 Tables in Iceberg format. The shift from schedule-driven to event-driven significantly improved freshness. Lambda functions spin up only when there’s data to process, and Airflow sensors trigger downstream transformations the moment new data lands. This replaced rigid hourly batch windows with data freshness measured in minutes.

The orchestration layer manages 200+ DAGs across 90+ flows and processes data from dozens of sources at scale. The migration required rewiring destinations from Amazon Redshift COPY to Iceberg writes, but the orchestration logic itself carried over with minimal changes.

This phase alone eliminated roughly 40% of compute costs by severing the always-on cluster dependency for ingestion.

Phase 3: Transform processing pipelines

This was the most technically demanding phase, and where Razor Group learned the most. The team migrated 1,000+ SQL models from Amazon Redshift to Apache Spark, working incrementally up the dependency chain across 40+ schemas. The models moved through a medallion structure: Bronze for raw ingested data, Silver for cleaned and conformed data, and Gold for business-ready aggregates.

Razor Group built automated conversion tooling and a validation framework that ran both Amazon Redshift and Spark outputs in parallel, comparing results row-by-row before decommissioning anything. Several categories of transformation pushed the limits of what automation could handle:

• Window functions: The QUALIFY clause in Amazon Redshift has no Spark equivalent. Each instance required wrapping in a subquery with explicit row numbering, which affected dozens of models in the inventory schema alone.
• JSON serialization: The most time-consuming category. Complex columns stored as JSON STRING in Amazon Redshift needed from_json() with hand-written STRUCT definitions in Spark. Every nested payload column across ads, orders, and transaction pipelines required schema introspection, with no shortcuts.
• Function dialect: More than 20 function-level conversions, including NVL to COALESCE, DATEADD to interval arithmetic, and LISTAGG to ARRAY_JOIN(COLLECT_LIST()).
• Snapshot elimination: The single biggest hidden cost. Full table copies that ran multiple times daily only to preserve point-in-time state consumed more than 35 hours of weekly Amazon Redshift compute. With Iceberg’s native time travel, these became zero-cost operations overnight.

When migrating 1,000+ SQL models, automated tooling handles the mechanical syntax conversions well. But roughly 30% of the models required human judgment: those with complex JSON payloads, deeply nested window functions, or cross-schema snapshot dependencies. These models consumed 70% of the migration effort.

Razor Group built a structured migration workflow that used Claude to accelerate this work: read source SQL, identify dependencies, convert syntax, resolve missing base tables, add JSON parsing, validate outputs, and write to the lakehouse. The system did more than translate SQL. It applied schema context, traced cross-model dependencies, and flagged edge cases that would have taken engineers hours to find manually. What could have been a multi-year effort became a systematic, repeatable process measured in weeks. This approach fundamentally changed the speed of migration.

Phase 4: Unify the serving layer

With data flowing through Iceberg tables, Razor Group collapsed the serving layer. End users query Gold-layer Iceberg tables through Amazon Redshift Serverless, and internal exploration and machine learning (ML) workloads read the same tables through Spark Connect. This removed the need to maintain separate data copies, materialized views, or extract jobs for different consumers.

This is the strategic payoff of an open table format. Iceberg tables on S3 are engine-agnostic: Spark for batch transforms today, Trino for interactive queries tomorrow, Flink for streaming next quarter. Any engine that speaks Iceberg can read the data without conversion or migration. Razor Group went from being locked into a single vendor’s SQL dialect to having the freedom to adopt new engines without touching the storage layer.

Phase 5: Operationalize and observe

The final phase made the lakehouse production-grade. Razor Group built a comprehensive observability stack that aggregates metrics, traces, and logs from every pipeline component into a unified view. This view supports centralized log search, anomaly detection, and automated alerting that correlates failures across the entire data platform.

This observability layer did more than provide visibility. It gave the team confidence. When you’re running thousands of pipeline executions daily, you need to know within minutes when something breaks, what caused it, and which downstream consumers are affected. That’s the difference between reactive firefighting and proactive operations.

Pipeline orchestration consolidated around three patterns: a daily pipeline (ingestion to materialization to export to AI agent analysis), an operations worker polling every 15 minutes, and weekly scraper jobs.

The cutover was zero-downtime by design: both schedulers ran in parallel for two weeks. Automated comparison checks validated that every pipeline produced identical outputs before the prior architecture system was disabled.

Results and business impact

The lakehouse architecture delivered measurable improvements across every dimension:

The 63% reduction compares the lakehouse run-rate (January–March 2026) with the pre-rearchitecture run-rate (October–December 2025), the trailing three months before the rearchitecture. The figure is an apples-to-apples blended infrastructure number that includes compute and storage across both architectures. The before column covers Amazon Redshift cluster compute and managed storage. The after column covers Amazon EC2 (Spark workers, both on-demand and Spot), AWS Lambda, AWS Glue, Amazon Athena, Amazon Redshift Serverless, and S3 Tables storage. Data-transfer and ancillary services are excluded because they were not materially different between the two periods. Workload mix (the number of pipelines, models, and end-user query volume) was held broadly comparable across the two windows.

Lessons learned

Start with the decision loops, not the tools, and know your workload before you replace your warehouse.

The most valuable activity of the entire migration wasn’t writing a line of code. It was the Amazon Redshift workload analysis we ran before making any architectural decisions. Discovering that 98% of compute was ETL, with only a sliver going to analyst queries, validated the move to on-demand Spark. It also prevented us from over-provisioning the replacement infrastructure for interactive workloads that barely existed. Architecture decisions should always trace back to core business requirements: pricing accuracy, promotional responsiveness, intraday P&L visibility. Start there, not with the technology.

Design for multiple compute engines, and choose the right engine per workload.

One of the clearest lessons from running a single-engine architecture is what you give up. Avoid locking yourself into one compute layer for BI, ingestion, backfills, and ML alike, because they have fundamentally different cost and performance profiles. Iceberg, Spark, and S3 Tables work well together out of the box once you make the shift. The technology isn’t the hard part. The hard part is mapping 1,000+ models across 40+ schemas, tracing dependencies through 200+ DAGs, and discovering that a column is actually a JSON string silently serialized differently between two engines. Migration is as much an excavation project as an engineering one.

Automate conversion, but budget for the 30%.

Automated tooling handles mechanical syntax conversions well, and it should be the first tool you reach for. But models with complex JSON payloads, deeply nested window functions, or cross-schema snapshot dependencies require human judgment, and that work doesn’t compress. Roughly 30% of our models needed significant manual intervention, and those models consumed 70% of the total migration effort. Plan for it honestly from the start.

Observability must include cost attribution, and watch out for hidden cost bombs.

Snapshot operations were our biggest surprise. Full table copies that ran multiple times daily to preserve point-in-time state were costing more than 35 hours of weekly compute, and nobody questioned it because “that’s how snapshots work.” Iceberg’s time-travel capability eliminated their cost, and that single feature justified a meaningful portion of the migration on its own. More broadly, you cannot optimize what you cannot see, so track query-level usage and attribute it to teams and functions. Cost observability is not a nice-to-have. It’s foundational.

Governance isn’t optional. Build it into the foundation, and align stakeholders from day one.

Catalog and access control need to come first, before you scale adoption, not after. The same principle applies to people: migration is a cross-functional program, not an infrastructure project. Our two-week parallel run caught edge cases that row-level validation missed entirely: time zone differences between Amazon Redshift and Spark, partition pruning behavior under concurrent writes, and subtle ordering differences in non-deterministic window functions. That parallel run wasn’t a safety net. It was where the migration actually proved itself. None of it works without the right stakeholders involved and aligned from the very beginning.

Conclusion

Razor Group’s journey offers valuable lessons for organizations looking to optimize their data architectures:

1. Analyze your workload mix first. Understanding that 98% of compute was ETL rather than interactive queries guided the decision to offload heavy processing to elastic Spark, while preserving Amazon Redshift Serverless for the interactive analytics it handles best.
2. Design for multi-engine flexibility. Open table formats like Apache Iceberg eliminate the need to choose a single engine. Each workload runs on the engine best suited to its access pattern, cost profile, and performance requirements.
3. Automate migration, but budget for complexity. Automated transpilation handled 70% of SQL models, but the remaining 30% consumed 70% of engineering effort. Plan accordingly.
4. Observability must include cost attribution. Without per-workload cost visibility, optimization is guesswork. Razor Group discovered that Iceberg snapshot maintenance alone consumed more than 35 hours of compute weekly, a hidden cost that observability surfaced and automation resolved.
5. Build governance into the foundation. AWS Lake Formation and AWS Glue Data Catalog provided fine-grained access control from day one, not retrofitted after the migration.
6. Validate with parallel systems. A two-week parallel run between old and new architectures caught edge cases that automated testing missed, which supported a confident production cutover.

The road ahead

With the lakehouse foundation in place, Razor Group is positioned to accelerate innovation, from real-time pricing models to AI-driven inventory optimization, all powered by a unified, open, and governed data platform on AWS.

The company’s transformation demonstrates that modern data architectures aren’t about choosing between services. They’re about placing each workload where it performs best, using open formats to eliminate silos, and scaling each layer independently as the business grows.

To learn how other organizations are implementing similar lakehouse architectures on AWS, see How BigBasket uses the Iceberg-based lakehouse architecture on AWS to power lightning-fast grocery delivery across India.

About the authors