A Data lakehouse solution on AWS that helps in analysing assortment, planning, pricing and sales trends
Point of Sales Data Hub is a data lake solution built using lake house architecture on AWS. It helps to automate ingestion from various data sources. Some of common business reasons for consolidating sales data sources are -
- lack of single source of truth regarding company's operating revenue drivers
- no clarity on stalling profits/operating margins
- no clear-cut understanding of each brand's contribution to revenue growth and profit margins
- with pandemic, increasing inflation and changing consumer habits, CPGs need to enhance their pricing and promotion strategies to optimize share of market (SOM) and revenue growth
Even if CPGs have identified the problem statement, there are numerous technical and data-source specific challenges that they come across. Some of them are mentioned here:
- lack of automated way of ingesting data from sources
- Error-prone manual data ingestion mechanism that consumes more time and delays decision making
- some APIs are not stable and therefore needs web scraping solutions in tandem with API integration with custom logics
- different solutions for historical versus real-time data ingestion
- building, translating and standardizing metadata and schema across all data sources to enable common reporting and dashboarding across different functions
- preparing data for different consumption layers like BI dashboards and ML/AI models
- conducting pilots (test and learn) on Amazon Comprehend and Forecast8. data quality monitoring and governance
Therefore, CPGs want to build a lake house solution comprising data sources like Retailers (Offline and Online), syndicate data providers(Nielsen/IRI/Ipsos), distributors or franchises, on-trade channels (restaurants/cafes/hotels), sell-in data source, field sales applications, CRM, Loyalty, promotions, demographics and seasonality. Four key parameters are looked while integrating these sources:
- Integration Methodology (Email, API, SFTP, XLS/CSV, Web Scraping)
- Data refresh rates
- Ingestion methodology (MSK, Kinesis, Glue)
- Data dictionary
- Consumption layers (Quicksight/Tableau on AWS, ML/AI model development)
Availability
Global
Benefits
Reduced time to insights
90% reduction in time to insights with automated data ingestion.
1.5 M USD in cost savings
Getting access to sales data directly without third party helps in optimizing costs.
Increased accuracy in pricing
Improvement in accuracy of pricing and promotions models.
Better Share of Market predictions
Improvement in SOM prediction by analysing Assortment, Merchandising, Planning, Pricing and Sales.
-
How it works
-
Key activities
-
Customer contribution
-
About this consultant
-
Architecture diagram
-
How it works
-
1. Typically, the engagement starts with a scoping exercise where Sigmoid conducts discussions with the marketing, sales, pricing/RGM and IT/data team. The intent is to understand existing and new data sources, data engineering and management processes, and current business goals, like sales trends analysis, revenue growth parameters, and key technical constraints. After the scoping exercise, a scope of work (SOW) document is shared with the customer, incorporating key shortcomings, major goals, the solution approach, timelines, effort estimation, workload/tool choices on cloud, data sources, file formats, connectivity methodology (SFTP, API, web-scraping, XLS, and CSV, to name a few), best practices, and a detailed technical architecture
2. Once the customer is aligned, a team of data engineers and architects headed by a technical lead is assigned to start the integration process with each data source using pre-defined frameworks and templates built by Sigmoid. Each integrated data source is thoroughly tested and data is stored in Amazon S3. Data is then fed into Amazon Redshift through data transformation workloads, like AWS Glue, Amazon EMR, and Amazon Kinesis.
3. Once all the data sources are onboarded to Amazon S3, Sigmoid can help customers to build custom dashboards and visualizations on various tools, like Amazon Quicksight and Tableau, leading to insights on share of market (SOM), revenue growth drivers, assortment, competitive intelligence, pricing, sales trends etc.
4. The logical extension of this solution is to build and operate machine-learning (ML) models using Amazon Sagemaker and Amazon EMR to optimize SOM, pricing, merchandising to name a few
-
Key activities
-
1) Infrastructure provisioning
Development, Test, Staging and Production environments are setup.
2) Data CollectionIdentifying the sources, integration methodology (API, SFTP and Web scraping), data refresh rates.
3) Data TransformationOnce the data is ingested into S3, creating custom KPIs and building data dictionary.
4) Data ConsumptionReporting use-cases built for authorized users to either visualize the data as dashboards or consume as XLS.
5) Data PreparationPreparing the data for AI models that helps in arresting dwindling market share, optimize revenue growth etc.
6) Primary Coding LanguagesPython, JAVA.
7) Code RepositoryGithub
8) Data Quality and GovernanceOverall governance and data quality management of data pipelines in real-time.
-
Customer contribution
-
Access to Enviroments
Customers facilitate different environments to build, test, and deploy integrations.
Data Providers Access and API Docs
Customers set up a bridge with data providers to set up the secured access to APIs.
User Acceptance Testing (UAT)
Customers test the visualization and dashboards.
-
About this consultant
-
Sigmoid enables business transformation using data and analytics, leveraging real-time decisions through insights, by building modern data architectures using cloud and open source. Some of the world's largest data producers are engaging with Sigmoid to solve complex business problems. Sigmoid brings deep expertise in data engineering, marketing analytics, artificial intelligence, and DataOps.
Sigmoid is an Ace Elidgible Select tier AWS Partner having Amazon EMR Delivery, AWS Glue Delivery and Amazon Redshift Delivery services validated.
-
Architecture diagram
Ready to get started?
Related Resources
AWS Partner Highlights
Cloudwick’s AWS validated qualifications, customer references, and office locations.
AWS Competency Details
Cloudwick has demonstrated deep AWS technical expertise and proven customer success.
Browse our portfolio of Consulting Offers to get AWS verified help with solution deployment.
Browse our library of AWS self-deploy solutions to common architectural problems.
Engage with AWS Partners for secure, innovative, and cost-effective custom solutions that leverage the power and scalability of AWS services to meet your needs.