[text]
Important: This Guidance requires the use of Amazon Forecast, which is no longer available to new customers. Existing customers of Amazon Forecast can continue using and deploying this Guidance as normal.
This Guidance helps you increase customer engagement through a Product 360 framework. Product 360 is a data analytics framework that provides insights by aggregating relevant information from data sources for a comprehensive view of products. This comprehensive view enhances sales motions based on product usage patterns and guides product strategy and roadmap development. This Guidance is also combined with artificial intelligence and machine learning (AI/ML)-augmented recommendations to support data-driven decision-making for future products.
Please note: [Disclaimer]
Architecture Diagram
[text]
Step 1
Sources for Product 360 include product, customer relationship management (CRM), sales transaction, pricing, customer interaction, and clickstream data.
Step 2
AWS Database Migration Service (AWS DMS) ingests data from database and analytical sources, and Amazon AppFlow ingests data from software-as-a-service (SaaS) services.
Step 3
Ingested data is sent in its original, immutable format to an Amazon Simple Storage Service (Amazon S3) raw zone bucket.
Step 4
Data processing and pipeline orchestration is conducted using purpose-built data processing components and transformation libraries through AWS Step Functions and Amazon EMR. Amazon DynamoDB stores pipeline configuration and schema information about data sources.
Step 5
Amazon EventBridge initiates Step Functions, which invokes Amazon EMR to transform raw data into an efficient data format (Parquet). This data is moved to a clean zone S3 bucket.
Step 6
EventBridge initiates Step Functions again, which invokes Amazon EMR to use Apache Spark-based batch and streaming pipelines to create Apache Iceberg-based data in the curated zone S3 bucket from the clean zone S3 bucket.
Step 7
The analytics layer uses Amazon QuickSight and Amazon Athena to natively integrate with the curated zone for analytics, dashboards, ad-hoc reporting, and ML.
Step 8
Amazon Forecast forecasts product demand, and an S3 bucket stores forecast output. Athena and QuickSight query and visualize the forecast output.
Step 9
Using AWS Lake Formation, Product 360 users will get fine-grained access to data assets in the curated zone and forecast S3 buckets for consumption.
Step 10
AWS Glue Data Catalog stores technical metadata for all data assets in S3 buckets, which is used for querying the data assets from Athena and QuickSight.
Step 1
Sources for Product 360 include product, customer relationship management (CRM), sales transaction, pricing, customer interaction, and clickstream data.
Step 2
AWS Database Migration Service (AWS DMS) ingests data from database and analytical sources, and Amazon AppFlow ingests data from software-as-a-service (SaaS) services.
Step 3
Ingested data is sent in its original, immutable format to an Amazon Simple Storage Service (Amazon S3) raw zone bucket.
Step 4
Data processing and pipeline orchestration is conducted using purpose-built data processing components and transformation libraries through AWS Step Functions and Amazon EMR. Amazon DynamoDB stores pipeline configuration and schema information about data sources.
Step 5
Amazon EventBridge initiates Step Functions, which invokes Amazon EMR to transform raw data into an efficient data format (Parquet). This data is moved to a clean zone S3 bucket.
Step 6
EventBridge initiates Step Functions again, which invokes Amazon EMR to use Apache Spark-based batch and streaming pipelines to create Apache Iceberg-based data in the curated zone S3 bucket from the clean zone S3 bucket.
Step 7
The analytics layer uses Amazon QuickSight and Amazon Athena to natively integrate with the curated zone for analytics, dashboards, ad-hoc reporting, and ML.
Step 8
Amazon Forecast forecasts product demand, and an S3 bucket stores forecast output. Athena and QuickSight query and visualize the forecast output.
Step 9
Using AWS Lake Formation, Product 360 users will get fine-grained access to data assets in the curated zone and forecast S3 buckets for consumption.
Step 10
AWS Glue Data Catalog stores technical metadata for all data assets in S3 buckets, which is used for querying the data assets from Athena and QuickSight.
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaper
AWS services used in this Guidance are provisioned in the same AWS Region to reduce data transfer charges. Additionally, services such as QuickSight do not accrue any data transfer charges. Additionally, most services in this Guidance are either serverless or are available with a serverless option, such as Amazon EMR Serverless. Serverless services do not need to run for extended periods of time. You can take advantage of sustained usage through reserved instances for Amazon EMR.
-
SecurityRead the Security whitepaper
Lake Formation applies central audited governance, fine-grained access controls, and data classification tagging. This helps you secure data at the object-, database-, table-, column-, and row-level.
-
ReliabilityRead the Reliability whitepaper
Serverless services such as AWS Glue and DynamoDB scale horizontally, automatically responding to the velocity of data ingestion and processing. This scaling enables the architecture to adapt to changes imposed upon it.
-
Performance EfficiencyRead the Performance Efficiency whitepaper
The component-oriented architecture allows you to build independent layers using infrastructure as code (IaC). By separating the ingestion, processing, storage, unified governance, cataloging, and consumption, you can more easily test and deploy modules.
-
Cost OptimizationRead the Cost Optimization whitepaper
The serverless architecture reduces the amount of underlying infrastructure that needs to be managed, allowing you to focus on differentiated work, such as onboarding new customers and building new feature enhancements.
-
SustainabilityRead the Sustainability whitepaper
Amazon Forecast summarizes large data sets into meaningful insights. This reduces data volumes and processing, helping ensure that you only use the minimum resources required.
Implementation Resources
A detailed guide is provided to experiment and use within your AWS account. Each stage of building the Guidance, including deployment, usage, and cleanup, is examined to prepare it for deployment.
The sample code is a starting point. It is industry validated, prescriptive but not definitive, and a peek under the hood to help you begin.
Related Content
[Title]
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.
References to third-party services or organizations in this Guidance do not imply an endorsement, sponsorship, or affiliation between Amazon or AWS and the third party. Guidance from AWS is a technical starting point, and you can customize your integration with third-party services when you deploy the architecture.