Prepare for customer demand with accurate forecasts powered by machine learning
This Guidance helps you improve the customer experience by using demand forecasting to estimate future customer demand. By predicting future demand and purchasing patterns, you can plan ahead for supply chain optimization and inventory management. This architecture shows the process of developing an estimate of future customer demand. Retail organizations can apply this Guidance to increase demand forecast accuracy, analyze various internal and external variables, and highlight low-code, no-code, or other advanced approaches.
Architecture Diagram
Step 1
Current and historical event data is gathered from retail stores (point of sale), manufacturing and distribution centers, customer relationship management (CRM), enterprise resource planning (ERP), enterprise performance management (EPM), and third-party systems like product vendors and logistic partners. For hybrid architectures, customers can use AWS Direct Connect or AWS Site-to-Site VPN.
Step 2
Data is ingested in batch or real-time pipelines. Amazon Kinesis can be used for streaming data ingestion and Amazon IoT Core for IoT devices AWS Database Migration Service (AWS DMS) can be used to fetch data from existing databases and AWS DataSync and AWS Storage Gateway for transfers. Customers can upload data into Amazon Simple Storage Service (Amazon S3) and use Amazon AppFlow for SAP and Salesforce
Step 3
Data stores like Amazon S3, Amazon Timestream, Amazon Relational Database Service (Amazon RDS), Amazon DynamoDB, and Amazon Redshift provide an intermediate landing zone (also called a raw zone) for historical and real-time time series data, item metadata, and related time series data.
Step 4
Customers can select and query the data from various sources, cleanse and explore the data, check for outliers, statistical bias, analyze feature importance, perform feature engineering and finally export the prepared data. Amazon SageMaker Data Wrangler is a low-code/no-code option to select data from multiple data sources such as Amazon S3, Amazon Athena, Amazon Redshift, AWS Lake Formation, Snowflake, and Databricks Delta Lake.
AWS Glue DataBrew provides a visual data preparation tool to clean and normalize data to prepare for machine learning. Amazon EMR and Amazon SageMaker processing jobs are advanced options for data transformation and feature engineering. AWS Glue can also filter, aggregate, and reshape data. Amazon Athena can perform federated queries on multiple relational, non-relational, object, and custom data sources. AWS Lambda can be used for data transformation.
Step 5
Prepared data is placed into special datasets on Amazon S3 and Amazon SageMaker Feature Store, ready to be ingested for training.
Step 6
In this step, training and hyperparameter tuning take place. Customers can use orchestrators like AWS Step Functions, Amazon Managed Workflow for Apache Airflow, or Amazon SageMaker Pipeline to help orchestrate the end-to-end ML process.
Step 6a
Training can be performed in Amazon SageMaker Canvas or Amazon SageMaker Autopilot as low-code/no-code options. Customers can simply browse the data and select the target, then SageMaker generates a model. Additionally, SageMaker performs hyperparameter tuning according to the defined objective metrics. This option is perfect for business analysts and data analysts.
Step 6b
Optionally, customers can also use Amazon Forecast, a fully managed service. Customers can use neural network algorithms like CNN-QR or DeepAR+ when an item’s metadata dataset is available, or prophet, ARIMA, or ETS. Customers can use RETAIL data domain options available in Forecast. This option is great for data engineers and developers.
Step 6c
SageMaker is an advanced option for developers and data scientists who can use build-in algorithms like DeepAR or AutoML libraries such as AutoGluon, or import existing models and perform transfer learning. Customers can use advanced options like data parallelism and model parallelism, then use built-in features like Amazon SageMaker Clarify for bias and Amazon SageMaker Model Monitor for monitoring.
Step 7
The model from the prior phase can be deployed for an ML inference to predict the forecast. There are two options: Batch inference for asynchronous forecasting or real-time inference that leverages real-time API. For batch inference, the forecast is placed in Amazon S3 as a CSV file. For real-time inference, the model is hosted in Sagemaker or Forecast and served as an API for business applications to invoke and get predictions.
Step 8
The output can be consumed and distributed through channels like Amazon QuickSight dashboards for business intelligence and analysis, exposed as an API using Amazon API Gateway, or pushed to a data lake or warehouse like Amazon Redshift. After the channel set up, the data can be consumed by the source systems like ERP, EPM, or CRM, or by other internal or external applications. The latest business and customer data is fed in continuously and the model is retrained to ensure accuracy
Step 9
Security: During the integration of various AWS services, AWS Identity and Access Management (IAM) roles are used to ensure least privileged access and AWS Key Management Service (AWS KMS) is used to ensure data encryption.
Step 1
Current and historical event data is gathered from retail stores (point of sale), manufacturing and distribution centers, customer relationship management (CRM), enterprise resource planning (ERP), enterprise performance management (EPM), and third-party systems like product vendors and logistic partners. For hybrid architectures, customers can use AWS Direct Connect or AWS Site-to-Site VPN.
Step 2
Data is ingested in batch or real-time pipelines. Amazon Kinesis can be used for streaming data ingestion and Amazon IoT Core for IoT devices AWS Database Migration Service (AWS DMS) can be used to fetch data from existing databases and AWS DataSync and AWS Storage Gateway for transfers. Customers can upload data into Amazon Simple Storage Service (Amazon S3) and use Amazon AppFlow for SAP and Salesforce
Step 3
Data stores like Amazon S3, Amazon Timestream, Amazon Relational Database Service (Amazon RDS), Amazon DynamoDB, and Amazon Redshift provide an intermediate landing zone (also called a raw zone) for historical and real-time time series data, item metadata, and related time series data.
Step 4
Customers can select and query the data from various sources, cleanse and explore the data, check for outliers, statistical bias, analyze feature importance, perform feature engineering and finally export the prepared data. Amazon SageMaker Data Wrangler is a low-code/no-code option to select data from multiple data sources such as Amazon S3, Amazon Athena, Amazon Redshift, AWS Lake Formation, Snowflake, and Databricks Delta Lake.
AWS Glue DataBrew provides a visual data preparation tool to clean and normalize data to prepare for machine learning. Amazon EMR and Amazon SageMaker processing jobs are advanced options for data transformation and feature engineering. AWS Glue can also filter, aggregate, and reshape data. Amazon Athena can perform federated queries on multiple relational, non-relational, object, and custom data sources. AWS Lambda can be used for data transformation.
Step 5
Prepared data is placed into special datasets on Amazon S3 and Amazon SageMaker Feature Store, ready to be ingested for training.
Step 6
In this step, training and hyperparameter tuning take place. Customers can use orchestrators like Amazon Step Functions, Amazon Managed Workflow for Apache Airflow, or Amazon SageMaker Pipeline to help orchestrate the end-to-end ML process.
Step 6a
Training can be performed in Amazon SageMaker Canvas or Amazon SageMaker Autopilot as low-code/no-code options. Customers can simply browse the data and select the target, then SageMaker generates a model. Additionally, SageMaker performs hyperparameter tuning according to the defined objective metrics. This option is perfect for business analysts and data analysts.
Step 6b
Optionally, customers can also use Amazon Forecast, a fully managed service. Customers can use neural network algorithms like CNN-QR or DeepAR+ when an item’s metadata dataset is available, or prophet, ARIMA, or ETS. Customers can use RETAIL data domain options available in Forecast. This option is great for data engineers and developers.
Step 6c
SageMaker is an advanced option for developers and data scientists who can use build-in algorithms like DeepAR or AutoML libraries such as AutoGluon, or import existing models and perform transfer learning. Customers can use advanced options like data parallelism and model parallelism, then use built-in features like Amazon SageMaker Clarify for bias and Amazon SageMaker Model Monitor for monitoring.
Step 7
The model from the prior phase can be deployed for an ML inference to predict the forecast. There are two options: Batch inference for asynchronous forecasting or real-time inference that leverages real-time API. For batch inference, the forecast is placed in Amazon S3 as a CSV file. For real-time inference, the model is hosted in Sagemaker or Forecast and served as an API for business applications to invoke and get predictions.
Step 8
The output can be consumed and distributed through channels like Amazon QuickSight dashboards for business intelligence and analysis, exposed as an API using Amazon API Gateway, or pushed to a data lake or warehouse like Amazon Redshift. After the channel set up, the data can be consumed by the source systems like ERP, EPM, or CRM, or by other internal or external applications. The latest business and customer data is fed in continuously and the model is retrained to ensure accuracy
Step 9
Security: During the integration of various AWS services, AWS Identity and Access Management (IAM) roles are used to ensure least privileged access and AWS Key Management Service (AWS KMS) is used to ensure data encryption.
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaper
This Guidace highlights low-code and advanced options for customers to achieve more accurate forecasting. Each option (SageMaker and Forecast) has service observability built in to ensure models train efficiently according to objective metrics. The entire flow is an iterative process that ensures the business objective is achieved and the process is efficient.
-
SecurityRead the Security whitepaper
Direct Connect or Site-to-Site VPN is used to provide a secure connection between retail store or external systems and the AWS Cloud. Also, we leverage IAM and AWS KMS for securing and encrypting data.
-
ReliabilityRead the Reliability whitepaper
Highly available and reliable managed services like Amazon S3, DynamoDB, AWS Glue, and SageMaker help with platform scalability and high availability. Options exist for regional failover for all services with proper architecting, but they are not called out in this design specifically.
-
Performance EfficiencyRead the Performance Efficiency whitepaper
Scalable and highly available services like DynamoDB and Amazon S3 are used as core components to improve performance. During model training, customers can leverage automatic tuning features, which can help achieve the most performant models.
-
Cost OptimizationRead the Cost Optimization whitepaper
Customers are only charged for the time the models are trained. Customers can additionally leverage spot instances and elastic inference. Customers can also use cost effective methods like batch inference and serverless inference, although these are not specifically mentioned. Serverless services like Lambda, Step Functions, Athena, and AWS Glue are used to reduce the cost of the solution.
-
SustainabilityRead the Sustainability whitepaper
AWS managed services help with scale up and down according to business requirement and traffic, and are inherently more sustainable than on-premises solutions. Additionally, we have leveraged serverless components to automate the process of infrastructure management and make it more sustainable.
Implementation Resources
A detailed guide is provided to experiment and use within your AWS account. Each stage of building the Guidance, including deployment, usage, and cleanup, is examined to prepare it for deployment.
The sample code is a starting point. It is industry validated, prescriptive but not definitive, and a peek under the hood to help you begin.
Related Content
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.
References to third-party services or organizations in this Guidance do not imply an endorsement, sponsorship, or affiliation between Amazon or AWS and the third party. Guidance from AWS is a technical starting point, and you can customize your integration with third-party services when you deploy the architecture.