Beyond Traditional Market Price Tracking: A Generative AI Solution for SAP Customers

Retail companies managing finished goods face mounting pressure to maintain competitive pricing while protecting profit margins in today’s dynamic marketplace. Traditional price monitoring approaches fall short when dealing with inconsistent product identification methods across vendors and the sheer volume of pricing data available across digital platforms.
Retail companies selling finished goods through multiple channels need comprehensive visibility into competitor pricing strategies to make informed decisions about their own pricing models. These companies typically struggle with manual price monitoring processes that are time-consuming, error-prone, and fail to provide the real-time insights needed for competitive positioning.
Without automated price tracking capabilities, these organizations risk losing market share to competitors who can respond more quickly to market changes or

In this blog post, we present a solution that combines SAP material master data extraction with generative AI to create a smart, automated price monitoring system tailored for SAP retail customers. You’ll see how the system pulls product information from SAP, sends AI agents out to scan online marketplaces for pricing and customer sentiment, and brings the results back into SAP for action, all with minimal manual effort. We’ll walk through the architecture, explain how Amazon Bedrock Agents and AWS services power the agentic workflow, and share practical considerations around integration, cost, and implementation so you can evaluate this approach for your own organization.

Business Value:

The solution delivers tangible benefits for retail companies through improved pricing intelligence and faster market response capabilities. Organizations typically see margin improvements through better pricing decisions enabled by real-time competitive data. The automated nature of the system reduces manual monitoring efforts, allowing pricing teams to focus on strategic analysis rather than data collection.
Additionally, the system’s ability to identify pricing trends and regional variations enables retailers to optimize their promotional strategies and inventory management decisions, leading to improved inventory turnover and reduced markdowns.

Architecture:

Key Components 

Connecting SAP data with AWS 

Before an agent can work with SAP data, that data needs to land in AWS — securely and in line with SAP’s licensing and governance rules. The modern way to do this is SAP Business Data Cloud (SAP BDC), which shares SAP data with AWS without copying it around.

At SAP SAPPHIRE 2026, SAP and AWS announced SAP BDC Connect for Amazon Athena — a bi-directional, zero-copy integration between SAP Business Data Cloud and Amazon Athena. SAP and AWS Enable Next-Generation AI. Rather than extracting and duplicating data, teams query semantically rich SAP data products in place, in near real time. The data keeps its original business meaning and stays compliant with SAP’s governance, and can then be landed in an Amazon S3 bucket for the agent to scan and analyze SAP and AWS Next-Generation AI.

The flow runs through SAP Datasphere, SAP BDC’s data fabric. From there, non-SAP and SAP data can be federated live into Amazon Athena (no copy), or written to Amazon S3 when a stored copy is needed for downstream processing of large datasets..

Other SAP Data Extraction options AWS Glue with OData/CDS views, partner tools (BryteFlow, Theobald, Qlik), and SAP-native SLT/Data Services — are available for scenarios that require extracting SAP data. See AWS’s SAP Data Integration Guidance and Integrating SAP BDC with AWS data sources.

Agentic Workflow

The Agentic workflow uses Amazon ECS Fargate to deploy containerized AI agents that autonomously analyze product pricing across multiple online marketplaces using Amazon Bedrock foundation models.

How does the workflow operate?

Step 1: BDC data extraction

Step 2: Product CSV files containing EAN codes and descriptions are uploaded to the SapLake S3 bucket, triggering the price monitoring workflow.

Workflow Initiation

When a CSV file containing product information is uploaded to Amazon S3, Amazon EventBridge triggers an AWS Step Functions workflow. The Step Functions state machine orchestrates the entire price monitoring process from data extraction to result aggregation. Additional information on setting triggers with S3 can be found here

Task Dispatch

Step 3: Lambda function (search-and-dispatch.py) processes CSV files, performs internet searches for each product, and dispatches ECS tasks for marketplace analysis.

AWS Step Functions uses a Map state to process multiple products concurrently, with controlled concurrency of three simultaneous agents to respect marketplace rate limits. Each product triggers an individual ECS Fargate task with specific environment variables including the target marketplace URL, product EAN code, description, and execution tracking ID.

Agent Execution

Step 4: ECS Fargate tasks run containerized Strands Agents using Amazon Bedrock and MCP servers to analyze product pricing across multiple online marketplaces.

Each ECS task runs a containerized AI agent configured with 2GB memory and 1 vCPU on ARM64 architecture for optimal cost-performance. The agent has access to Amazon Bedrock foundation
models including Claude, Titan, and Nova, with a 10-minute execution timeout for comprehensive marketplace analysis.

The Amazon Strands SDK orchestrates AI reasoning while the Model Context Protocol server bridges Bedrock models with Playwright browser automation. This architecture enables agents to plan multi-step navigation strategies across different marketplace layouts, adapt to dynamic content and anti-bot measures automatically, and use AI models to identify pricing elements while maintaining context throughout complex web interactions.

Step 5: The agent workflow combines Strands planning, MCP tools for Playwright execution to transform traditional web scraping into intelligent, adaptive market research that can handle the complexity and variability of modern e-commerce platforms.

Step 6: Each agent stores pricing analysis results in DynamoDB for real-time tracking and intermediate processing status updates.

Step 7: Lambda function (aggregate-results.py) consolidates all agent results from DynamoDB into comprehensive pricing reports.

Step 8: Aggregated pricing analysis and market intelligence reports are stored in S3 for further analysis and SAP integration.

Intelligent Analysis Process

The AI agent performs sophisticated analysis:

Web Navigation: Browser automation handles dynamic content
Product Matching: Bedrock foundation models identify matching products using EAN codes and descriptions
Price Extraction: extracts pricing information from various marketplace formats
Sentiment Analysis: Foundation models analyze customer reviews and market positioning
Result Storage: Structured data is written to Amazon DynamoDB with confidence scores
Error Handling and Resilience
The workflow includes automatic retry logic with exponential backoff for transient failures. Failed tasks are logged with detailed error context, while partial results are preserved in
DynamoDB for debugging and recovery.

Step 9: Integration back to SAP using cloud connector for further analysis.

BTP Integration Suite with Cloud Connector

The integration leverages SAP Business Technology Platform (BTP) Integration Suite as the primary method for securely connecting AWS services with SAP systems. The SAP Cloud Connector acts as a secure reverse proxy that enables on-premise or cloud SAP systems to communicate with cloud applications on SAP BTP without exposing internal systems directly to the internet. This approach maintains security boundaries while enabling seamless data flow between AWS and SAP environments.
When price analysis completes and results are stored in S3, the BTP Integration Suite orchestrates the data retrieval process through pre-configured integration flows. These flows authenticate with AWS using stored credentials, retrieve the pricing analysis data from S3, and transform it into the appropriate format for SAP consumption. The Integration Suite then updates material master records, pricing conditions, and other relevant business objects within the SAP system, enabling data-driven pricing decisions based on real-time market intelligence.
The Cloud Connector ensures that all communication remains secure by establishing encrypted tunnels between BTP and on-premise SAP systems, while the Integration Suite provides monitoring, error handling, and retry mechanisms for reliable data processing. This architecture allows organizations to maintain their existing SAP security policies while benefiting from cloud-based AI capabilities.
As an alternative approach, organizations can implement direct integration using EventBridge notifications combined with the ABAP SDK for AWS, which provides native AWS service integration directly within SAP systems for simpler deployment scenarios.

Key Advantages of ABAP SDK Approach:

Native Integration: No custom HTTP handling or JSON parsing libraries needed

Automatic Authentication: SDK handles AWS signature v4 authentication automatically
Error Handling: Built-in AWS error handling and retry mechanisms
Simplified Code: Reduces integration code by ~70% compared to custom HTTP calls
SAP Support: Officially supported by SAP, ensuring compatibility with future releases

Updated Architecture Benefits

Fewer Dependencies: No need for custom REST clients or JSON libraries
Better Performance: Optimized AWS API calls through native SDK
Easier Maintenance: Standard SAP development patterns
Enhanced Security: SDK manages credentials through SAP’s secure store
This approach transforms the integration into a standard ABAP development task, making it accessible to any SAP developer familiar with object-oriented ABAP programming.

Implementation Best Practices 

Error Handling 

• Implement robust error handling for SAP connectivity
• Handle website structure changes gracefully
• Monitor and log agent failures

Performance Optimization

• Use caching for frequently accessed data
• Implement parallel processing for agents

Compliance and Ethics

• Respect robots.txt and crawling policies
• Implement appropriate delays between requests
• Store only publicly available information

Implementation Best Practices

Error Handling

Robust error handling forms the foundation of a reliable price monitoring system, particularly given the complexity of integrating multiple external systems. SAP connectivity errors can occur due to network timeouts, authentication failures, or system maintenance windows, requiring implementation of backoff retry mechanisms and circuit breaker patterns to prevent failures. Website structure changes represent another critical challenge, as e-commerce platforms frequently update their layouts and HTML structures, potentially breaking traditional web scraping approaches. The solution addresses this by leveraging generative AI models that can adapt to structural changes by understanding content semantically rather than relying on fixed CSS selectors or XPath expressions. Comprehensive logging and monitoring of agent failures enables rapid identification of issues and provides valuable insights for system optimization, while automated alerting ensures that critical failures receive immediate attention from operations teams.

Performance Optimization 

Effective performance optimization directly impacts both system costs and user experience, making it essential for production deployments. Caching frequently accessed data, such as product catalogs and marketplace layouts, reduces redundant API calls and improves response times while minimizing costs associated with repeated data retrieval operations. This approach is particularly valuable for SAP material master data, which changes infrequently but is accessed repeatedly during price analysis workflows. Parallel processing of agents maximizes throughput by allowing multiple products to be analyzed simultaneously across different marketplaces, significantly reducing the total time required for comprehensive price monitoring cycles. The controlled concurrency approach prevents overwhelming target websites while optimizing resource utilization, ensuring that the system can scale efficiently as product catalogs grow.

Compliance and Ethics 

Ethical web scraping practices protect both the organization and target websites while ensuring sustainable long-term operations. Respecting robots.txt files and crawling policies demonstrates good digital citizenship and reduces the risk of IP blocking or legal challenges from marketplace operators. Implementing appropriate delays between requests prevents overwhelming target servers and mimics human browsing patterns, reducing the likelihood of triggering anti-bot measures that could disrupt data collection activities. Storing only publicly available information ensures compliance with data protection regulations and maintains ethical boundaries, while also reducing storage costs and simplifying data governance requirements. These practices build trust with marketplace partners and create a foundation for potential future API partnerships or data sharing agreements.

Cost Consideration

To implement this SAP price monitoring solution, you need to consider the following key cost drivers:

Key Cost Elements:

Compute time (ECS Fargate vCPU and memory hours)
Function invocations and duration (Lambda GB-seconds)
Workflow executions and state transitions (Step Functions)
Data storage and API requests (DynamoDB and S3)
Data extraction and processing (AWS Glue)

System Architecture Assumptions:

1,000 CSV files uploaded monthly (one per workflow execution)
10,000 ECS Fargate tasks for marketplace analysis (10 minutes each, 1 vCPU, 2GB ARM64)
2,000 Lambda function invocations (search-and-dispatch + aggregate-results per workflow)
1,000 Step Functions executions averaging 15 state transitions each
1,000 AWS Glue ETL jobs processing CSV files (5 minutes each with 1 DPU)
50,000 DynamoDB write requests and 25,000 read requests monthly
5GB S3 storage with 15,000 API requests monthly
11,000 EventBridge custom events monthly

Detailed Cost Breakdown (without Free Tier):

Amazon ECS Fargate pricing is reflected here. For ARM64 architecture: $0.03238 per vCPU-hour and $0.00356 per GB-hour. With 10,000 tasks
running 10 minutes each monthly: 1,666.67 vCPU-hours and 3,333.33 GB-hours = $65.83/month.
AWS Lambda pricing is reflected here. With 2,000 invocations and 30,000 GB-seconds monthly = $0.50/month.
AWS Step Functions pricing is reflected here. With 1,000 executions × 15 transitions = 15,000 transitions = $0.38/month.
Amazon DynamoDB pricing is reflected here. With 50,000 writes and 25,000 reads monthly = $0.07/month.
Amazon S3 pricing is reflected here. With 5GB storage and 15,000 API requests (10,000 PUT, 5,000 GET) = $0.12/month.
AWS Glue pricing is reflected here. Assuming 1,000 extraction jobs running 5 minutes each (83.33 DPU-hours monthly) = $36.67/month.
Amazon EventBridge pricing is reflected here. With 11,000 events monthly (1,000 workflows + 10,000 task completions) = $0.01/month.

Total Monthly Cost Estimate

For processing 10,000 URLs across 1,000 workflow executions:

ECS Fargate (ARM64) = $65.83
Lambda functions = $0.50
Step Functions = $0.38
DynamoDB = $0.07
S3 storage = $0.12
AWS Glue = $36.67
EventBridge = $0.01
Total estimated monthly cost = $103.58

Cost Optimization Options

Fargate Spot instances: Up to 70% discount, reducing Fargate cost to ~$19.75
Reserved capacity: DynamoDB and other services offer reserved pricing
S3 Intelligent Tiering: Automatic cost optimization for infrequently accessed data
Glue job optimization: Right-sizing DPU allocation and job duration

With Fargate Spot pricing: $57.50/month total

Conclusion/Summary

The system provides comprehensive price monitoring for finished goods across multiple online marketplaces, automated trend analysis for price fluctuations, intelligent identification of comparable products and correlates customer sentiment data with pricing effectiveness to provide deeper market insights.
This intelligence empowers SAP customers in the retail sector to make data-driven pricing decisions, respond quickly to competitive threats, and maintain their market position in an increasingly dynamic retail landscape.
Strands Agents together with AWS services provide a powerful platform for building intelligent price monitoring systems. By combining SAP data extraction with Agentic AI capabilities, organizations can gain valuable insights into market positioning and competitive pricing strategies.