AWS for SAP

SAP ODP based Change Data Capture with Amazon AppFlow SAP OData Connector

Authors: Krishnakumar Ramadoss, Rozal Singh, Manoj Muthukrishnan, Ram Borhade, Rajendra  Narikimelli, Damian Maldonado, Ganesh Suryanarayanan

Introduction

5,000+  customers are running their SAP workloads on AWS. Besides saving costs, customers are reaping significant benefits from the depth and breadth of AWS services. Tens of thousands of data lakes are already deployed on AWS. Customers are benefiting from storing their data in Amazon S3 and analyzing that data with the broadest set of analytics and machine learning services to increase their pace of innovation. AWS offers the broadest and most complete portfolio of native ingest and data transfer services, plus more partner ecosystem integrations with S3.

Since we launched the Amazon AppFlow SAP OData connector in 2021, the most common request from customers was to provide them with a capability to extract data in increments using the built-in Change Data Capture (CDC) capabilities offered by the SAP Operational Data Provisioning (ODP) framework.

Based on customer feedback, today we are announcing the launch of a new Amazon AppFlow feature that supports data transfers from SAP applications to AWS Services using the SAP ODP framework in just a few clicks and comes with built-in Change Data Capture capabilities. With this launch, customers can use the Amazon AppFlow SAP OData connector to seamlessly perform full and incremental data transfers, including SAP Operational Delta Queue (ODQ) from SAP ERP/BW applications like SAP ECC, SAP BW, S/4 HANA, and BW/4 HANA.

Amazon AppFlow

Amazon AppFlow is a fully managed data integration service that helps customers build data flows to securely transfer data between AWS services, SaaS applications, and SAP ERP applications. These data flows help accelerate the creation of data marts and data lakes on AWS for analytical purposes or for combining the data from various source systems.

In this blog, we will walk through how you can setup an Amazon AppFlow data flow to extract data using the newly released feature of the SAP OData connector to connect to ODP providers that are exposed as OData services in your source SAP system.

The high-level architecture of ODP-based extraction with Amazon AppFlow data flows

ODP Architecture(1)

With ODP-based data transfers, business context is preserved. Retaining this business context from SAP data sources reduces business logic–mapping efforts to integrate data with business objects from other SAP and non-SAP data sources.

The ODP framework works on “provider” and “subscriber” models to enable data transfers between SAP systems and SAP to non-SAP data targets. Amazon AppFlow uses this ODP framework to support full-data extraction and change data capture through the Operational Delta Queues (ODQ) mechanism.

ODP Provider – The data provided by the source SAP system is called the ODP Provider. Below is a list of various ODP providers supported by the framework.

ODQ (Operational Delta Queue) – In the case of full or delta extractions, the data from the source system is written as data packages to an ODQ by the ODP provider using an update process.

ODP Consumers/ODQ Subscribers – The target applications that retrieve the data from the delta queue and continue processing the data are referred to as “ODQ Subscribers” or, more generally, “ODP Consumers” In this case, Amazon AppFlow plays the role of a consumer or subscriber.

ODP providers, in turn, can act as a data source for OData services, enabling REST-based integrations with external consumers like Amazon AppFlow. The ODP-Based Data Extraction via OData document details this approach on how customers can generate a service for consumers to extract ODP data via OData.

In Amazon AppFlow data flows, customers can use the SAP OData connector to connect to ODP providers that are exposed as OData services. The connector supports full extraction, which would allow customers to extract data in micro-batches from the ODQ.

Also, the connector supports incremental data transfers, which uses the built-in CDC capabilities provided by the ODP framework, allowing customers to seamlessly get the changed data at the source using a delta token with the operation of change, e.g., Inserted/Deleted/Updated. The support for the built-in CDC feature would allow customers to seamlessly pull the data from supported ODP source providers in micro batches using the intrinsic delta tokens provided by the ODP framework, thus making the data transfers more optimal.

Key Benefits of using ODP-based data extraction with Amazon AppFlow

  • Amazon AppFlow is a fully managed service that allows customers to build data flows in just a few clicks. It’s a low-code, no-code service.
  • Since the data extraction works at the SAP application layer, the business context of the data is retained.
  • Seamless integration into the well-established SAP ODP/OData framework to minimize the ramp-up or setup efforts.
  • Amazon AppFlow APIs can be used to integrate with other applications seamlessly.

Prerequisites for creating an ODP-based OData flow

  • The provider of your data source must be ODP enabled.
  • To generate an OData service based on ODP data sources, SAP Gateway Foundation must be installed locally in your ERP/BW stack or in a hub configuration. For your ERP/BW applications such as SAP ECC, SAP BW, S/4 HANA, and BW/4 HANA, the SAP NetWeaver AS ABAP stack must be at 7.50 SP00. The NetWeaver AS ABAP of the hub system (SAP Gateway) must be 7.50 SP00 or higher for remote hub setup. When the SAP ERP or SAP BW applications run on a lower version of NetWeaver that is less than 7.50 SP00, the hub scenario is recommended.
  • You must create an OData service from the SAP ODP source and register it for consumption in your service gateway. Refer to the SAP documentation for more details on generating and registering an OData service for an ODP provider. Additionally, you can refer to our workshop where we showcased the setup of OData services based on ODP data sources
  • The Amazon AppFlow SAP OData connector supports only secure connections; thus, you must enable a secure setup for connecting over HTTPS. Note: Amazon AppFlow supports private connectivity using an AWS PrivateLink for secure data transfers. Please see the Amazon AppFlow with AWS PrivateLink blog here for more information on how to set up a private flow for SAP OData connector.
  • You must implement SAP note 1931427 if you are running your ERP/BW application on a lower version of NetWeaver, such as 7.40 SP04 or lower.
    It is recommended to implement the following SAP notes in your local gateway or in a hub system: 2854759 , 2878969, 3062232, 3023446, 2888122.

Creating an Amazon AppFlow data flow utilizing the SAP ODP based OData service

Use the identified ODP provider and its generated OData service, and the service is either registered locally or in an SAP Gateway hub system.

Create an SAP OData connection from the Amazon AppFlow initial screen. Extend the sidebar and select Connections. Then, choose SAP OData from the connectors drop-down. Select the “Create Connection” tab and furnish the required information. Refer setup instructions for more details about the input values.

Then, select Flows from the sidebar and “Create flow”

  • Configure Flow and connect to the source by picking the SAP connection
  • Discover SAP OData Services, including the ODP based OData services
  • Select SAP Service Entity
  • Define Flow trigger (On Demand or On Schedule)
  • Map Fields, Define Validations and Set Filters
  • Run Flow

You can find further details on how to create a flow in Amazon AppFlow SAP OData Connector documentation.

Schedule flow vs On Demand flow in the context of SAP ODP based OData services

Scheduled Flow

Amazon AppFlow will initiate a fill data transfer by choosing this option, and subsequent runs at the defined frequency will run incremental transfers using the SAP ODQ mechanism. For the incremental flows, Amazon AppFlow makes use of the delta tokens provided by the ODP framework for the following delta data transfers

Note: The initial execution will also reset this entity’s previous delta queue subscriptions. Make sure that no active scheduled flows exist for the same ODP data source.

Amazon AppFlow automatically detects OData services that expose ODP-based data sources and creates a subscription (Initial Data with Delta Init) to ODQ during the initial flow run. You can also monitor the ODQ subscriptions in the SAP provider system via transaction ODQMON after a successful flow execution.

On-demand flow:

Running an on-demand flow using an ODP-enabled OData service will not create a delta queue subscription in ODQ of the SAP provider source system; instead, the data is retrieved in full.

After the successful OnDemand flow run, check transaction code ODQMON in the SAP provider source system. You will not see any active subscriptions for the OnDemand flows

Things to consider

The ODQ subscription termination does not automatically happen when you delete the SAP ODP-based OData flow in Amazon AppFlow. This leftover subscription has to be handled in the SAP provider system. Use the transaction code ODQMON to manage the subscriptions and end them if they are no longer required. You can schedule re-organization jobs in the SAP provider system to clean up the queue.

Summary

The Amazon AppFlow SAP OData Connector is a serverless managed service that extracts SAP data directly into Amazon S3 via OData. This capability paves the way for integrating SAP data into cloud-native AWS services. This launch further enhances the Amazon AppFlow SAP OData connector to use the SAP ODP framework. This feature simplifies extraction of data from multiple SAP ERP/BW data sources, including Transactional Data, Master Data, and Presentation Data, with built-in Change Data Capture capabilities.

To get started, visit the Amazon AppFlow page. To learn why AWS is the platform of choice and innovation for more than 5000 active SAP customers, visit the SAP on AWS page.