AWS for SAP

Extract data from SAP ERP and BW with Amazon AppFlow

Introduction

At AWS, we commonly hear from customers that they are looking to not only move their SAP systems to the cloud, but transform the way they use data and analytics capabilities within the organisation. Furthermore, they want to combine the SAP data with non-SAP data in a single data and analytics solution on AWS. That’s why earlier this week, in response to customer feedback, we announced that you can now extract data from SAP ERP/BW systems for use with AWS services using the Amazon AppFlow SAP OData Connector. This launch makes it super easy to set up a data flow from SAP to Amazon S3 in just a few clicks.

Many SAP customers are already leveraging AWS–based data lakes to combine SAP and non-SAP data and take advantage of AWS’s industry-leading storage and analytics capabilities. Some of them have already been doing this for several years. Examples include Bizzy, Invista, Zalando, Burberry, Visy, Delivery Hero and Engie. However, many customers have continued to request additional guidance on the best option to extract the data from their SAP systems.

In the Building data lakes with SAP on AWS blog, we outlined various data extraction patterns available for extracting data from SAP and are now offering these as hands on labs with our SAP Beyond  Infrastructure Immersion Days.

This launch expands those options, giving customers even more choices for storing and analyzing their SAP data on AWS. In this blog, I will show you to how to run data flows from SAP using the new AppFlow SAP OData Connector.

Operational Data Provisioning–based extraction

The Operational Data Provisioning (ODP) framework enables data replication capabilities between SAP applications and data targets using a provider and subscriber model. ODP supports both full data extraction as well as change data capture using operational delta queues.

Solutions including SAP Data Services and SAP Data Intelligence can be integrated with ODP, using native remote function call (RFC) libraries. Non-SAP solutions including AWS Glue, AWS Lambda or Amazon AppFlow SAP OData Connector can use the OData layer for integration via HTTP or HTTPS.

What are the key benefits of using Amazon AppFlow?

Our customers are looking for quick ways to innovate and deliver value for their customers. To help their developers focus on activities that drive business value, many want to use prebuilt connectors, without building and maintaining on their own.

Amazon AppFlow is a fully managed integration service that enables you to securely transfer data between Software-as-a-Service (SaaS) applications including Salesforce, SAP, Zendesk, Slack, and ServiceNow, and AWS services including Amazon S3 and Amazon Redshift in just a few clicks. With AppFlow, you can run data flows at enterprise scale at the frequency you choose – on a schedule, in response to a business event, or on demand. You can configure data transformation capabilities including filtering and validation to generate rich, ready-to-use data as part of the flow itself, without additional steps. AppFlow automatically encrypts data in motion, and allows users to restrict data from flowing over the public Internet for SaaS applications that are integrated with AWS PrivateLink, reducing exposure to security threats.

About the Amazon AppFlow SAP OData Connector

The Amazon AppFlow SAP OData Connector is integrated with Amazon S3 so data can be easily configured via the simple interface and extracted to your target S3 bucket(s), even in multiple file formats. Once in Amazon S3, the data can be further processed using native AWS services or third party solutions.

The Amazon AppFlow SAP OData Connector supports AWS PrivateLink which adds an extra layer of security and privacy. When the data flows between the SAP application and your target Amazon S3 bucket with AWS PrivateLink, the traffic stays on the AWS network rather than using the public internet (see further reference of Private Amazon AppFlow flows).

This image shows how AWS PrivateLink is used to keep data secure between the source application and the target S3 bucket

With the straightforward user interface in the AWS Console, you can connect and configure your data flows from SAP. All you need is to create connection and configure the flow under Amazon AppFlow.

The required configuration inputs to create new Amazon AppFlow SAP OData connection are as follows:

  • SAP Application URL
  • SAP Web Service Port
  • SAP Client
  • SAP Logon Language
  • SAP Application Catalog Service Path
  • Authentication method (Basic Auth or OAuth2)

You can find further details on how to create the SAP connection in Amazon AppFlow SAP OData Connector documentation.

The configuration steps to create new Amazon AppFlow SAP OData flow are as follows:

  1. Configure Flow
  2. Discover SAP Services
  3. Select SAP Service Entity Sets
  4. Define Flow trigger (On Demand or On Schedule)
  5. Map Fields, Define Validations and Set Filters
  6. Run Flow

Within the flow configuration you can define what SAP Service Entity you want to export using OData. Additionally, you can map the source table fields to the destination, which enables you to filter the data based on your requirements.
In last step, you simply activate the flow by specifying if you want to run it on demand or on schedule, including how often the flow will run and further details of the incremental transfer options.

This image shows the process of building your data flow with Amazon Appflow-- including connecting source/ destination, mapping source fields to destination, adding filters/ validation, and finally, activating or running the flow.

Supported file formats for writing data to Amazon S3 are as follows:

  • JSON
  • CSV
  • Parquet

You can also specify data transfer preference for aggregating all records or split them in multiple files, adding a timestamp. You can also place them in different S3 folders.

For detailed guidance, please follow the Extract Data from SAP applications to AWS services with Amazon AppFlow demo video on YouTube.

Customers running SAP Systems on-premises can also use the Amazon AppFlow SAP OData Connector by using AWS VPN or AWS Direct Connect connections to configure AWS PrivateLink, as an alternative of using public IP address of your SAP OData endpoint.

The benefits of using the SAP ODP approach include:

  • Because business logic for extractions is supported at application layer, the business context for the extracted data is fully retained.
  • All table relationships, customizations, and package configurations in the SAP application are also retained, resulting in less transformation effort.
  • Change data capture is supported using operation delta queue mechanisms. Full data load with micro batches is also supported using OData query parameters.
  • SAP Data Services and SAP Data Intelligence might have better performance in pulling the data from SAP because they have access to ODP integration using RFC layer. SAP hasn’t opened the native RFC integration capability to ODP for non-SAP applications, so AWS Glue and Lambda have to rely on HTTP-based access to OData. Conversely, this might be an advantage for certain customers who want to standardize on open integration technologies. For more information about ODP capabilities and limitations, see Operational Data Provisioning (ODP) FAQ.
  • Amazon AppFlow removes the requirement for a third-party application and thus reduce the total cost of ownership. With simple configuration and an integrated UI directly in AWS Management Console you can quickly connect your SAP system with AppFlow and extract data into Amazon S3. This comes without needs for customer development apart of the SAP OData configuration for the extraction.

Amazon AppFlow Pricing

Amazon AppFlow offers significant cost-savings advantage compared to building connectors in-house or using other application integration services. There are no upfront charges or fees to use AppFlow, and customers only pay for the number of flows they run and the volume of data processed.

You pay for every successful SAP flow run (a flow run is a call to the SAP system to transfer data to Amazon S3). Flow runs to check for new data will count towards your flow run costs, even if no new data is available in the SAP system for transfer.

Price per flow run $0.001
Maximum number of flow runs per AWS account per month 10 Million

There are additional charges for Amazon AppFlow data processing, Amazon S3 and AWS Key Management Service (if used).

You can get detailed pricing on Amazon AppFlow pricing page.

Summary

The Amazon AppFlow SAP OData Connector provides an easy and efficient way of extracting SAP data using OData directly to Amazon S3 for use with other AWS services. Customers running SAP workloads on AWS can start using this service within the AWS Management Console. Customers still running SAP systems on-premises can also feed SAP data directly to Amazon S3 and start creating powerful AWS Data Lakes to get more value from their SAP investments and enterprise data.

To get started, visit the Amazon AppFlow page. To learn why AWS is the platform of choice and innovation for more than 5000 active SAP customers, visit the SAP on AWS page.