Maintain an SAP landscape inventory with AWS Systems Manager and Amazon Athena

Introduction

Effective maintenance and operation of SAP systems rely on access to system information to support decision-making. Inquiries about, for example, SAP kernel version, installed ABAP components, or simply active SAP systems are often part of IT operation activities. Furthermore, these inquiries are typically more elaborate, for example, listing systems matching a particular version of both the SAP kernel and operating system kernel.

It is not uncommon that SAP administrators keep an inventory of systems to help in the planning of maintenance activities. Typical places to store inventory data are text files or spreadsheets. Although these data sources provide quick access to inventory data they are difficult to update and share with team members. More elaborate alternatives to keep an inventory may involve extracting data directly from the SAP database or calling SAP transactions remotely, but these are difficult to scale as the SAP landscape grows. SAP products like Solution Manager keep updated inventory data, but querying the data is rather done through a User Interface (UI) or an Application Programing Interface (API).

Third-party configuration management tools can help capture some of this data, but AWS customers are often looking for cost-effective, scalable and highly available cloud-native solutions, where no additional infrastructure or software needs to be deployed by the customer, with low implementation and maintenance efforts involved.

In this blog we will show you how to use Amazon EventBridge, AWS Systems Manager Inventory, Amazon Athena and SAP Host Agent to maintain an SAP landscape inventory that is automatically updated and can be queried using standard SQL.

Solution overview

The following diagram shows the AWS services and components used to create an SAP landscape inventory that can be queried using Amazon Athena.

We leverage the instance discovery and inventory features of SAP Host Agent to extract information from each SAP server in the landscape. Amazon EventBridge and AWS Systems Manager Run Command support the automation of calls to SAP Host Agent on a defined schedule. The automation also calls custom scripts to create inventory files in JSON format for AWS Systems Manager. The inventory JSON files are picked up by the AWS Systems Manager Agent (SSM Agent) to create an AWS Systems Manager Inventory.

AWS Systems Manager Resource Data Sync sends inventory data to an Amazon Simple Storage Service (Amazon S3) bucket. Finally, AWS Systems Manager Inventory prepares the inventory data stored in an Amazon S3 bucket and makes it available to Amazon Athena where it can be queried using standard SQL.

To demonstrate how an SAP landscape inventory is created with AWS Systems Manager we used the following systems:

• An EC2 instance running an SAP (A)SCS instance.
• An EC2 instance running SAP ERS, SAP gateway and SAP webdispatcher instances.
• An EC2 instance running SAP PAS.
• An EC2 instance running Oracle database.

For demonstration purposes the IAM instance profile for these EC2 instances includes the AWS-managed policies AmazonS3ReadOnlyAccess and AmazonSSMManagedInstanceCore. These allow the EC2 instances interact with Amazon S3 and use AWS Systems Manager service core functionality.

All systems in this SAP landscape use the Linux operating system and have the following software packages installed and configured:

• SAP Host Agent 7.2
• AWS SSM Agent version 3.1
• AWS CLI
• jq (to parse output of OS commands into JSON format)
• dos2unix (to convert plain text files in DOS/MAC format to UNIX format)

To control the scope of the data collection process, each EC2 instance has these tags:

• sap:inventory = yes
• sap:sid = <SAP SID>

Replace <SAP SID> with the corresponding SID of your SAP system.

We used a single Amazon S3 bucket to store the following:

• Shell scripts
• SSM Inventory synchronization data
• Amazon Athena query results

Before moving on to the walk through, verify that your SAP EC2 instances are integrated into AWS Systems Manager. Open AWS Systems Manager, navigate to Node Management, Fleet Manager and look for your EC2 instances. The following image shows our SAP systems being listed in AWS Systems Manager, Fleet Manager:

Walk through

Creating Scripts to Collect Custom Metrics

Create a shell script called SAPInventory.sh to call SAP Host Agent to discover running SAP instances and generate the corresponding inventory file in JSON format.

The following shell script obtains the list of running SAP instances and generates a corresponding JSON inventory file:

#!/usr/bin/sh
SHA=/usr/sap/hostctrl/exe/saphostctrl
SCNTRL=/usr/sap/hostctrl/exe/sapcontrol

# Get my EC2 ID
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
EC2ID=$(curl http://169.254.169.254/latest/meta-data/instance-id -H "X-aws-ec2-metadata-token: $TOKEN")

# Inventory file: SAP Instances
SSMINVSAPINST="/var/lib/amazon/ssm/${EC2ID}/inventory/custom/SAPInstanceList.json"

# Inventory header
echo -n -e "{\"SchemaVersion\": \"1.0\",\"TypeName\": \"Custom:SAPInstanceList\",\"Content\": [" > ${SSMINVSAPINST}

# Get list of SAP instances
SAPINSTANCELIST=$(${SHA} -function ListInstances -running 2>&1)

# Iterate through list and add to inventory file
for I in $(echo ${SAPINSTANCELIST}|sed -E -e 's/\s//g' -e 's/InstInfo:/\n/g')
do
SID=$(echo ${I}|cut -d"-" -f1)
SN=$(echo ${I}|cut -d"-" -f2)
VH=$(echo ${I}|cut -d"-" -f3)
IN=$(${SCNTRL} -nr ${SN} -function GetInstanceProperties |grep INSTANCE_NAME|awk 'BEGIN { FS = "," } ; { print $NF }'|sed -E 's/\s//g')

echo -n -e "{\"SID\": \"${SID}\",\"System Number\": \"${SN}\",\"Virtual hostname\": \"${VH}\",\"Instance Name\": \"${IN}\"}," >> ${SSMINVSAPINST}
done

# Complete the JSON file
sed -i 's/,$//' ${SSMINVSAPINST}
echo -n -e "]}" >> ${SSMINVSAPINST}

A similar approach can be used to get information about SAP kernel version, SAP instance access points, SAP instance processes and SAP ABAP components version.

This is an example of an inventory file in JSON format generated by script SAPInventory.sh:

{
   "SchemaVersion": "1.0",
   "TypeName": "Custom:SAPInstanceList",
   "Content": [
     {
      "SID": "SC3",
      "System Number":  "02",
      "Virtual hostname":  "sc3gw",
      "Instance Name":  "G02"
     },
     {
      "SID": "SC2",
      "System Number":  "01",
      "Virtual hostname":  "sc2wd",
      "Instance Name":  "W01"
     },
     {
      "SID": "SC1",
      "System Number":  "10",
      "Virtual hostname":  "sc1ers",
      "Instance Name":  "ERS10"
     }
   ]
}

Refer to the documentation about working with custom inventory for additional details about the JSON format used by AWS Systems Manager Inventory.

You could also extend the use-case and capture operating system metrics that may be relevant to your analysis. Suppose that you want to know what SAP systems currently have the most unused file system space in order to prioritize cost optimization efforts. This next sample script (FileSystems.sh) captures the relevant file system metrics. It also uses an EC2 tag value to help aggregate results per SAP System:

#!/usr/bin/sh
TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
EC2ID=$(curl http://169.254.169.254/latest/meta-data/instance-id -H "X-aws-ec2-metadata-token: $TOKEN")
REGION=$(curl http://169.254.169.254/latest/dynamic/instance-identity/document | jq .region -r)

# Inventory file: SAP Instances
SSMINVFS="/var/lib/amazon/ssm/${EC2ID}/inventory/custom/FileSystems.json"

# Inventory header
echo -n -e "{\"SchemaVersion\": \"1.0\",\"TypeName\": \"Custom:FileSystems\",\"Content\": " > ${SSMINVFS}

# Capturing a Tag Value (Ex: tag key = SAPSID)
SID=`aws ec2 describe-tags \
--region $REGION \
--filters "Name=resource-id,Values=$EC2ID" \
"Name=key,Values=SAPSID" \
| jq .Tags[0].Value | sed 's/"//g'`

# Capturing list of filesystems, appending SAP SID 
df | tr -s ' ' | sed "s/$/ $SID/" | jq -sR 'split("\n") | .[1:-1] | map(split(" ")) | map({"SID": .[6], "file_system": .[0], "total":.[1], "used": .[2], "available": .[3], "used_percent": .[4], "mounted": .[5]})' >> ${SSMINVFS}

# Complete the JSON file
echo -n -e "}" >> ${SSMINVFS}

Upload these shell scripts to an Amazon S3 bucket. In our example the scripts are stored in an AWS S3 bucket with the prefix /scripts/.

Creating AWS Systems Manager Document

Running the custom shell sripts on EC2 instances is done through an AWS Systems Manager Document.

1. Open AWS Systems Manager.
2. In the navigation bar go to Shared Resources and choose Documents.
3. Then choose Create document and choose Command or Session.
4. Provide a name for the document and leave other fields unchanged.
5. You can use the following JSON content, but replace the AWS S3 bucket name with one of your own:

{
  "schemaVersion": "2.2",
  "description": "Create SAP SSM Inventory files",
  "mainSteps": [
    {
      "inputs": {
        "timeoutSeconds": "300",
          "runCommand": [
          "mkdir -p /root/tmpscripts",
          "aws s3 cp s3://<bucket name>/scripts/SAPInventory.sh /root/tmpscripts/",
          "aws s3 cp s3://<bucket name>/scripts/FileSystems.sh /root/tmpscripts/",
          "sudo dos2unix /root/tmpscripts/* ",
          "sudo chmod 755 /root/tmpscripts/* ",
          "/root/tmpscripts/SAPInventory.sh",
          "/root/tmpscripts/FileSystems.sh",
          "rm -rf /root/tmpscripts "
        ]
      },
      "name": "runCommands",
      "action": "aws:runShellScript"
    }
  ]
}

This is how the document looks like in AWS Systems Manager, Documents:

Defining the Schedule-based Amazon EventBridge Rule

The Amazon EventBridge Rule will run the AWS Systems Manager Document periodically. The AWS Systems Manager Document, in turn, will run the data collection shell scripts.

1. Open the Amazon EventBridge in the AWS Console.
2. Select Rules, Create rule.
3. Provide a Name and Description for the rule.
4. In the Define pattern section select Schedule and type the Cron expression to invoke targets.

Use the following image as reference to create the schedule for this rule, for example every 30 minutes:

5. In the Select targets section select Systems Manager Run Command as the Target.
6. For Document select the AWS Systems Manager document you created in the previous section.
7. As Target key type tag:sap:inventory.
8. As the Target value(s) type yes.
9. Finally choose Create. The rule will be triggered according to the defined schedule.

Use the following image as reference to select the target for this rule:

Displaying Inventory Data

To look at the custom inventory data:

1. Go to AWS Systems Manager in the AWS Console
2. Navigate to Node Management, Fleet Manager.
3. From the list of Managed nodes choose one the instances where SAP inventory was collected.
4. Choose the Inventory tab.
5. Open the drop down list Inventory type and choose Custom:SAPInstanceList.

The following image shows and example of the custom inventory data for one of the EC2 instances in our SAP landscape:

Preparing the AWS Systems Manager Inventory data

Before the inventory data can be queried using Amazon Athena, a data source must be prepared. This consists of several steps, but AWS Systems Manager simplifies the process as described next.

1. Open AWS System Manager in the AWS Console.
2. Navigate to Node Management and choose Inventory.
3. Select the Detailed View tab.
4. Choose Create a resource data sync.
5. Provide a name for the data sync, the name of an Amazon S3 bucket to store the inventory data and a prefix to identify the data.

Use the following image as reference to create the Resource data sync:

6. Wait a few minutes and return to AWS Systems Manager, Inventory, Detailed View.
7. The drop-down list under Resource data syncs has the new sync.
8. Select the new sync, in this case SAP-inventory, and choose Run Advanced Queries

This will take you to Amazon Athena where the Data source and Database corresponding to AWS Systems Manager Inventory are preselected. The following image shows the table corresponding to running SAP instances (for example, custom_sapinstancelist):

Note that all the objects present in the Amazon S3 bucket at the time of the Resource data sync creation will be catalogued. This may result in a larger set of tables in addition to those of Systems Manager Inventory.

Querying the Inventory with Amazon Athena

If you are using Amazon Athena for the first time, specify an Amazon S3 bucket to store query results.

1. Choose Settings in the main screen of Amazon Athena.
2. Specify the Amazon S3 bucket (and prefix) to store query results:

To Preview the data from one of the SAP inventory tables, for example custom_sapinstancelist:

1. Click on the ellipsis menu button next to the table name.
2. Choose Preview table.
3. This will add a new tab with the corresponding SQL and results at the bottom.

The following image shows example results:

Creating Custom Athena Queries

Now that the Systems Manager Inventory is available to Amazon Athena, it is possible to run more complex queries. For example, the following query combines data from the standard AWS Systems Manager Inventory with our custom SAP inventory to get the version of the C++ standard library in our SAP systems:

SELECT a.name, a.version, a.packageid, a.publisher, b.sid, b."instance name", a.resourceid
FROM "myxferbucket-us-west-2-database"."aws_application" a, "myxferbucket-us-west-2-database"."custom_sapinstancelist" b
WHERE a.resourceid=b.resourceid
AND a.name='libstdc++';

If you also included file system statistics when you captured your AWS Systems Manager Inventory data you could now run a query like the one shown next to retrieve the top ten SAP systems with the most available space in file systems used for Oracle, DB2 or HANA data. This could reveal potential candidates for storage cost optimization, for example:

SELECT sid as "SAP System", sum(cast(available as bigint))/1024/1024 as "Available Data FS Space (GB)"
FROM   custom_filesystems 
WHERE  (mounted like '/oracle/___/sapdata%') 
OR     (mounted like '/db2/___/sapdata%') 
OR     (mounted = '/hana/data')
GROUP BY sid
ORDER BY 2 desc
LIMIT 10;

Cost

AWS services provide cost-effective solutions to respond to requirements like the ones described in this blog. The following table provides cost estimates for each service used as part of the scenarios presented in this blog. For these estimates, we assumed:

• AWS Region utilized: us-east-1 (N. Virginia)
• The SAP landscape was composed of 2000 SAP servers (EC2 instances)
• The captured metrics were queried 100 times a day, on average
• Both SAP and file system custom metrics were part of the custom Systems Manager Inventory data. In addition, all standard Systems Manager Inventory data for Linux was also included.

Cleanup

To remove the configuration of services and objects created in the walk through section, we suggest following these steps:

1. Delete the AWS Systems Manager inventory resource sync
2. Delete the AWS Glue Crawler
3. Delete the AWS Glue Database
4. Remove the Amazon EventBridge Rule
5. Remove the AWS Systems Manager document
6. Delete the Amazon Simple Storage Service (AWS S3) bucket
7. Delete the json inventory files from the OS
8. Remove the inventory definitions using AWS CLI

Conclusion

This blog presents just a few ideas on how you can leverage AWS services to enhance visibility over your SAP systems inventory. With a few configuration steps you can have Amazon EventBridge and AWS Systems Manager working together to automatically gather, store and aggregate SAP system information data. Then you can use Amazon Athena and standard SQL queries to quickly access this information. Furthermore, this can be achieved without deploying additional infrastructure.

The examples provided in this blog can be easily extended to:

• Use PowerShell commands to capture custom inventory data for Windows workloads
• Include database metrics in your custom inventory data, using a combination of shell scripting and database command line tools
• Enhance your custom inventory data by using additional components of your AWS tagging strategy to enable more advanced query scenarios

Furthermore, observability of your SAP environment, especially those where SAP HANA is a core component, can be enhanced by Amazon CloudWatch Application Insights. To setup monitoring for your SAP HANA databases today, refer to the Amazon CloudWatch Application Insights documentation for detailed tutorials on how to get started.

These ideas can be leveraged to support different aspects of your SAP-on-AWS environment. Operational support, audit and compliance, capacity planning, and cost optimization are just a few examples. We are excited to see our customers build upon these ideas. We encourage you to log on to the AWS Console and start exploring the services we discussed in this blog.

If you are looking for expert guidance and project support as you move your SAP systems to AWS, the AWS Professional Services Global SAP Specialty Practice can help. Increasingly, SAP on AWS customers—including CHS and Phillips 66—are investing in engagements with our team to accelerate their SAP transformation. Please contact our AWS Professional Services team if you would like to learn more about how we can help.

To learn why more than 5,000 active customers run SAP on AWS, visit aws.amazon.com/sap