Ingesting administrative logs from Microsoft Azure to AWS CloudTrail Lake

In January 2023, AWS announced the support of ingestion for activity events from non-AWS sources using CloudTrail Lake. Making CloudTrail Lake a single location of immutable user and API activity events for auditing and security investigations.

AWS CloudTrail Lake is a managed data lake for capturing, storing, accessing, and analyzing user and API activity on AWS and non-AWS sources for audit, security, and operational purposes. You can aggregate and immutably store your activity events, and run SQL-based queries for search and analysis.

In this post, we will walk you through the process of ingesting administrative logs from Microsoft Azure into CloudTrail Lake.

Overview of solution

Figure 1: Administrative logs to AWS CloudTrail Lake – Architecture approach

Using Amazon EventBridge Scheduler to set a schedule for the execution with AWS Secrets Manager storing connection secrets for Azure Event Hub. The AWS Lambda function connects to Azure Event Hub, pulls messages from Microsoft Azure Event Hub and then sends the messages to Amazon Simple Queue Service (Amazon SQS). This Lambda function has a scheduled invocation every 60 minutes. The Amazon SQS queue invoke the second Lambda function that transforms the messages and then ingest them to CloudTrail lake.

We will use AWS Serverless Application Model (AWS SAM), and AWS Command Line Interface (AWS CLI) to deploy this solution.

Prerequisites

The following prerequisites need to be completed to deploy this solution:

1. 1. 1. Install AWS CLI
      2. Install AWS SAM CLI
      3. Configuring the AWS SAM CLI
      4. Python 3.11+
      5. An AWS Account with an AWS Identity and Access Management (IAM) role that has sufficient access to provision the required resources.
      6. A Microsoft Azure account that has sufficient access to provision the required resources, and has access to Azure Activity Logs.

Walkthrough

Step 1 – Configure Azure services for Azure activity log forwarding
Configure Azure to export activity logs to an Azure cloud event hub within your desired Azure account or organization.

1. 1. 1. Create an event hub using Azure portal.
      2. Create connection string for the event hub using shared access policy. Note the name of the namespace, instance and one of the keys (primary or secondary).
      3. Create diagnostic settings in Azure Monitor, to exports administrative logs to event hub.
      4. Create a storage account container on Azure. Note the storage account name, storage container name, and the available access keys.

You are responsible for the protections of the access keys.

Step 2 – Deploy solution on your AWS account

Run the following commands in your local host shell.

1. 1. 1. Use git to clone aws-cloudtrail-lake-ingesting-activity-logs to your workspace area.
      2. Build and deploy the solution using AWS SAM CLI

git clone https://github.com/aws-samples/aws-cloudtrail-lake-ingesting-activity-logs.git
cd aws-cloudtrail-lake-ingesting-activity-logs
sam build
sam deploy –guide

Provide the required parameter values as listed in the github readme.

After AWS SAM has successfully deployed the solution, check the outputs and note the EventDataStoreId value that’s returned. This ID will be needed to query the CloudTrail Lake event data store.

Figure 2: AWS SAM deployment Output

Step 3 – Test solution

To verify if the Azure Administrative Logs are available in AWS CloudTrail Lake data store. Use the sample query below to query your CloudTrail Lake event data store following these instructions Run a query and save query results.

Replace <event data store id> with the ID of the event data store, which can be found in the Outputs returned after a successful deployment with AWS SAM.

Sample query:

SELECT
eventData.eventSource,
eventData.eventName,
eventData.userIdentity.principalId,
eventData.sourceIpAddress,
eventData.eventTime
FROM
<event data store id>
AND eventData.eventTime < '2023-08-01 00:00:00' –- The date & time is optional.
LIMIT 10

Figure 3: To verify administrative logs in AWS CloudTrail Lake

Clean up

You can use AWS SAM CLI to delete the deployed resources and make sure that you don’t continue to incur charges. To delete the resources, run the following command from your shell and replace <stack-name> with the stack name you provided to SAM when running sam deploy. Follow the prompts to confirm the resource deletion.

sam delete --stack-name <stack-name>

Cost

This solution uses Lambda, Amazon SQS, EventBridge, and Secrets Manager. For CloudTrail Lake, you pay for data ingestion, retention and analysis. CloudTrail Lake offers two pricing options: (1) one-year extendable retention pricing and (2) seven-year retention pricing. For more information on CloudTrail costs, refer to CloudTrail pricing.

Conclusion

In this blog post, we showed you how to process access log files from external systems and store those in CloudTrail Lake. We demonstrated how CloudTrail Lake ingest and manage logs. And streamline the process of consolidating user administrative logs from Microsoft Azure.

About the author