Desktop and Application Streaming

Monitoring Amazon AppStream 2.0 with Amazon OpenSearch Service and Amazon Kinesis Data Firehose

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.

Amazon AppStream 2.0 provides scalable application streaming solutions. But the sheer number of sessions can complicate insight into each session’s information, such as logs and performance metrics. You can use the Amazon Kinesis Agent for Microsoft Windows to simplify the log and metric collection. Kinesis Agent for Windows can push logs and metrics through Amazon Kinesis Data Firehose, which supports multiple destinations.

In this post, I show you how to set up a process for a solution where Kinesis Agent for Windows pushes logs to Kinesis Data Firehose. That service then sends logs to Amazon OpenSearch Service (successor to Amazon Elasticsearch Service), where Kibana helps visualize the logs.

Overview

In this walkthrough, I use the us-east-1 Region for AppStream 2.0, Kinesis Data Firehose, and Amazon OpenSearch Service. The steps are as follows:

  1. Create an Amazon OpenSearch Service domain.
  2. Create a Kinesis Data Firehose delivery stream.
  3. Create an IAM Role for use by AppStream 2.0 instances.
  4. Create a custom image in Amazon AppStream 2.0.
  5. Create a stack and fleet in Amazon AppStream 2.0.
  6. Monitor the metrics in Kibana.

Prerequisites

To follow this walkthrough, you must have the following resources:

  • An AWS account
  • An IAM user with access to the AWS services used in this solution

Step 1: Create an Amazon OpenSearch Service domain

Create a new Amazon OpenSearch Service domain, using the following configuration. This configuration is not for production use. It takes about 10 minutes for the domain to be ready.

Step 1: Choose deployment type:

  • Deployment type: Development and testing

Step 2: Configure domain

  • Elasticsearch domain Name: The name must start with a lowercase letter and must be between 3 and 28 characters. Valid characters are a-z (lowercase only), 0-9, and – (hyphen).
  • Custom endpoint: Do not enable this and ignore this field
  • Data nodes:
    • Instance type: t2.medium.elasticsearch
    • Number of instances: 2
  • Data nodes storage:
    • Storage type: EBS
    • EBS volume type: SSD
    • EBS storage size per node: 35 (GB)
  • Dedicated master nodes: Do not enable this and ignore this field

Step 3: Configure access and security

  • Network configuration: Public access
  • Fine–grained access control: Do not enable this and ignore this field
  • SAML authentication for Kibana: Do not enable this and ignore this field
  • Amazon Cognito authentication: Do not enable this and ignore this field
  • Access policy: Domain access policy: JSON defined access policy
    • In the following sample policy for allowing access from an IP address range, replace region, account-id, domain, domain-name, and <IP Address Range> with your own information. You can visit the checkip site to check your public IP address.

{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Effect": "Allow",

      "Principal": {

        "AWS": "*"

      },

      "Action": "es:*",

      "Resource": "arn:aws:es:region:aws-account-id:domain/domain-name/*"

      "Condition": {

        "IpAddress": {

          "aws:SourceIp": [

            "<IP Address Range>"

          ]

        }

      }

    }

  ]

}

  • Encryption
    • Encryption: Enable the following options
      • Require HTTPS for all traffic to the domain
      • Node-to-node encryption
      • Enable encryption of data at rest
      • KMS master key: (Default) aws/es
  • Optional Elasticsearch cluster settings
    • Indices in request bodies: Enable Allow APIs that can span multiple indices and bypass index-specific access policies

Step 2: Create a Kinesis Data Firehose delivery stream

Now, create a Kinesis Data Firehose delivery stream, using the following configuration:

  • Delivery stream name: Example-Stream
  • Source: Direct PUT or other sources
  • Transform source records with AWS Lambda: Disabled
  • Convert record format: Disabled
  • Select destination: Amazon Elasticsearch Service
    • Domain: example-domain
    • Index: example
  • An index is analogous to a database. For example, for an easy way to access events from each of your email campaigns separately, you can use a different Kinesis Data Firehose stream and index for each campaign.
    • Index rotation: No rotation
    • Type: none (A new type will be created if the specified type name does not exist)
    • Retry duration: 60 seconds
  • S3 backup
    • Failed records only
    • Backup Amazon S3 bucket (Create a new one if there is no S3 bucket you can use for this walkthrough under your account.)
  • Buffer size: 1 MB
  • Buffer interval: 60 seconds
  • S3 compression/encryption: Disabled
  • Error logging: Enabled
  • IAM role
    • Choose to create a new IAM role. Required permissions are assigned to the IAM role automatically.

Step 3: Create an IAM role

Create a new IAM role for AppStream 2.0 streaming instances, using the following configuration:

  • IAM role name: AS2-KinesisAgent-Role
  • IAM trust relationship: See the following policy.

{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Effect": "Allow",

      "Principal": {

        "Service": [

          "appstream.amazonaws.com"

        ]

      },

      "Action": "sts:AssumeRole"

    }

  ]

}

IAM policy: See the following sample policy. Replace region, aws-account-id, and delivery-stream-name with your own information.
{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Action": [

                "firehose:PutRecord",

                "firehose:PutRecordBatch"

            ],

            "Resource": " arn:aws:firehose:region:aws-account-id:deliverystream/delivery-stream-name"

        }

    ]

}

Step 4: Create a custom image in Amazon AppStream 2.0

Create a custom AppStream 2.0 image, using the following configuration.

First, launch a new image builder called example-image using one of the base images whose name starts with AppStream-WinServer2019-<date>. Enable the internet access option in order to download and install Kinesis Agent for Windows.

Sample image builder configuration

  •  Name: example-image
  • Image: AppStream-WinServer2019-<date>
  • Instance type: stream.standard.medium
  • Network access:
    • Default Internet Access: Enable
    • VPC: vpc-12345678
    • Subnet: subnet-1234567890
    • Security group: sg-1234567890

Make sure that the subnet has a route to an internet gateway. Also, the selected security group must allow outbound TCP 443.

After the example-image image builder goes into running state, connect to the image builder and log in as administrator.

Install the latest version of Kinesis Agent for Windows via Amazon S3.

Next, configure appsettings.json for Kinesis Agent for Windows. Use the following configuration for C:\Program Files\Amazon\AWSKinesisTap\appsettings.json. For more information about appsettings.json, see Basic Configuration Structure. The settings include application and system event logs to send to the Kinesis Data Firehose delivery stream called Example-Stream.

{

    "Sources": [

        {

            "Id": "ApplicationLog",

            "SourceType": "WindowsEventLogSource",

            "LogName": "Application"

        },

        {

            "Id": "SystemLog",

            "SourceType": "WindowsEventLogSource",

            "LogName": "System"

        }

    ],

    "Sinks": [

        {

            "Id": "myKinesisFirehoseSink",

            "SinkType": "KinesisFirehose",

            "ProfileName": "appstream_machine_role",

            "Region": "us-east-1",

            "StreamName": "Example-Stream",

            "ObjectDecoration": "{appstreamVariables};APPSTREAM_SESSION_CONTEXT={env:APPSTREAM_SESSION_CONTEXT};AppStream_Image_Arn={env:AppStream_Image_Arn};AppStream_Instance_Type={env:AppStream_Instance_Type};AppStream_Resource_Type={env:AppStream_Resource_Type};AppStream_Resource_Name={env:AppStream_Resource_Name}",

            "Format": "json"

        }

    ],

    "Pipes": [

        {

            "Id": "ApplicationLogToKinesisFirehose",

            "SourceRef": "ApplicationLog",

            "SinkRef": "myKinesisFirehoseSink"

        },

        {

            "Id": "SystemLogToKinesisFirehose",

            "SourceRef": "SystemLog",

            "SinkRef": "myKinesisFirehoseSink"

        }

    ],

    "SelfUpdate": 0

}

Next, configure AppStream 2.0 session script. The system session start script reads the user variables and updates the appsettings.json file for Kinesis Agent for Windows.

Update C:\AppStream\SessionScripts\config.json with the following configuration. In the configuration, you run session start scripts under both user and system contexts.

config.json for session scripts

{
    "SessionStart": {
        "executables": [
            {
                "context": "system",
                "filename": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
                "arguments": "-NonInteractive -File C:\\Scripts\\system_session_start.ps1",
                "s3LogEnabled": true
            },
            {
                "context": "user",
                "filename": "",
                "arguments": "",
                "s3LogEnabled": true
            }
        ],
        "waitingTime": 30
    },
    "SessionTermination": {
        "executables": [
            {
                "context": "system",
                "filename": "",
                "arguments": "",
                "s3LogEnabled": true
            },
            {
                "context": "user",
                "filename": "",
                "arguments": "",
                "s3LogEnabled": true
            }
        ],
        "waitingTime": 30
    }
}

Create a directory called Scripts under C:\. Use the following PowerShell script and save it as system_session_start.ps1 under C:\Scripts\. The script retrieves the AppStream 2.0 user environment variables and uses them to update the appsettings.json file for Kinesis Agent for Windows. The environment variables are added to each log pushed to Amazon OpenSearch Service to make them searchable based on session information such as user name.

#########################################################
# Setting variables for the Kinesis Agent appsettings file path
$kinesisAgentConfigFile = "C:\Program Files\Amazon\AWSKinesisTap\appsettings.json"

#########################################################
# Log helper function
function Write-Log {
    param(
        [Parameter(Mandatory=$true)]
        $message,
        [ValidateSet('INFO','WARNING', 'ERROR')]
        $logLevel = 'INFO'
        )
        $timestamp = Get-Date -Format o
        Write-Host "[$($timestamp)] [$($logLevel)] $($message) "
    }

#########################################################
Write-Log "Starting the system session script"
Write-Log "Waiting for 5 seconds"
Start-Sleep -Seconds 5

#########################################################
# Getting AppStream user variables
New-PSDrive -PSProvider Registry -Name HKU -Root HKEY_USERS
#Check if user is connected, else sleep
if ((Get-WmiObject win32_computersystem).username) {
    $ConsoleUser = (Get-WmiObject win32_computersystem).username.split("\\")[1]
    $filterstring = "name = '" + $ConsoleUser + "'"
    $ConsoleUserSID = (Get-WmiObject win32_useraccount -Filter $filterstring).SID
    #Check if AppStream_UserName environment variable is set yet
    $waitSeconds = 0
    while($null -eq $AppStream_UserName){
        if($waitSeconds -eq 21){
            Write-Log "Waited for 20 seconds. The stdout log file is null." -logLevel 'ERROR'
            break
        }
        try{
            Get-ItemProperty -Path HKU:\$ConsoleUserSID\Environment -Name AppStream_UserName
            $AppStream_Stack_Name = (Get-ItemProperty -Path HKU:\$ConsoleUserSID\Environment -Name AppStream_Stack_Name -ErrorAction SilentlyContinue).AppStream_Stack_Name
            $AppStream_UserName = (Get-ItemProperty -Path HKU:\$ConsoleUserSID\Environment -Name AppStream_UserName -ErrorAction SilentlyContinue).AppStream_UserName
            $AppStream_Stack_Name = (Get-ItemProperty -Path HKU:\$ConsoleUserSID\Environment -Name AppStream_Stack_Name -ErrorAction SilentlyContinue).AppStream_Stack_Name
            $AppStream_User_Access_Mode = (Get-ItemProperty -Path HKU:\$ConsoleUserSID\Environment -Name AppStream_User_Access_Mode -ErrorAction SilentlyContinue).AppStream_User_Access_Mode
            $AppStream_Session_Reservation_DateTime = (Get-ItemProperty -Path HKU:\$ConsoleUserSID\Environment -Name AppStream_Session_Reservation_DateTime -ErrorAction SilentlyContinue).AppStream_Session_Reservation_DateTime
            $AppStream_Session_ID = (Get-ItemProperty -Path HKU:\$ConsoleUserSID\Environment -Name AppStream_Session_ID -ErrorAction SilentlyContinue).AppStream_Session_ID
        }catch{
            Write-Log "AppStream user environment variables are not ready." -logLevel 'WARNING'
            Start-Sleep -Seconds 1
        }
    }
}
$appstreamVariables="AppStream_Stack_Name=$($AppStream_Stack_Name);AppStream_User_Access_Mode=$($AppStream_User_Access_Mode);AppStream_Session_Reservation_DateTime=$($AppStream_Session_Reservation_DateTime);AppStream_UserName=$($AppStream_UserName);AppStream_Session_ID=$($AppStream_Session_ID)"

#########################################################
# Read the Kinesis Agent for Windows appsettings.json
Write-Log "Reading the Kinesis Agent appsettings.json"
$kinesisAgentConfig = Get-Content $kinesisAgentConfigFile
Write-Log "Original Kinesis Agent Config: $($kinesisAgentConfig)"
# Stopping the Kinesis Agent service
$processId = Get-Process -Name AWSKinesisTap -ErrorAction SilentlyContinue
if([string]::IsNullOrWhiteSpace($processId)){
    Write-Log "AWSKinesisTap is not running" -logLevel 'WARNING'
}else{
    Write-Log "AWSKinesisTap is running"
    taskkill /pid $processId.Id /F
    Write-Log "AWSKinesisTap is stopped"
}
# Update the appsettings.json file with the AppStream user environment variables
Write-Log "AppStream user variables: $($appstreamVariables)"
$updatedKinesisAgentConfig = $kinesisAgentConfig -replace("{appstreamVariables}", $appstreamVariables)
Write-Log "Updated Kinesis appsettings.json: $updatedKinesisAgentConfig"
# Write the updated appsettings.json content with the AppStream user variables.
Set-Content -Path $kinesisAgentConfigFile -Value $updatedKinesisAgentConfig
Start-Service -Name AWSKinesisTap
Write-Log "Ending the system session script."
#########################################################

After configuring the appsettings.json for Kinesis Agent for Windows, config.json for session scripts, and the system session start script, create a custom image through Image Assistant on the image builder. I used the following configuration for my example image.

Example configuration

Applications: 

App1

  • Name: Firefox
  • DisplayName: Firefox
  • Launch Path: C:\Program Files (x86)\Mozilla Firefox\firefox.exe
  • Working Directory: C:\Program Files (x86)\Mozilla Firefox

App2

  • Name: PowerShell
  • Display Name: PowerShell
  • Launch Path: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe

Image Details

  • Name: Example-Image

Step 5: Create a stack and fleet in Amazon AppStream 2.0

When the custom image is available, you create a stack and a fleet. While creating a fleet, select the custom image, Example-Image, created in Step 5. For the stack, use the new fleet created in this step.

  • Stack: Example-Stack
  • Fleet: Example-Fleet
    • Image: Example-Image
    • VPC:
      • Enable Default Internet Access. If not, make sure that the fleet has access to the internet through TCP port 443.
      • IAM role: AS2-KinesisAgent-Role

When a fleet is in running state, access the streaming session.

Step 6: Monitor data in Kibana

Kibana in Amazon OpenSearch Service makes it easy to visualize logs stored in the cluster. In this walkthrough, I used application and system logs as the source from the AppStream 2.0 fleet instances to analyze in Kibana. Follow the following steps to log in and read through application and system event logs.

  1. Log in to Kibana using the following URL:
    • https://your-elasticsearch-domain-endpoint/_plugin/kibana/.
  2. Under Management, create an index pattern called example.
  3. View logs under the Discover page.

After the index pattern is created, go to the Discover page. You can now see the logs with AppStream 2.0 variables.

Conclusion

In this post, I walked you through the steps to push application and system event logs from Amazon AppStream 2.0 to Amazon Kinesis Data Firehose and Amazon OpenSearch Service

Now, you can visualize the fleet session information in near-real time. Kinesis Agent for Windows can push system metrics and other logs. It can also deliver logs and metrics to not only Amazon OpenSearch Service, but also to other destinations such as Amazon S3, Amazon Redshift, and Splunk.