Desktop and Application Streaming

Monitoring Amazon AppStream 2.0 with Amazon ES and Amazon Kinesis Data Firehose

Amazon AppStream 2.0 provides scalable application streaming solutions. But the sheer number of sessions can complicate insight into each session’s information, such as logs and performance metrics. You can use the Amazon Kinesis Agent for Microsoft Windows to simplify the log and metric collection. Kinesis Agent for Windows can push logs and metrics through Amazon Kinesis Data Firehose, which supports multiple destinations.

In this post, I show you how to set up a process for a solution where Kinesis Agent for Windows pushes logs to Kinesis Data Firehose. That service then sends logs to Amazon ES, where Kibana helps visualize the logs.

Architecture

Diagram for pushing logs from AppStream 2.0 using Kinesis Agent for Windows to Kinesis Data Firehose and Amazon ES

  • A client, Amazon AppStream 2.0 streaming instance, uses the Kinesis Agent for Windows to push logs to Kinesis Data Firehose.
  • Kinesis Data Firehose sends logs to Amazon ES.

Overview

In this walkthrough, I use the us-east-1 Region for AppStream 2.0, Kinesis Data Firehose, and Amazon ES. The steps are as follows:

  1. Create an Amazon ES domain.
  2. Create a Kinesis Data Firehose delivery stream.
  3. Create an IAM role.
  4. Create a custom image in Amazon AppStream 2.0.
  5. Create a stack and fleet in Amazon AppStream 2.0.
  6. Monitor the metrics in Kibana.

Prerequisites

To follow this walkthrough, you must have the following resources:

  • An AWS account
  • An IAM user with access to the AWS services used in this solution

Step 1: Create an Amazon ES domain

Create a new Amazon ES domain, using the following configuration. This configuration is not for production use. It takes about 10 minutes for the domain to be ready.

  • Name: example-domain
  • Deployment type: Development and testing
  • Elasticsearch version: 7.1
  • Instance type: t2.medium.elasticsearch
  • Number of instances: 2
  • Storage:
    • Storage type: EBS
    • EBS volume type: SSD
    • EBS storage size per node: 35 (GB)
  • Public access
  • Access policy
    • Allow access from your network location. You can visit the checkip site to check your public IP address.

Sample policy for allowing access from an IP address range:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:123456789012:domain/example-domain/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "192.16.0.0/32"
          ]
        }
      }
    }
  ]
}

Step 2: Create a Kinesis Data Firehose delivery stream

Now, create a Kinesis Data Firehose delivery stream, using the following configuration:

  • Delivery stream name: Example-Stream
  • Source: Direct PUT or other sources
  • Transform source records with AWS Lambda: Disabled
  • Convert record format: Disabled
  • Select destination: Amazon Elasticsearch Service
    • Domain: example-domain
    • Index: example
  • An index is analogous to a database. For example, for an easy way to access events from each of your email campaigns separately, you can use a different Kinesis Data Firehose stream and index for each campaign.
    • Index rotation: No rotation
    • Type: none (A new type will be created if the specified type name does not exist)
    • Retry duration: 60 seconds
  • S3 backup
    • Failed records only
    • Backup Amazon S3 bucket (Create a new one if there is no S3 bucket you can use for this walkthrough under your account.)
  • Buffer size: 1 MB
  • Buffer interval: 60 seconds
  • S3 compression/encryption: Disabled
  • Error logging: Enabled
  • IAM role
    • Choose to create a new IAM role. Required permissions are assigned to the IAM role automatically.

Step 3: Create an IAM role

Create a new IAM role for AppStream 2.0 streaming instances, using the following configuration:

  • IAM role name: AS2-KinesisAgent-Role
  • IAM trust relationship: See the following policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "appstream.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
  • IAM policy: See the following sample policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "firehose:PutRecord",
                "firehose:PutRecordBatch"
            ],
            "Resource": "arn:aws:firehose:us-east-1:123456789012:deliverystream/Example-Stream"
        }
    ]
}

Step 4: Create a custom image in Amazon AppStream 2.0

Create a custom AppStream 2.0 image, using the following configuration.

First, launch a new image builder called example-image using one of the base images whose name starts with AppStream-WinServer2019-<date>. Enable the internet access option in order to download and install Kinesis Agent for Windows.

Sample image builder configuration

  •  Name: example-image
  • Image: AppStream-WinServer2019-<date>
  • Instance type: stream.standard.medium
  • Network access:
    • Default Internet Access: Enable
    • VPC: vpc-12345678
    • Subnet: subnet-1234567890
    • Security group: sg-1234567890

Make sure that the subnet has a route to an internet gateway. Also, the selected security group must allow outbound TCP 443.

After the example-image image builder goes into running state, connect to the image builder and log in as administrator.

Install the latest version of Kinesis Agent for Windows via Amazon S3.

Next, configure appsettings.json for Kinesis Agent for Windows. Use the following configuration for C:\Program Files\Amazon\AWSKinesisTap\appsettings.json. For more information about appsettings.json, see Basic Configuration Structure. The settings include application and system event logs to send to the Kinesis Data Firehose delivery stream called Example-Stream.

{
    "Sources": [
        {
            "Id": "ApplicationLog",
            "SourceType": "WindowsEventLogSource",
            "LogName": "Application"
        },
        {
            "Id": "SystemLog",
            "SourceType": "WindowsEventLogSource",
            "LogName": "System"
        }
    ],
    "Sinks": [
        {
            "Id": "myKinesisFirehoseSink",
            "SinkType": "KinesisFirehose",
            "ProfileName": "appstream_machine_role",
            "Region": "us-east-1",
            "StreamName": "Example-Stream",
            "ObjectDecoration": "{appstreamVariables};APPSTREAM_SESSION_CONTEXT={env:APPSTREAM_SESSION_CONTEXT};AppStream_Image_Arn={env:AppStream_Image_Arn};AppStream_Instance_Type={env:AppStream_Instance_Type};AppStream_Resource_Type={env:AppStream_Resource_Type};AppStream_Resource_Name={env:AppStream_Resource_Name}",
            "Format": "json"
        }
    ],
    "Pipes": [
        {
            "Id": "ApplicationLogToKinesisFirehose",
            "SourceRef": "ApplicationLog",
            "SinkRef": "myKinesisFirehoseSink"
        },
        {
            "Id": "ApplicationLogToKinesisFirehose",
            "SourceRef": "SystemLog",
            "SinkRef": "myKinesisFirehoseSink"
        }
    ],
    "SelfUpdate": 0
}

Next, configure AppStream 2.0 session script. The system session start script reads the user variables and updates the appsettings.json file for Kinesis Agent for Windows.

Update C:\AppStream\SessionScripts\config.json with the following configuration. In the configuration, you run session start scripts under both user and system contexts.

config.json for session scripts

{
    "SessionStart": {
        "executables": [
            {
                "context": "system",
                "filename": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
                "arguments": "-NonInteractive -File C:\\Scripts\\system_session_start.ps1",
                "s3LogEnabled": true
            },
            {
                "context": "user",
                "filename": "",
                "arguments": "",
                "s3LogEnabled": true
            }
        ],
        "waitingTime": 30
    },
    "SessionTermination": {
        "executables": [
            {
                "context": "system",
                "filename": "",
                "arguments": "",
                "s3LogEnabled": true
            },
            {
                "context": "user",
                "filename": "",
                "arguments": "",
                "s3LogEnabled": true
            }
        ],
        "waitingTime": 30
    }
}

Create a directory called Scripts under C:\. Use the following PowerShell script and save it as system_session_start.ps1 under C:\Scripts\. The script retrieves the AppStream 2.0 user environment variables and uses them to update the appsettings.json file for Kinesis Agent for Windows. The environment variables are added to each log pushed to Amazon ES to make them searchable based on session information such as user name.

#########################################################
# Setting variables for the Kinesis Agent appsettings file path
$kinesisAgentConfigFile = "C:\Program Files\Amazon\AWSKinesisTap\appsettings.json"

#########################################################
# Log helper function
function Write-Log {
    param(
        [Parameter(Mandatory=$true)]
        $message,
        [ValidateSet('INFO','WARNING', 'ERROR')]
        $logLevel = 'INFO'
        )
        $timestamp = Get-Date -Format o
        Write-Host "[$($timestamp)] [$($logLevel)] $($message) "
    }

#########################################################
Write-Log "Starting the system session script"
Write-Log "Waiting for 5 seconds"
Start-Sleep -Seconds 5

#########################################################
# Getting AppStream user variables
New-PSDrive -PSProvider Registry -Name HKU -Root HKEY_USERS
#Check if user is connected, else sleep
if ((Get-WmiObject win32_computersystem).username) {
    $ConsoleUser = (Get-WmiObject win32_computersystem).username.split("\\")[1]
    $filterstring = "name = '" + $ConsoleUser + "'"
    $ConsoleUserSID = (Get-WmiObject win32_useraccount -Filter $filterstring).SID
    #Check if AppStream_UserName environment variable is set yet
    $waitSeconds = 0
    while($null -eq $AppStream_UserName){
        if($waitSeconds -eq 21){
            Write-Log "Waited for 20 seconds. The stdout log file is null." -logLevel 'ERROR'
            break
        }
        try{
            Get-ItemProperty -Path HKU:\$ConsoleUserSID\Environment -Name AppStream_UserName
            $AppStream_Stack_Name = (Get-ItemProperty -Path HKU:\$ConsoleUserSID\Environment -Name AppStream_Stack_Name -ErrorAction SilentlyContinue).AppStream_Stack_Name
            $AppStream_UserName = (Get-ItemProperty -Path HKU:\$ConsoleUserSID\Environment -Name AppStream_UserName -ErrorAction SilentlyContinue).AppStream_UserName
            $AppStream_Stack_Name = (Get-ItemProperty -Path HKU:\$ConsoleUserSID\Environment -Name AppStream_Stack_Name -ErrorAction SilentlyContinue).AppStream_Stack_Name
            $AppStream_User_Access_Mode = (Get-ItemProperty -Path HKU:\$ConsoleUserSID\Environment -Name AppStream_User_Access_Mode -ErrorAction SilentlyContinue).AppStream_User_Access_Mode
            $AppStream_Session_Reservation_DateTime = (Get-ItemProperty -Path HKU:\$ConsoleUserSID\Environment -Name AppStream_Session_Reservation_DateTime -ErrorAction SilentlyContinue).AppStream_Session_Reservation_DateTime
            $AppStream_Session_ID = (Get-ItemProperty -Path HKU:\$ConsoleUserSID\Environment -Name AppStream_Session_ID -ErrorAction SilentlyContinue).AppStream_Session_ID
        }catch{
            Write-Log "AppStream user environment variables are not ready." -logLevel 'WARNING'
            Start-Sleep -Seconds 1
        }
    }
}
$appstreamVariables="AppStream_Stack_Name=$($AppStream_Stack_Name);AppStream_User_Access_Mode=$($AppStream_User_Access_Mode);AppStream_Session_Reservation_DateTime=$($AppStream_Session_Reservation_DateTime);AppStream_UserName=$($AppStream_UserName);AppStream_Session_ID=$($AppStream_Session_ID)"

#########################################################
# Read the Kinesis Agent for Windows appsettings.json
Write-Log "Reading the Kinesis Agent appsettings.json"
$kinesisAgentConfig = Get-Content $kinesisAgentConfigFile
Write-Log "Original Kinesis Agent Config: $($kinesisAgentConfig)"
# Stopping the Kinesis Agent service
$processId = Get-Process -Name AWSKinesisTap -ErrorAction SilentlyContinue
if([string]::IsNullOrWhiteSpace($processId)){
    Write-Log "AWSKinesisTap is not running" -logLevel 'WARNING'
}else{
    Write-Log "AWSKinesisTap is running"
    taskkill /pid $processId.Id /F
    Write-Log "AWSKinesisTap is stopped"
}
# Update the appsettings.json file with the AppStream user environment variables
Write-Log "AppStream user variables: $($appstreamVariables)"
$updatedKinesisAgentConfig = $kinesisAgentConfig -replace("{appstreamVariables}", $appstreamVariables)
Write-Log "Updated Kinesis appsettings.json: $updatedKinesisAgentConfig"
# Write the updated appsettings.json content with the AppStream user variables.
Set-Content -Path $kinesisAgentConfigFile -Value $updatedKinesisAgentConfig
Start-Service -Name AWSKinesisTap
Write-Log "Ending the system session script."
#########################################################

After configuring the appsettings.json for Kinesis Agent for Windows, config.json for session scripts, and the system session start script, create a custom image through Image Assistant on the image builder. I used the following configuration for my example image.

Example configuration

Applications: 

App1

  • Name: Firefox
  • DisplayName: Firefox
  • Launch Path: C:\Program Files (x86)\Mozilla Firefox\firefox.exe
  • Working Directory: C:\Program Files (x86)\Mozilla Firefox

App2

  • Name: PowerShell
  • Display Name: PowerShell
  • Launch Path: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe

Image Details

  • Name: Example-Image

Step 5: Create a stack and fleet in Amazon AppStream 2.0

When the custom image is available, you create a stack and a fleet. While creating a fleet, select the custom image, Example-Image, created in Step 5. For the stack, use the new fleet created in this step.

  • Stack: Example-Stack
  • Fleet: Example-Fleet
    • Image: Example-Image
    • VPC:
      • Enable Default Internet Access. If not, make sure that the fleet has access to the internet through TCP port 443.
      • IAM role: AS2-KinesisAgent-Role

When a fleet is in running state, access the streaming session.

Step 6: Monitor data in Kibana

Kibana in Amazon ES makes it easy to visualize logs stored in the cluster. In this walkthrough, I used application and system logs as the source from the AppStream 2.0 fleet instances to analyze in Kibana. Follow the following steps to log in and read through application and system event logs.

  1. Log in to Kibana using the following URL:
    • https://your-elasticsearch-domain-endpoint/_plugin/kibana/.
  2. Under Management, create an index pattern called example.
  3. View logs under the Discover page.

After the index pattern is created, go to the Discover page. You can now see the logs with AppStream 2.0 variables.

Conclusion

In this post, I walked you through the steps to push application and system event logs from Amazon AppStream 2.0 to Amazon Kinesis Data Firehose and Amazon ES.

Now, you can visualize the fleet session information in near-real time. Kinesis Agent for Windows can push system metrics and other logs. It can also deliver logs and metrics to not only Amazon ES, but also to other destinations such as Amazon S3, Amazon Redshift, and Splunk.