AWS Blog

New Amazon AppStream 2.0 Features – Fleet Auto Scaling, Image Builder, SAML, Metrics, and Fleet Management

by Jeff Barr | on | in Amazon AppStream | | Comments

My colleague Gene Farrell introduced you to Amazon AppStream 2.0 late last year. In his guest post, Gene explained how AppStream 2.0 lets you run desktop applications securely on any device, from within the comfort of an HTML5 web browser (read the entire post to learn more). For example, I used the AppStream 2.0 Try it Now page to launch and then immediately start using Siemens Solid Edge. I simply chose the desired application from the Try it Now page:

I was running Solid Edge a few seconds later, no installation or setup needed:

By Popular Request – New Features for Enterprises, SMBs, and ISVs
Since that re:Invent launch, we have been fine-tuning AppStream 2.0, adding in some features that our customers have been asking for. These features will allow our customers to more easily deploy, access, manage and track the applications that they make available for use through AppStream. We’ve rolled most of these out without individual blog posts, and today I’d like to let you know what we’ve been up to. Here are the newest features:

Fleet Auto Scaling – This brand-new feature allows you to use the CloudWatch metrics to scale your fleet up and down in response to changes in demand. This allows you to deliver applications as economically as possible, while still providing instant access.

Image Builder – You can build your own AppStream 2.0 images that contain your choice of applications.

SAML 2.0 Authentication – You can use your existing SAML 2.0 compliant directory with AppStream 2.0. Your users can use their existing credentials to log in.

Fleet Management – You have additional management options for the instances that run your applications.

CloudWatch Metrics – You can observe and monitor seven Amazon CloudWatch metrics, including the size and overall utilization of your fleets.

Let’s take a look at each one!

Fleet Auto Scaling
This feature is brand new, and is powered by the new CloudWatch metrics! You can now associate scaling policies with each of your fleets and use them to meet varying levels of user demand and to control costs. If you are using AppStream 2.0 to deliver productivity applications to your users, you can use the scaling policies to ensure that capacity comes online as needed during office hours, and goes away in the evening when your users are done for the day. Here is a fleet with scale out (add capacity) and scale in (remove capacity) policies:

In order to take advantage of this feature, you set the minimum and maximum capacity when you create the fleet:

This will create the default policies, which you can later edit, add, or remove (you can have up to 50 policies per fleet). To learn more, read about AppStream Fleet Auto Scaling.

Image Builder
This feature allows you to create custom images that contain your choice of commercial or proprietary applications. In order to do this, you launch an instance called an image builder. Then you log in to the instance, install and configure the applications as desired, and capture the state of the instance as an image. The entire login and customization process takes place within your web browser; you don’t have to download any keys or remember any passwords. The application appears in the Image Registry and is available to your users.

I can launch an image builder from the AppStream 2.0 Console:

Next, I choose the starting point (an existing image):

Then I configure the builder by giving it a name, choosing an instance size, and setting up the VPC:

I click on Review, confirm my settings, and then wait for the builder to launch:

Then I can connect to the image builder, set up the apps, and create an image. I have my choice of two identities when I connect, Admin and Test:

I select ImageBuildAdmin and (when prompted for a password), click on Log me in in the Admin Commands menu:

After logging in, I launch the Image Assistant app and use it to install and test my apps:

To learn more, read about Image Builders and follow the Using an AppStream 2.0 Image Builder Tutorial.

SAML 2.0 Authentication
This feature allows you to use any external identity provider that supports SAML 2.0 including Active Directory Federation Services, PingFederate Server, Okta, or Shibboleth:

After you follow the directions in Setting Up SAML, your users can log in to AppStream 2.0 using their existing identity and credentials. You can manage users and groups, control access to applications based on the identity or location of the user, and use Multi-Factor Authentication (MFA). To learn more, read Enabling Single Sign-on Access to AppStream 2.0 Using SAML 2.0. If you have already set up federated access to the AWS Management Console, much of what you already know will apply.

Fleet Management
This feature gives me additional control over my fleets (groups of instances that are running applications for users). I can see all of my fleets on a single screen:

I can select a fleet and then act on it:

Some properties of a fleet can be edited at any time. Others, including the VPC properties, can only be edited after the fleet has been stopped. To learn more, read about Stacks and Fleets.

CloudWatch Metrics
AppStream publishes eight metrics to CloudWatch for each fleet:

  • RunningCapacity – Number of instances running.
  • InUseCapacity – Number of instances in use.
  • DesiredCapacity – Number of instances that are either running or pending.
  • AvailableCapacity – Number of idle instances available for use.
  • PendingCapacity – Number of instances being provisioned.
  • CapacityUtilization – Percentage of fleet being used.
  • InsufficientCapacityError – Number of sessions rejected due to lack of capacity.

You can see these metrics from within the AppStream 2.0 Console:

These metrics will help you to measure overall usage to to fine tune the size of your fleet. As is the case with every CloudWatch metric, you can generate alerts and raise alarms when a metric is outside of the desired range. You could also use AWS Lambda functions to make changes to your environment or to generate specialized notifications. To learn more, read about Monitoring Amazon AppStream 2.0 Resources.

Available Now
All of these features are available now and you can start using them today!

Jeff;

EC2 Run Command is Now a CloudWatch Events Target

by Jeff Barr | on | in Amazon CloudWatch, EC2 Systems Manager | | Comments

Ok, time for another peanut butter and chocolate post! Let’s combine EC2 Run Command (New EC2 Run Command – Remote Instance Management at Scale) and CloudWatch Events (New CloudWatch Events – Track and Respond to Changes to Your AWS Resources) and see what we get.

EC2 Run Command is part of EC2 Systems Manager. It allows you to operate on collections of EC2 instances and on-premises servers reliably and at scale, in a controlled and selective fashion. You can run scripts, install software, collect metrics and log files, manage patches, and much more, on both Windows and Linux.

CloudWatch Events gives you the ability to track changes to AWS resources in near real-time. You get a stream of system events that you can easily route to one or more targets including AWS Lambda functions, Amazon Kinesis streams, Amazon SNS topics, and built-in EC2 and EBS targets.

Better Together
Today we are bringing these two services together. You can now create CloudWatch Events rules that use EC2 Run Command to perform actions on EC2 instances or on-premises servers. This opens the door to all sorts of interesting ideas; here are a few that I came up with:

Final Log Collection – Collect application or system logs from instances that are being shut down (either manually or as a result of a scale-in operation initiated by Auto Scaling).

Error Log Condition – Collect logs after an application crash or a security incident.

Instance Setup – After an instance has started, download & install applications, set parameters and configurations, and launch processes.

Configuration Updates – When a config file is changed in S3, install it on applicable instances (perhaps determined by tags). For example, you could install an updated Apache web server config file on a set of properly tagged instances, and then restart the server so that it picks up the changes. Or, update an instance-level firewall each time the AWS IP Address Ranges are updated.

EBS Snapshot Testing and Tracking – After a fresh snapshot has been created, mount it on a test instance, check the filesystem for errors, and then index the files in the snapshot.

Instance Coordination – Every time an instance is launched or terminated, inform the others so that they can update internal tracking information or rebalance their workloads.

I’m sure that you have some more interesting ideas; please feel free to share them in the comments.

Time for Action!
Let’s set this up. Suppose I want to run a specific Linux shell command every time Auto Scaling adds another instance to an Auto Scaling Group.

I start by opening the CloudWatch Events Console and clicking on Create rule:

I configure my Event Source to be my Auto Scaling Group (AS-Main-1), and indicate that I want to take action when EC2 instances are launched successfully:

Then I set up the target. I choose SSM Run Command, pick the AWS-RunShellScript document, and indicate that I want the command to be run on the instances that are tagged as coming from my Auto Scaling group:

Then I click on Configure details, give my rule a name and a description, and click on Create rule:

With everything set up, the command service httpd start will be run on each instance launched as a result of a scale out operation.

Available Now
This new feature is available now and you can start using it today.

Jeff;

 

New – Amazon EMR Instance Fleets

by Randall Hunt | on | in Amazon EMR, Launch | | Comments

Today we’re excited to introduce a new feature for Amazon EMR clusters called instance fleets. Instance fleets gives you a wider variety of options and intelligence around instance provisioning. You can now provide a list of up to 5 instance types with corresponding weighted capacities and spot bid prices (including spot blocks)! EMR will automatically provision On-Demand and Spot capacity across these instance types when creating your cluster. This can make it easier and more cost effective to quickly obtain and maintain your desired capacity for your clusters.

You can also specify a list of Availability Zones and EMR will optimally launch your cluster in one of the AZs. EMR will also continue to rebalance your cluster in the case of Spot instance interruptions by replacing instances with any of the available types in your fleet. This will make it easier to maintain your cluster’s overall capacity. Instance fleets can be used instead of instance groups. Just like groups your cluster will have master, core, and task fleets.

Let’s take a look at the console updates to get an idea of how these fleets work.

We’ll start by navigating to the EMR console and clicking the Create Cluster button. That should bring us to our familiar EMR provisioning console where we can navigate to the advanced options near the top left.

We’ll select the latest EMR version (instance fleets are available for EMR versions 4.8.0 and greater, with the exception of 5.0.x) and click next.

Now we get to the good stuff! We’ll select the new instance fleet option in the hardware options.
Screenshot 2017-03-09 00.30.51

Now what I want to do is modify our core group to have a couple of instance types that will satisfy the needs of my cluster.

CoreFleetScreenshot

EMR will provision capacity in each instance fleet and availability zone to meet my requirements in the most cost effective way possible. The EMR console provides an easy mapping of vCPU to weighted capacity for each instance type, making it easy to use vCPU as the capacity unit (I want 16 total vCPUs in my core fleet). If the vCPU units don’t match my criteria for weighting instance types I can change the “Target capacity” selector to include arbitrary units and define my own weights (this is how the API/CLI consume capacity units as well).

When the cluster is being provisioned if it’s unable to obtain the desired spot capacity within a user defined timeout you can have it terminate or fall back onto On-Demand instances to provision the rest of the capacity.

All this functionality for instance fleets is also available from the AWS SDKs and the CLI. Let’s take a look at how we would provision our own instance fleet.

First we’ll create our configuration json in my-fleet-config.json:

[
  {
    "Name": "MasterFleet",
    "InstanceFleetType": "MASTER",
    "TargetOnDemandCapacity": 1,
    "InstanceTypeConfigs": [{"InstanceType": "m3.xlarge"}]
  },
  {
    "Name": "CoreFleet",
    "InstanceFleetType": "CORE",
    "TargetSpotCapacity": 11,
    "TargetOnDemandCapacity": 11,
    "LaunchSpecifications": {
      "SpotSpecification": {
        "TimeoutDurationMinutes": 20,
        "TimeoutAction": "SWITCH_TO_ON_DEMAND"
      }
    },
    "InstanceTypeConfigs": [
      {
        "InstanceType": "r4.xlarge",
        "BidPriceAsPercentageOfOnDemandPrice": 50,
        "WeightedCapacity": 1
      },
      {
        "InstanceType": "r4.2xlarge",
        "BidPriceAsPercentageOfOnDemandPrice": 50,
        "WeightedCapacity": 2
      },
      {
        "InstanceType": "r4.4xlarge",
        "BidPriceAsPercentageOfOnDemandPrice": 50,
        "WeightedCapacity": 4
      }
    ]
  }
]

Now that we have our configuration we can use the AWS CLI’s ’emr’ subcommand to create a new cluster with that configuration:

aws emr create-cluster --release-label emr-5.4.0 \
--applications Name=Spark,Name=Hive,Name=Zeppelin \
--service-role EMR_DefaultRole \
--ec2-attributes InstanceProfile="EMR_EC2_DefaultRole,SubnetIds=[subnet-1143da3c,subnet-2e27c012]" \
--instance-fleets file://my-fleet-config.json

If you’re eager to get started the feature is available now at no additional cost and you can find detailed documentation to help you get started here.

Thanks to the EMR service team for their help writing this post!

Randall Hunt

Launch: Amazon ElastiCache Launches Enhanced Redis Backup and Restore with Cluster Resizing

by Tara Walker | on | in Amazon ElastiCache, Launch | | Comments

Most of us equate in-memory caching with improved performance and lower cost at scale when designing applications or building solutions. Now if there was only a service that would continually make it simpler to deploy and utilize in-memory cache in the cloud while increasing the ability to scale.

Okay no more joking around, the cloud service that provides this great functionality is, of course, Amazon ElastiCache. Amazon ElastiCache is an AWS managed service that provides a performant in-memory data store or cache in the cloud while offering a straightforward way to create, scale, and manage a distributed environment for low-latency, secure, access of I/O intensive or compute heavy data. Additionally, ElastiCache reduces the overhead of managing infrastructure for your in-memory data structure server or cache by detecting and replacing failed nodes while providing enhanced visibility into key performance metrics of the caching system nodes via Amazon CloudWatch. This exciting service is now launching support for Enhanced Redis Backup and Restore with Cluster Resizing.

For those of you familiar with Amazon ElastiCache, you are likely aware that ElastiCache currently supports two in-memory key-value engines:

  • Memcached: an open source, high-performing, distributed memory object caching system developed in 2003 with the initial goal of speeding up dynamic web applications by alleviating database load
  • Redis: an open source in-memory data structure store launched in 2009 developed as a broker for caching, messaging, and databases with built-in replication, atomic operation support, various levels of on-disk persistence, and high availability via Redis Cluster.

In October of 2016, support was added for Redis Cluster with Redis 3.2.4. This allowed ElastiCache Redis users to, not only take advantage of Redis clusters, but also gave users the ability to:

  • Create cluster-level backups.
  • Produce snapshots of each of the cluster’s shards contained within backups.
  • Scale their workloads with 3.5TiB of data across up to 15 shards.

You can read more about using Redis with ElastiCache and the related features by reviewing the product page for Amazon ElastiCache for Redis.

With the launch of the Enhanced Backup and Restore with Cluster Resizing feature, ElastiCache is providing even deeper support for Redis with a clear-cut migration path to a managed Redis Cluster experience. There are several benefits of this enhancement for ElastiCache and Redis users alike, such as:

  • Ability to restore backup into a Redis Cluster with a different number of shards and slot distribution
  • Deliver the capability for users to resize Redis workloads
  • Allow Redis database file (RDB) snapshots as input for creating a sharded Redis Cluster
  • Offer option to use snapshot(s) of Redis on EC2 implementations (both Redis Cluster and single-shard Redis) as data input for sharded Redis Cluster creation

To accomplish these tasks, ElastiCache will parse the Redis key space across the backup’s individual snapshots, and redistribute the keys in the new Cluster according to the requested number of shards and hash slots. You would simply take your RDB snapshots and store them on S3, then provide ElastiCache with the desired number of shards and the snapshot file. ElastiCache handles the heavy lifting of restoring the Redis data store into a Redis cluster.

I am sure that you all may be thinking; Is it really that easy to leverage the Enhanced Redis Backup and Restore with Cluster Resizing feature in ElastiCache? Well, there is no time like the present to find out. Let’s take a trip to the AWS Management Console, and put this newly launched enhancement in action by restoring an external RDB snapshot to a new cluster using ElastiCache.

My first stop in the AWS Management console is to the Amazon S3 console. I have some Redis .rdb snapshot files I received from some of my peers here at AWS in order to test the restore of an external Redis snapshot to ElastiCache. I will need to put these files into Amazon S3 so that I can access the snapshots as input for my ElastiCache Redis cluster.

In the S3 console, I will go to my S3 bucket, aws-blog-tew-posts, that I created for testing and development purposes. I’ll upload the .rdb snapshot files that were provided to me into this S3 bucket.

 

It is important to note that the name of your S3 bucket must conform to DNS standards. To be DNS-compliant, the name must be at least three characters, must contain only lowercase letters, numbers, and/or dashes, and it must start and end with a lowercase letter or number. While this may be obvious, I will also note that the bucket name cannot be in an IP address format. You can learn more about the S3 Bucket Restrictions at the link provided here.

With my .rdb files successfully uploaded into my aws-blog-tew-posts bucket, I need to take note of the S3 path to these backup files. For these files, the path would be aws-blog-tew-posts/dump_1.rdb or aws-blog-tew-posts/dump_10.rdb. If you have placed your files into a folder, the folder name would need to be included in this path, i.e. thebucketname/thefoldername/thefilename.

For ElastiCache to access these files, I need to ensure that the service has read permissions for each of the files. To provide access, I will update the permissions for each of .rdb files by assigning the Grantee as the canonical id for my region and grant the user Open/Download permissions. The canonical id for all regions, outside of China (Beijing) and AWS GovCloud (US), is as follows:

540804c33a284a299d2547575ce1010f2312ef3da9b3a053c8bc45bf233e4353

After I click the Save button, I am all set to use these files as input for an ElastiCache Redis cluster.

The next step is to go to the ElastiCache console. Here I will create a new ElastiCache Redis cluster and seed this new cluster with data from one of the RDB snapshots located in the files in my S3 bucket. I’ll choose the dump_1.rdb snapshot file to use as my data input to seed this new cluster. Since I want to explore the ElastiCache Redis capabilities added on this past October with 3.2.4 support of Redis Cluster, as well as, discuss the new Backup and Restore with Cluster Resizing enhancements, I’ll create a new Redis Cluster and ensure I have cluster mode enabled. At this point, I should note that you cannot restore from a backup created using a Redis (cluster mode enabled) cluster to a Redis (cluster mode disabled) cluster.

First, I will click the Get Started Now button from the ElastiCache console dashboard or the Create button based upon your console view.

In the Create your Amazon ElastiCache cluster dialog window, I’ll select Redis for my caching and make sure I click the checkbox for Cluster Mode enabled (Scale Out). The name of my new cluster will be, tew-rediscluster and I since I am enabling a Cluster mode, my ElastiCache Redis Engine version is 3.2.4. For this cluster, I will keep the default Redis port of 6379.

The key benefit of the ElastiCache enhanced Redis Backup and Restore feature is the cluster resizing capability that allows me to build a new cluster with a different number of shards than was originally used for the backup file. To build the new Redis Cluster, I am using only one RDB snapshot file, dump_1.rdb which is a small Redis instance backup with only one shard. However, in the creation of my new tew-rediscluster, I have opted for 3 shards with 2 replicas per shard.

In addition, I have the ability to specify a node type for my new cluster that is a different size than my original instance from the RDB snapshot. As I mentioned, the dump_1.rdb is a backup of a Redis instance that is significantly smaller than the size of the chosen node type for my tew-rediscluster shown below.

There are other options and data input needed in order to complete the creation of my ElastiCache Redis cluster that I will not show in this blog post. However, if you want to go through each of the steps necessary for creating an ElastiCache Redis cluster you can find more information in the AWS ElastiCache Getting Started documentation for Launch a Cluster.

Once I have provided all the information needed to create my ElastiCache Redis cluster, I will need to tell ElastiCache how to seed the cluster with the .rdb file by providing the file location from my S3 bucket. In the Import Data to Cluster section of the create dialog, I will enter the S3 path to my dump_1.rdb in the Seed RDB file S3 location textbox. Remember, the nomenclature for the S3 file path is Bucket/Folder/ObjectName so I will enter aws-blog-tew-posts/dump_1.rdb as the path to the RDB file in S3. All that is left now is to click the Create button.

 

That’s it! ElastiCache goes to work to creating the new Redis cluster. After a short time period, the ElastiCache console shows my new Amazon ElastiCache Redis cluster as available and I have successfully created this cluster with data restored from an external RDB snapshot file.

 

I just demonstrated how you have the capability to create an ElastiCache Redis cluster using an external RDB snapshot, but of course, you can create backups and restore from backups from your existing ElastiCache Redis clusters as well. To dig deeper into information about this newly launched feature, visit Restoring From a Backup with Cluster Resizing in the Amazon ElastiCache User Guide.

To learn more about making your applications more performant with Amazon ElastiCache, visit the AWS Amazon ElastiCache page for product details, resources, and customer testimonials.

– Tara

S3 Storage Management Update – Analytics, Object Tagging, Inventory, and Metrics

by Jeff Barr | on | in Amazon S3 | | Comments

Today I would like to tell you about four S3 features that will give you detailed insights into your storage and your access patterns. You can see what and how much you are storing, how it is being used, and you can make informed decisions about the use of S3 storage classes as a result. These features will be of value to everyone who uses S3, whether they have tens, thousands, millions, or billions of objects in their buckets. Here’s an overview:

S3 Analytics – You can analyze the storage and retrieval patterns for your objects and use the results to choose the most appropriate storage class. You can inspect the results of the analysis from within the S3 Console, or you can load them into your favorite BI tool and dive deep. Either way, you now have the means to gain a deep understanding of your storage patterns and to see how they relate to usage and growth.

S3 Object Tagging – You can associate multiple key-value pairs (tags) with each of your S3 objects, with ability to change them at any time. The tags can be used to manage and control access, set up S3 Lifecycle policies, customize the S3 Analytics, and filter the CloudWatch metrics. You can think of the bucket as a data lake, and use tags to create a taxonomy of the objects within the lake. This is more flexible than using the bucket and a prefix, and allows you to make semantic-style changes without renaming, moving, or copying objects.

S3 Inventory – You can speed up your business workflows and your big data jobs using S3 Inventory. This feature provides you with a CSV-formatted flat-file representation of the contents of all or part (as identified by a prefix) of a bucket, on a daily or weekly basis.

S3 CloudWatch Metrics – You can improve the performance of your S3-powered applications by monitoring and alarming on 13 new Amazon CloudWatch metrics.

Let’s dive in…

S3 Analytics
As an S3 user, you have your choice of three storage classes (Standard, Standard – Infrequent Access, and Glacier) and the ability to use S3’s Object Lifecycle Management feature to indicate when objects should be either expired or transitioned to Standard – Infrequent Access or Glacier.

The S3 Analytics feature gives you the information that you need to have in order to choose between Standard and Standard – Infrequent Access for your objects. Because many customers store several different types or categories of information in a single bucket for ease of processing, you have the ability to run the analytics on subsets (defined by a common prefix or tag value) of the objects in a bucket. Each subset is defined by a filter; each bucket can have up to 1000 filters. Here are the filters on my jbarr-public bucket:

Here’s how I define a simple filter that analyzes objects that are prefixed with www:

I could choose to filter by tag name and value instead. Here’s how I would analyze objects that have a tag named type with the value page:

Each filter can have at most one prefix, along with any desired number of tags. I can also choose to have the analytics exported to a file each day:

Once one or more filters are in place, the analytics are run on a daily basis and I can view them by simply clicking on the filter. For example, here’s what I see when I click on my Archival filter:

There’s a lot of good information in this analysis! I can see that:

  • Looking back 127 days, most of my objects that are older than 30 days are accessed infrequently. I can save money by creating a Lifecycle rule that will transition these objects to Standard – Infrequent Access storage.
  • At this point, the majority (6.39 PB) of my storage is in Standard storage; just a tiny fraction (3.24 GB) is in Standard – Infrequent Access storage (I don’t actually have that much data in my bucket! The S3 team was kind enough to load up my account with some test metrics that were, shall we say, very generous).
  • Over the 127 day observation period, between 16% and 30% of the Standard storage was retrieved on a per-day basis.
  • Objects between 30-45 days old, 45-60 days old, 60-90 days old, 90-180 days old, and over 180 days old are all infrequently accessed and good candidates for Standard – Infrequent Access storage.

You can set up storage class analysis from the Console, the CLI, or through the S3 APIs.

To learn more, read S3 Storage Class Analysis.

S3 Object Tagging
Tags supplement the existing location-based S3 object management model (buckets and prefixes) with the ability to manage storage based on what the object represents. For example, you can tag objects with the name of a department and then construct IAM policies that grant access based on the tag. This gives you a lot of flexibility , including the ability to effect changes by simply altering tags.

You can create, update, or delete them during any part of the object’s life cycle. Tags can be referenced in IAM policies, S3 Lifecycle policies, and in the Storage Analytics filters that I just showed you. Each object can have up to ten tags and each version of an object has a distinct set of tags. You can use tags to manage the expiration of objects via a lifecycle policy and you can set up CloudWatch metrics that reflect the activity around a particular tag.

The Properties page for each object displays the current set of tags and allows me to edit them:

You can also set up and access tags from the CLI or through the S3 APIs. When you use either of these two methods, you must always work in terms of the full tag set. For example, if an object has four tags and you want to add a fifth one, you must read the current set, add the new one, and then update the entire set.

Tags can be replicated across regions via Cross-Region Replication, but your IAM policy on the source side must enable the s3:GetObjectVersionTagging and s3:ReplicateTags actions.

Tags cost $0.01 per 10,000 tags per month. Requests that add or update tags (PUT and GET, respectively) are charged at the usual rates.

To learn more, read about S3 Object Tagging.

S3 Inventory
S3’s LIST operation is synchronous, and returns up to 1000 objects at a time, along with a key that can be used to initiate a second LIST that picks up where the first one leaves off. While this works well for small to medium-sized buckets and single-threaded programs, it can be challenging to use in conjunction with huge buckets or multiple threads.

The S3 team told me that many of our customers store tens or hundreds of millions of objects in a single bucket, and that buckets with a billion or more objects are not unusual. These buckets are often processed daily or weekly as part of a larger workflow and can benefit from a more direct way to gain access to the list of objects in a bucket.

With S3 Inventory, you can now arrange to receive daily or weekly inventory reports for any of your buckets. You can use a prefix to filter the report and you can choose to include optional fields such as size, storage class, and replication status. Reports can be sent to an S3 bucket in your account or (with proper permission settings) in another account.

Here’s how I request a daily inventory report called WebStuff, for all objects that start with www:

I also need to give S3 permission to write to the destination bucket (jbarr-s3-inventory). Here’s the policy that I used (see Granting Permission for Amazon S3 Inventory and Amazon S3 Analytics to learn more):

The inventory mechanism creates three files: a manifest, a checksum for the manifest, and a data file. The manifest contains the location of the data file along with a checksum for it:

{
   "sourceBucket":"jbarr-public",
   "destinationBucket":"arn:aws:s3:::jbarr-s3-inventory",
   "version":"2016-11-30",
   "fileFormat":"CSV",
   "fileSchema":"Bucket, Key, Size, LastModifiedDate, ETag, StorageClass",
   "files":[
      {
         "key":"jbarr-public/DailyInventory/data/cf1da322-f52b-4c61-802a-b5c14f4f32e2.csv.gz",
         "size":2270,
         "MD5checksum":"be6d0eb3f9c4f497fe40658baa5a2e7c"
      }
   ]
}

Here’s what the data file looks like after it has been unzipped:

If you are using the inventory reports to power your daily or weekly processing workflow, don’t forget about S3 Notifications! The checksum file is written after the other two, so you can safely use it to get things moving. Also, don’t forget to use a lifecycle policy to manage your collection of inventory files since they will accumulate over time.

As an added bonus, using daily or weekly reports can save you up to 50%, when compared to multiple LIST operations. To learn more about this feature, read about S3 Storage Inventory.

S3 CloudWatch Metrics
S3 can now publish storage, request, and data transfer metrics to CloudWatch. The storage metrics are reported daily and are available at no extra cost. The request and data transfer metrics are available at one minute intervals (after some processing latency) and are billed at the standard CloudWatch rate. As is the case for the S3 Analytics, the CloudWatch metrics can be collected and reported for an entire bucket or for a subset selected via prefix or tags.

Here is the full set of metrics:

Storage Requests Data Transfer
  • Bucket Size Bytes
  • Number Of Objects
  • GET
  • LIST
  • PUT
  • POST
  • DELETE
  • ALL
  • HEAD
  • 4xx Errors
  • 5xx Errors
  • Total Request Latency
  • First Byte Latency
  • Bytes Uploaded
  • Bytes Downloaded

The metrics are available within the S3 and CloudWatch consoles. Here’s what the Total Request Latency looks like in the S3 Console for my bucket:

I can also click on View in CloudWatch and set CloudWatch alarms for any desired metric. Perhaps I want to be notified if my bucket grows beyond 100 MB:

To learn more, read How Do I Configure Request Metrics for an S3 Bucket?

Available Now
You have had access to these features since late last year!. They are accessible through the updated S3 Console, which also includes many other new features (watch Introduction to the New Amazon S3 Console to learn more).

Jeff;

 

Launch: Amazon GameLift Now Supports All C++ and C# Game Engines

by Tara Walker | on | in Amazon GameLift, Amazon GameLift Server SDK for C#, Amazon GameLift Server SDK for C++, Games, Launch | | Comments

Calling all Game Developers! GDC 2017 was a blast in San Francisco a couple of weeks ago, so there is no better time to be inspired and passionate about learning and building cool games.

Therefore, I am excited to share that Amazon GameLift is now available for all C++ and C# game engines, including Amazon Lumberyard, Unreal Engine, and Unity, all with enhanced game session matching capabilities. For those of you not familiar with Amazon GameLift, let me introduce this managed service designed to aid game developers in delivering fun and innovative online game experiences.

Amazon GameLift is a managed AWS service for hosting dedicated game servers, making it easier for game developers to scale their game capacity and match players into available game sessions. With Amazon GameLift, you can host servers, track game availability, defend game servers from distributed denial of service (DDoS) attacks, and deploy updates without taking your game offline. The Amazon GameLift service powers dedicated game servers for Amazon Game Studios, as well as external game development customers, and is designed to support session-based games with game loops that start and end within a specified time.

The latest Amazon GameLift release enhances the current functionality of the service, as well as adding awesome new features to help simplify game development and deployment for developers. Let us review some of the cool features of the Amazon GameLift service:

  • Multi-engine support: Initially, Amazon GameLift service could only be used with the Amazon Lumberyard game engine. The service is now enhanced to integrate with popular game engines like Unreal Engine, Unity, as well as, custom C# and C++ game engines.
  • New server SDK language support: In order to support a larger set of customers and developers, the service provides an Amazon GameLift Server SDK available for C# and C++. This includes an Unreal Engine plugin, which is a customized version of the C++ Server SDK that is compatible with the Unreal Engine API for Amazon GameLift.
  • Client SDK language support expansion: The Amazon GameLift Client SDK is bundled with the AWS SDK, which is available in a myriad of different languages. This allows game developers to build game clients with an integration of the Amazon GameLift service in their language of choice.
  • Matchmaking: Amazon GameLift continually scans available game servers around the world and matches them against player requests to join games. If low-latency game servers are not available, you can configure the service to automatically add more capacity near your players. Amazon GameLift maintains a queue of waiting players until new games start or new instances launch, then places waiting players into the lowest latency game.
  • Player data handling: Game developers can now store custom player information and pass it directly to a game server. A game server or other game entity with an API call can then retrieve Player data from Amazon GameLift.
  • Console Support: Amazon GameLift supports games developed and architected for Xbox One and PS4.

Amazon GameLift does the heavy lifting of tasks once required to create session-based multiplayer games by simplifying the process of deploying, scaling, and maintaining game servers while reducing the time, cost, and risks associated with building the infrastructure from scratch.

The reference architecture of a gaming solution that utilizes the Amazon GameLift would look as follows:

 

Integrating Amazon GameLift into Your Games

The process of integrating Amazon GameLift into your game build can be broken down in a few simple steps:

  1. Prepare your game server for hosting on Amazon GameLift by setting up your game server project with the Amazon GameLift Server SDK and adding communication code to the project.
  2. Package and upload your game server build to the AWS region targeted for game deployment
  3. Create and build a fleet of computing resources to host the game.
  4. Prepare your game client to connect to game sessions maintained by Amazon GameLift using the AWS SDK with Amazon GameLift APIs and add code to game client for calls to Amazon GameLift service and identifying the player region.
  5. Test your Amazon GameLift integration by connecting an Amazon GameLift-hosted game session and verifying game sessions are being created.

Let’s get started putting these steps into practice by setting up the Amazon GameLift Server SDK in a simple game server project using the Unreal game engine.

Unreal Engine (UE)

We start with Epic’s Unreal game engine. For simplicity, we will create the sample Shooter Game project with online multiplayer functionality built-in, and save it locally on the computer.

Now that I have the Multiplayer Shooter Game sample downloaded and open locally on my machine, I will need to be able to manipulate the C++ code to add the Amazon GameLift service to the UE Online Sub-System to manage the online game sessions. The Shooter Game sample is leveraging the Blueprints Visual Scripting system in Unreal Engine. The Blueprints system is a gameplay scripting system based on node-based interfaces in the UE editor, which enables game designers and content creators to create gameplay elements and functionality within UE editor.

Since it is my goal to use the Amazon GameLift C++ SDK to include the Amazon GameLift service in the game and alter the game code, I will need to create Visual Studio project solution to tie in the game and correlate the source code and any binaries from the Shooter Game to the project. To accomplish this I navigate to the context menu and select the File menu option. In the menu dropdown, I find and select the Generate Visual Studio Project Files option.

Once the project has generated, I only need to return to the Context menu and select File, then Open with Visual Studio in order to open the project and view the source code.

In preparation for adding the Amazon Game Lift service to the Shooter Game as the game service and for game session management, you will need to enable the OnlineSubSystem module in your project. In order to do this, open the game build settings file in the Visual Studio project. Since this game project is named ShooterGame, the build file is named ShooterGame.Build.cs and is located in the Source/ShooterGame folder(s) as shown below.

Open your Build files and uncomment the line for the OnlineSubsystemNull module. Since I am using the sample that already utilizes a multiplayer online system, my build options are set appropriately, and the code looks like this:

public class ShooterGame : ModuleRules
{
	public ShooterGame(TargetInfo Target)
	{
		PrivateIncludePaths.AddRange(
			new string[] { 
				"ShooterGame/Classes/Player",
				"ShooterGame/Private",
				"ShooterGame/Private/UI",
				"ShooterGame/Private/UI/Menu",
				"ShooterGame/Private/UI/Style",
				"ShooterGame/Private/UI/Widgets",
            		}
		);
       PublicDependencyModuleNames.AddRange(
			new string[] {
				"Core",
				"CoreUObject",
				"Engine",
				"OnlineSubsystem",
				"OnlineSubsystemUtils",
				"AssetRegistry",
             			"AIModule",
				"GameplayTasks",
			}
		);
       PrivateDependencyModuleNames.AddRange(
			new string[] {
				"InputCore",
				"Slate",
				"SlateCore",
				"ShooterGameLoadingScreen",
				"Json"
			}
		);
		DynamicallyLoadedModuleNames.AddRange(
			new string[] {
				"OnlineSubsystemNull",
				"NetworkReplayStreaming",
				"NullNetworkReplayStreaming",
				"HttpNetworkReplayStreaming"
			}
		);
		PrivateIncludePathModuleNames.AddRange(
			new string[] {
				"NetworkReplayStreaming"
			}
		);
	}
}

Now that we are set with the Shooter Game project, let’s turn our attention on the Amazon GameLift SDK. I want to leverage the C++ SDK as a plugin for the Unreal Engine, therefore, I need to compile the SDK using the using a compilation directive that builds the binaries for this game engine.

With the SDK source downloaded, I can compile the SDK from the source based upon my operating system. Since I am using a Windows machine for this project, I will complete the following steps:

  • Make an out directory to hold the binaries generated from the code compilation:

mkdir out

  • Change to the previously created directory:

cd out

  • Use CMake to specify a build system generator for VS 2015 Win x64 and set UE compilation flag:

cmake -DBUILD_FOR_UNREAL=1 -G “Visual Studio 14 2015 Win64” <source directory>

  • Build C++ project to create binaries using selected Build System (MS Build for this project):

msbuild ALL_BUILD.vcxproj /p:Configuration=Release

With my libraries compiled, I should have the following binary files required to use the Amazon GameLift Unreal Engine plugin.

Linux:

* out/prefix/lib/aws-cpp-sdk-gamelift-server.so

Windows:

* out\prefix\bin\aws-cpp-sdk-gamelift-server.dll

* out\prefix\lib\aws-cpp-sdk-gamelift-server.lib

As you can see below, since I am on Windows, my compiled Amazon GameLift libraries, aws-cpp-sdk-gamelift-server.dll and aws-cpp-sdk-gamelift-server.lib, are located in the prefix\bin and prefix\lib folders respectively.

After copying the binaries to the GameLiftSDK Unreal Engine plugin folder, my Amazon GameLift plugin folder is configured and ready to be added to an Unreal Engine game project.

Given this, it is now time to add the Amazon GameLift plugin to the Unreal Engine ShooterGame project. I could use the Unreal Engine Editor to add the plugin, but instead, I will stay in the Visual Studio project and add the plugin by updating the game directory and project file.

In Windows Explorer, I add a folder called Plugins in the ShooterGame directory and copy my prepared GameLiftServerSDK folder into the directory as noted by the Unreal Engine documentation on plugins.

Now I will open up the ShooterGame.Build.cs file, which is a C# file that holds information about game dependencies.

Within the file I will add the following code:

PublicDependencyModuleNames.AddRange(
            new string[] {
                "Core",
                "CoreUObject",
                "Engine",
                "InputCore",
                "GameLiftServerSDK"
            }
       );

Just to ensure all is in sync with the changes made thus far, I close Visual Studio, go back to the UE Editor, and select Refresh Visual Studio Project.

Upon completion, I select Open Visual Studio and the Plugins folder I added in the ShooterGame directory is now included in the project and able to be viewed in Solution Explorer.

Next, I rebuild my entire solution to get the Amazon GameLift SDK binaries integrated into the project.

I’ll go back to the UE Editor and select Build from the toolbar to ensure the aspects of the Amazon GameLift plugin are included in my ShooterGame. Once compilation is complete, a quick visit to the Settings toolbar and Plugins option shows that the Amazon GameLift plugin is added and is recognized in the project. I will select the Enabled checkbox, which will prompt me to restart the UE Editor. I select Restart Now and allow the Unreal Engine to rebuild the game code files.

Upon completion of the build, the editor will restart and reopen my ShooterGame.

Now things are set for the use of the Amazon GameLift SDK in the ShooterGame project.

With the Unreal editor open, I’ll go into the Open Visual Studio menu option to get back to the ShooterGame code. This will open up Visual Studio and the game code. With Visual Studio open, I go to the ShooterGameMode.cpp file to add the code to initialize the Amazon GameLift SDK. Some key things I must do in order to correctly add the code for Amazon GameLift within my Shooter game project are:

  1. Enclose the Amazon GameLift code within a preprocessor condition using the flag WITH_GAMELIFT=1
  2. Build a dedicated server in Unreal Engine for my targeted server OS ex. Linux
  3. Ensure my build target is a game server type i.e. Type == TargetRules.TargetType.Server

You can find an example of the code needed to add Amazon GameLift in your Unreal Engine project in the documentation here. In addition, you can learn how to build a dedicated server for Unreal Engine by following the Dedicated Server Guide for Windows and Linux provided in the Unreal Engine wiki. With these resources in hand, you should be well on your way to integrating Amazon GameLift into a game project.

I just did a quick review of incorporating the Amazon GameLift SDK in the Unreal Engine game engine, but don’t forget you have the option to add the Amazon GameLift SDK into C# engines like Unity. By downloading the Amazon GameLift Server SDK and compiling the .Net framework 3.5 solution, GameLiftServerSDKNet35.sln. The GameLiftServerSDKNet35.sln solution will enable you to add the Amazon GameLift libraries your Unity3D project. Review the Amazon GameLift SDK documentation, Using the C# Server SDK for Unity, in order to learn more about setting up and using the Amazon GameLift C# Server SDK plugin.

Summary

We reviewed just one of the new aspects added of the Amazon GameLift managed service, but the service provides game developers and game studios with even more. Amazon GameLift enables the building of distributed games by making it easy to manage infrastructure, scale capacity, and match players into available game sessions while defending games from DDoS attacks.

You can learn more about the Amazon GameLift service by reviewing the Amazon GameLift documentation, the Amazon GameLift developer guide and/or check out the Amazon GameLift tutorials on the Amazon GameDev tutorial page in order to hit the ground running with game development with Amazon GameLift service.

Happy Gaming!

– Tara

Amazon WorkDocs Update – Commenting & Reviewing Enhancements and a New Activity Feed

by Jeff Barr | on | in Amazon WorkDocs | | Comments

As I have told you in the past, we like to drink our own Champagne at Amazon. Practically speaking, this means that we make use of our own services, tools, and applications as part of our jobs, and that we supply the development teams with feedback if we have an idea for an improvement or if we find something that does not work as expected.

I first talked about Amazon WorkDocs (which was originally called Zocalo) back in the middle of 2014, and have been using it ever since (at busy times I often have drafts of 7 or 8 posts circulating).

I upload drafts of every new blog post (usually as PDFs) to WorkDocs and then share them with the Product Manager, Product Marketing Manager, and other designated reviewers. The reviewers leave feedback for me, I update the draft, and I wait for more feedback. After a couple of iterations the draft settles down and I wait for the go-ahead to publish the post. The circle of reviews often grows to include developers, senior management, and so forth. I simply share the document with them and look forward to even more feedback. My job is to read and to process all of the feedback (lots of suggestions, and the occasional question) as quickly as possible and to make sure that I did not miss anything.

Today I would like to tell you about some recent recent enhancements that makes WorkDocs even more useful. We have added some more commenting and reviewing features, along with an activity feed.

Enhanced Commenting
Over the course of a couple of revisions, some comments will spur a discussion. There might be a question about the applicability of a particular feature or the value of a particular image. In order to make it easier to start and to continue conversations, WorkDocs now supports threaded replies. I simply click on Reply and respond to a comment:

It is displayed like this:

If I click on Private, the comment is accessible only to the person who wrote the original.

In order to strengthen my message, I can also use simple formatting (bold, italic, and strikethrough) in my comments. Here’s how I specify each one:

And here’s the result:

Clicking on the ? displays a handy guide to formatting:

When the time for comments has passed, I can now disable feedback with a single click:

To learn more about these features, read Giving Feedback in the WorkDocs User Guide.

Enhanced Reviewing
As the comments accumulate, I sometimes need to draw a reviewer’s attention to a particular comment. I can do this by entering an @ in the comment and then choosing their name from the popup menu:

The user will be notified by email in order to let them know that their feedback is needed.

From time to time, a potential reviewer will come in to possession of a URL to a WorkDocs document but will not have access to the document. They can now request access to the document like this:

The request will be routed to the owner of the document via email for approval.

Similarly, someone who has been granted Viewer-level access can now request Contributor-level access:

Again, the request will be routed to the owner of the document via email for approval:

 

Activity Feed
With multiple blog posts out for review at any given time, keeping track of what’s coming and going can be challenging. In order to give me a big-picture view, WorkDocs now includes an Activity Feed. The feed shows me what is going on with my own documents and with those that have been shared with me. I can watch as files and folders are created, changed, removed, and commented on. I can also see who is making the changes and track the times when they were made:

I can enter a search term to control what I see in the feed:

And I can further filter the updates by activity type or by date:

Available Now
These features are available now and you can start using them today.

Jeff;

 

Amazon Elasticsearch Service support for Elasticsearch 5.1

by Tara Walker | on | in Amazon Elasticsearch Service, Launch | | Comments

The Amazon Elasticsearch Service is a fully managed service that provides easier deployment, operation, and scale for the Elasticsearch open-source search and analytics engine. We are excited to announce that Amazon Elasticsearch Service now supports Elasticsearch 5.1 and Kibana 5.1.

Elasticsearch 5 comes with a ton of new features and enhancements that customers can now take advantage of in Amazon Elasticsearch service. Elements of the Elasticsearch 5 release are as follow:

  • Indexing performance: Improved Indexing throughput with updates to lock implementation & async translog fsyncing
  • Ingestion Pipelines: Incoming data can be sent to a pipeline that applies a series of ingestion processors, allowing transformation to the exact data you want to have in your search index. There are twenty processors included, from simple appending to complex regex applications
  • Painless scripting: Amazon Elasticsearch Service supports Painless, a new secure and performant scripting language for Elasticsearch 5. You can use scripting to change the precedence of search results, delete index fields by query, modify search results to return specific fields, and more.
  • New data structures: Lucene 6 data structures, new data types; half_float, text, keyword, and more complete support for dots-in-fieldnames
  • Search and Aggregations: Refactored search API, BM25 relevance calculations, Instant Aggregations, improvements to histogram aggregations & terms aggregations, and rewritten percolator & completion suggester
  • User experience: Strict settings and body & query string parameter validation, index management improvement, default deprecation logging, new shard allocation API, and new indices efficiency pattern for rollover & shrink APIs
  • Java REST client: simple HTTP/REST Java client that works with Java 7 and handles retry on node failure, as well as, round-robin, sniffing, and logging of requests
  • Other improvements: Lazy unicast hosts DNS lookup, automatic parallel tasking of reindex, update-by-query, delete-by-query, and search cancellation by task management API

The compelling new enhancements of Elasticsearch 5 are meant to make the service faster and easier to use while providing better security. Amazon Elasticsearch Service is a managed service designed to aid customers in building, developing and deploying solutions with Elasticsearch by providing the following capabilities:

  • Multiple configurations of instance types
  • Amazon EBS volumes for data storage
  • Cluster stability improvement with dedicated master nodes
  • Zone awareness – Cluster node allocation across two Availability Zones in the region
  • Access Control & Security with AWS Identity and Access Management (IAM)
  • Various geographical locations/regions for resources
  • Amazon Elasticsearch domain snapshots for replication, backup and restore
  • Integration with Amazon CloudWatch for monitoring Amazon Elasticsearch domain metrics
  • Integration with AWS CloudTrail for configuration auditing
  • Integration with other AWS Services like Kinesis Firehose and DynamoDB for loading of real-time streaming data into Amazon Elasticsearch Service

Amazon Elasticsearch Service allows dynamic changes with zero downtime. You can add instances, remove instances, change instance sizes, change storage configuration, and make other changes dynamically.

The best way to highlight some of the aforementioned capabilities is with an example.

During a presentation at the IT/Dev conference, I demonstrated how to build a serverless employee onboarding system using Express.js, AWS Lambda, Amazon DynamoDB, and Amazon S3. In the demo, the information collected was personnel data stored in DynamoDB about an employee going through a fictional onboarding process. Imagine if the collected employee data could be searched, queried, and analyzed as needed by the company’s HR department. We can easily augment the onboarding system to add these capabilities by enabling the employee table to use DynamoDB Streams to trigger Lambda and store the desired employee attributes in Amazon Elasticsearch Service.

The result is the following solution architecture:

We will focus solely on how to dynamically store and index employee data to Amazon Elasticseach Service each time an employee record is entered and subsequently stored in the database.
To add this enhancement to the existing aforementioned onboarding solution, we will implement the solution as noted by the detailed cloud architecture diagram below:

Let’s look at how to implement the employee load process to the Amazon Elasticsearch Service, which is the first process flow shown in the diagram above.

Amazon Elasticsearch Service: Domain Creation

Let’s now visit the AWS Console to check out Amazon Elasticsearch Service with Elasticsearch 5 in action. As you probably guessed, from the AWS Console home, we select Elasticsearch Service under the Analytics group.

The first step in creating an Elasticsearch solution is to create a domain.  You will notice that now when creating an Amazon Elasticsearch Service domain, you now have the option to choose the Elasticsearch 5.1 version.  Since we are discussing the launch of the support of Elasticsearch 5, we will, of course, choose the 5.1 Elasticsearch engine version when creating our domain in the Amazon Elasticsearch Service.


After clicking Next, we will now setup our Elasticsearch domain by configuring our instance and storage settings. The instance type and the number of instances for your cluster should be determined based upon your application’s availability, network volume, and data needs. A recommended best practice is to choose two or more instances in order to avoid possible data inconsistencies or split brain failure conditions with Elasticsearch. Therefore, I will choose two instances/data nodes for my cluster and set up EBS as my storage device.

To understand how many instances you will need for your specific application, please review the blog post, Get Started with Amazon Elasticsearch Service: How Many Data Instances Do I Need, on the AWS Database blog.

All that is left for me is to set up the access policy and deploy the service. Once I create my service, the domain will be initialized and deployed.

Now that I have my Elasticsearch service running, I now need a mechanism to populate it with data. I will implement a dynamic data load process of the employee data to Amazon Elasticsearch Service using DynamoDB Streams.

Amazon DynamoDB: Table and Streams

Before I head to the DynamoDB console, I will quickly cover the basics.

Amazon DynamoDB is a scalable, distributed NoSQL database service. DynamoDB Streams provide an ordered, time-based sequence of every CRUD operation to the items in a DynamoDB table. Each stream record has information about the primary attribute modification for an individual item in the table. Streams execute asynchronously and can write stream records in practically real time. Additionally, a stream can be enabled when a table is created or can be enabled and modified on an existing table. You can learn more about DynamoDB Streams in the DynamoDB developer guide.

Now we will head to the DynamoDB console and view the OnboardingEmployeeData table.

This table has a primary partition key, UserID, that is a string data type and a primary sort key, Username, which is also of a string data type. We will use the UserID as the document ID in Elasticsearch. You will also notice that on this table, streams are enabled and the stream view type is New image. A stream that is set to a New image view type will have stream records that display the entire item record after it has been updated. You also have the option to have the stream present records that provide data items before modification, provide only the items’ key attributes, or provide old and new item information.  If you opt to use the AWS CLI to create your DynamoDB table, the key information to capture is the Latest Stream ARN shown underneath the Stream Details section. A DynamoDB stream has a unique ARN identifier that is outside of the ARN of the DynamoDB table. The stream ARN will be needed to create the IAM policy for access permissions between the stream and the Lambda function.

IAM Policy

The first thing that is essential for any service implementation is getting the correct permissions in place. Therefore, I will first go to the IAM console to create a role and a policy for my Lambda function that will provide permissions for DynamoDB and Elasticsearch.

First, I will create a policy based upon an existing managed policy for Lambda execution with DynamoDB Streams.

This will take us to the Review Policy screen, which will have the selected managed policy details. I’ll name this policy, Onboarding-LambdaDynamoDB-toElasticsearch, and then customize the policy for my solution. The first thing you should notice is that the current policy allows access to all streams, however, the best practice would be to have this policy only access the specific DynamoDB Stream by adding the Latest Stream ARN. Hence, I will alter the policy and add the ARN for the DynamoDB table, OnboardingEmployeeData, and validate the policy. The altered policy is as shown below.

The only thing left is to add the Amazon Elasticsearch Service permissions in the policy. The core policy for Amazon Elasticsearch Service access permissions is as shown below:

 

I will use this policy and add the specific Elasticsearch domain ARN as the Resource for the policy. This ensures that I have a policy that enforces the Least Privilege security best practice for policies. With the Amazon Elasticsearch Service domain added as shown, I can validate and save the policy.

The best way to create a custom policy is to use the IAM Policy Simulator or view the examples of the AWS service permissions from the service documentation. You can also find some examples of policies for a subset of AWS Services here. Remember you should only add the ES permissions that are needed using the Least Privilege security best practice, the policy shown above is used only as an example.

We will create the role for our Lambda function to use to grant access and attach the aforementioned policy to the role.

AWS Lambda: DynamoDB triggered Lambda function

AWS Lambda is the core of Amazon Web Services serverless computing offering. With Lambda, you can write and run code using supported languages for almost any type of application or backend service. Lambda will trigger your code in response to events from AWS services or from HTTP requests. Lambda will dynamically scale based upon workload and you only pay for your code execution.

We will have DynamoDB streams trigger a Lambda function that will create an index and send data to Elasticsearch. Another option for this is to use the Logstash plugin for DynamoDB. However, since several of the Logstash processors are now included in Elasticsearch 5.1 core and with the improved performance optimizations, I will opt to use Lambda to process my DynamoDB stream and load data to Amazon Elasticsearch Service.
Now let us head over to the AWS Lambda console and create the lambda function for loading employee data to Amazon Elasticsearch Service.

Once in the console, I will create a new Lambda function by selecting the Blank Function blueprint that will take me to the Configure Trigger page. Once on the trigger page, I will select DynamoDB as the AWS service which will trigger Lambda, and I provide the following trigger related options:

  • Table: OnboardingEmployeeData
  • Batch size: 100 (default)
  • Starting position: Trim Horizon

I hit Next button, and I am on the Configure Function screen. The name of my function will be ESEmployeeLoad and I will write this function in Node.4.3.

The Lambda function code is as follows:

var AWS = require('aws-sdk');
var path = require('path');

//Object for all the ElasticSearch Domain Info
var esDomain = {
    region: process.env.RegionForES,
    endpoint: process.env.EndpointForES,
    index: process.env.IndexForES,
    doctype: 'onboardingrecords'
};
//AWS Endpoint from created ES Domain Endpoint
var endpoint = new AWS.Endpoint(esDomain.endpoint);
//The AWS credentials are picked up from the environment.
var creds = new AWS.EnvironmentCredentials('AWS');

console.log('Loading function');
exports.handler = (event, context, callback) => {
    //console.log('Received event:', JSON.stringify(event, null, 2));
    console.log(JSON.stringify(esDomain));
    
    event.Records.forEach((record) => {
        console.log(record.eventID);
        console.log(record.eventName);
        console.log('DynamoDB Record: %j', record.dynamodb);
       
        var dbRecord = JSON.stringify(record.dynamodb);
        postToES(dbRecord, context, callback);
    });
};

function postToES(doc, context, lambdaCallback) {
    var req = new AWS.HttpRequest(endpoint);

    req.method = 'POST';
    req.path = path.join('/', esDomain.index, esDomain.doctype);
    req.region = esDomain.region;
    req.headers['presigned-expires'] = false;
    req.headers['Host'] = endpoint.host;
    req.body = doc;

    var signer = new AWS.Signers.V4(req , 'es');  // es: service code
    signer.addAuthorization(creds, new Date());

    var send = new AWS.NodeHttpClient();
    send.handleRequest(req, null, function(httpResp) {
        var respBody = '';
        httpResp.on('data', function (chunk) {
            respBody += chunk;
        });
        httpResp.on('end', function (chunk) {
            console.log('Response: ' + respBody);
            lambdaCallback(null,'Lambda added document ' + doc);
        });
    }, function(err) {
        console.log('Error: ' + err);
        lambdaCallback('Lambda failed with error ' + err);
    });
}

The Lambda function Environment variables are:

I will select an Existing role option and choose the ESOnboardingSystem IAM role I created earlier.

Upon completing my IAM role permissions for the Lambda function, I can review the Lambda function details and complete the creation of ESEmployeeLoad function.

I have completed the process of building my Lambda function to talk to Elasticsearch, and now I test my function my simulating data changes to my database.

Now my function, ESEmployeeLoad, will execute upon changes to the data in my database from my onboarding system. Additionally, I can review the processing of the Lambda function to Elasticsearch by reviewing the CloudWatch logs.

Now I can alter my Lambda function to take advantage of the new features or go directly to Elasticsearch and utilize the new Ingest Mode. An example of this would be to implement a pipeline for my Employee record documents.

I can replicate this function for handling the badge updates to the employee record, and/or leverage other preprocessors against the employee data. For instance, if I wanted to do a search of data based upon a data parameter in the Elasticsearch document, I could use the Search API and get records from the dataset.

The possibilities are endless, and you can get as creative as your data needs dictate while maintaining great performance.

Amazon Elasticsearch Service: Kibana 5.1

All Amazon Elasticsearch Service domains using Elasticsearch 5.1 are bundled with Kibana 5.1, the latest version of the open-source visualization tool.

The companion visualization and analytics platform, Kibana, has also been enhanced in the Kibana 5.1 release. Kibana is used to view, search or and interact with Elasticsearch data with a myriad of different charts, tables, and maps.  In addition, Kibana performs advanced data analysis of large volumes of the data. Key enhancements of the Kibana release are as follows:

  • Visualization tool new design: Updated color scheme and maximization of screen real-estate
  • Timelion: visualization tool with a time-based query DSL
  • Console: formerly known as Sense is now part of the core, using the same configuration for free-form requests to Elasticsearch
  • Scripted field language: ability use new Painless scripting language in the Elasticsearch cluster
  • Tag Cloud Visualization: 5.1 adds a word base graphical view of data sized by importance
  • More Charts: return of previously removed charts and addition of advanced view for X-Pack
  • Profiler UI:1 provides an enhancement to profile API with tree view
  • Rendering performance improvement: Discover performance fixes, decrease of CPU load

Summary

As you can see this release is expansive with many enhancements to assist customers in building Elasticsearch solutions. Amazon Elasticsearch Service now supports 15 new Elasticsearch APIs and 6 new plugins. Amazon Elasticsearch Service supports the following operations for Elasticsearch 5.1:

You can read more about the supported operations for Elasticsearch in the Amazon Elasticsearch Developer Guide, and you can get started by visiting the Amazon Elasticsearch Service website and/or sign into the AWS Management Console.

Tara

 

Streamline AMI Maintenance and Patching Using Amazon EC2 Systems Manager | Automation

by Ana Visneski | on | | Comments

Here to tell you about using Automation to streamline AMI maintenance and patching is Taylor Anderson, a Senior Product Manager with EC2.

-Ana


 

Last December at re:Invent, we launched Amazon EC2 Systems Manager, which helps you automatically collect software inventory, apply OS patches, create system images, and configure Windows and Linux operating systems. These capabilities enable automated configuration and ongoing management of systems at scale, and help maintain software compliance for instances running in Amazon EC2 or on-premises.

One feature within Systems Manager is Automation, which can be used to patch, update agents, or bake applications into an Amazon Machine Image (AMI). With Automation, you can avoid the time and effort associated with manual image updates, and instead build AMIs through a streamlined, repeatable, and auditable process.

Recently, we released the first public document for Automation: AWS-UpdateLinuxAmi. This document allows you to automate patching of Ubuntu, CentOS, RHEL, and Amazon Linux AMIs, as well as automating the installation of additional site-specific packages and configurations.

More importantly, it makes it easy to get started with Automation, eliminating the need to first write an Automation document. AWS-UpdateLinuxAmi can also be used as a template when building your own Automation workflow. Windows users can expect the equivalent document―AWS-UpdateWindowsAmi―in the coming weeks.

AWS-UpdateLinuxAmi automates the following workflow:

  1. Launch a temporary EC2 instance from a source Linux AMI.
  2. Update the instance.
    • Invoke a user-provided, pre-update hook script on the instance (optional).
    • Update any AWS tools and agents on the instance, if present.
    • Update the instance’s distribution packages using the native package manager.
    • Invoke a user-provided post-update hook script on the instance (optional).
  3. Stop the temporary instance.
  4. Create a new AMI from the stopped instance.
  5. Terminate the instance.

Warning: Creation of an AMI from a running instance carries a risk that credentials, secrets, or other confidential information from that instance may be recorded to the new image. Use caution when managing AMIs created by this process.

Configuring roles and permissions for Automation

If you haven’t used Automation before, you need to configure IAM roles and permissions. This includes creating a service role for Automation, assigning a passrole to authorize a user to provide the service role, and creating an instance role to enable instance management under Systems Manager. For more details, see Configuring Access to Automation.

Executing Automation

      1. In the EC2 console, choose Systems Manager Services, Automations.
      2. Choose Run automation document
      3. Expand Document name and choose AWS-UpdateLinuxAmi.
      4. Choose the latest document version.
      5.  For SourceAmiId, enter the ID of the Linux AMI to update.
      6. For InstanceIamRole, enter the name of the instance role you created enabling Systems Manager to manage an instance (that is, it includes the AmazonEC2RoleforSSM managed policy). For more details, see Configuring Access to Automation.
      7.  For AutomationAssumeRole, enter the ARN of the service role you created for Automation. For more details, see Configuring Access to Automation.
      8.  Choose Run Automation.
      9. Monitor progress in the Automation Steps tab, and view step-level outputs.

After execution is complete, choose Description to view any outputs returned by the workflow. In this example, AWS-UpdateLinuxAmi returns the new AMI ID.

Next, choose Images, AMIs to view your new AMI.

There is no additional charge to use the Automation service, and any resources created by a workflow incur nominal charges. Note that if you terminate AWS-UpdateLinuxAmi before reaching the “Terminate Instance” step, shut down the temporary instance created by the workflow.

A CLI walkthrough of the above steps can be found at Automation CLI Walkthrough: Patch a Linux AMI.

Conclusion

Now that you’ve successfully run AWS-UpdateLinuxAmi, you may want to create default values for the service and instance roles. You can customize your workflow by creating your own Automation document based on AWS-UpdateLinuxAmi. For more details, see Create an Automaton Document. After you’ve created your document, you can write additional steps and add them to the workflow.

Example steps include:

      • Updating an Auto Scaling group with the new AMI ID (aws:invokeLambdaFunction action type)
      • Creating an encrypted copy of your new AMI (aws:encrypedCopy action type)
      • Validating your new AMI using Run Command with the RunPowerShellScript document (aws:runCommand action type)

Automation also makes a great addition to a CI/CD pipeline for application bake-in, and can be invoked as a CLI build step in Jenkins. For details on these examples, be sure to check out the Automation technical documentation. For updates on Automation, Amazon EC2 Systems Manager, Amazon CloudFormation, AWS Config, AWS OpsWorks and other management services, be sure to follow the all-new Management Tools blog.

 

New – Instance Size Flexibility for EC2 Reserved Instances

by Jeff Barr | on | in Amazon EC2 | | Comments

Reserved Instances allow AWS customers to receive a significant discount on their EC2 usage (up to 75% when compared to On-Demand pricing), along with capacity reservation when the RIs are purchased for use in a specific Availability Zone (AZ).

Late last year we made Reserved Instances more flexible with the launch of Regional RIs that apply the discount to any AZ in the Region, along with Convertible RIs that allow you to change the instance family and other parameters associated with a Reserved Instance. Both types of RIs reduce your management overhead and provide you with additional options. When you use Regional RIs you can launch an instance without having to worry about launching in the AZ that is eligible for the RI discount. When you use Convertible RIs you can ensure that your RIs remain well-fitted to your usage, even as your choice of instance types and sizes varies over time.

Instance Size Flexibility
Effective March 1, your existing Regional RIs are even more flexible! All Regional Linux/UNIX RIs with shared tenancy now apply to all sizes of instances within an instance family and AWS region, even if you are using them across multiple accounts via Consolidated Billing. This will further reduce the time that you spend managing your RIs and will let you be even more creative and innovative with your use of compute resources.

All new and existing RIs are sized according to a normalization factor that is based on the instance size:

Instance Size
Normalization Factor
nano 0.25
micro 0.5
small 1
medium 2
large 4
xlarge 8
2xlarge 16
4xlarge 32
8xlarge 64
10xlarge 80
16xlarge 128
32xlarge 256

Let’s say you already own an RI for a c4.8xlarge. This RI now applies to any usage of a Linux/UNIX C4 instance with shared tenancy in the region. This could be:

  • One c4.8xlarge instance.
  • Two c4.4xlarge instances.
  • Four c4.2xlarge instances.
  • Sixteen c4.large instances.

It also includes other combinations such as one c4.4xlarge and eight c4.large instances.

If you own an RI that is smaller than the instance that you are running, you will be charged the pro-rated, On-Demand price for the excess. This means that you could buy an RI for a c4.4xlarge, use that instance most of the time, but scale up to a c4.8xlarge instance on occasion. We’ll do the math and you’ll pay only half of the On-Demand, per-hour price for the larger instance (as always, our goal is to give you access to compute power at the lowest possible cost). If you own an RI for a large instance and run a smaller instance, the RI price will apply to the smaller instance. However, the unused portion of the reservation will not accumulate.

Now Available
This new-found flexibility is available now and will be applied automatically to your Regional Linux/UNIX RIs with shared tenancy, without any effort on your part.

Jeff;