Amazon Web Services Blog

  • Query Your EC2 Instances Using Tag and Attribute Filtering

    02 Sep 2014 in Amazon EC2 | permalink

    As an Amazon Elastic Compute Cloud (EC2) user, you probably know just how simple and easy it is to launch EC2 instances on an as-needed basis. Perhaps you got your start by manually launching an instance or two, and later moved to a model where you launch instances through a AWS CloudFormation template, Auto Scaling, or in Spot form.

    Today we are launching an important new feature for the AWS Management Console. You can now find the instance or instances that you are looking for by filtering on tags and attributes, with some advanced options including inverse search, partial search, and regular expressions.

    Instance Tags
    Regardless of the manner in which you launch them, you probably want to track the role (development, test, production, and so forth) internal owner, and other attributes of each instance. This becomes especially important as your fleet grows to hundreds or thousands of instances. We have long supported tagging of EC2 instances (and other resources) for many years. As you probably know already, you can add up to ten tags (name/value pairs) to many types of AWS resources. While I can sort by the tags to group like-tagged instances together, there's clearly room to do even better! With today's launch, you can use the tags that you assign, along with the instance attributes, to locate the instance or instances that you are looking for.

    Query With Tags & Attributes
    As I was writing this post, I launched ten EC2 instances, added Mode and Owner tags to each one (supplementing the default Name, and then configured the console to show the tags and their values:

    The new filter box offers many options. I'll do my best to show them all to you!

    In the examples that follow, I will filter my instances using the tags that I assigned to the instances. I'll start with simple examples and work up to some more complex ones. I can filter by keyword. Let's say that I am looking for an instance and can only recall part of the instance id (this turns out to be a very popular way to search). I enter the partial id ("2a27") in to the filter box and press Enter to find it:

    Let's say that I want to find all of the instances where I am listed as Owner. I click in the Filter box for some guidance:

    I select the Owner tag and select from among the values presented to me:

    Here are the results:

    I can add a second filter if I want to see only the instances where I am the owner and the Mode is "Production":

    I can also filter by any of the attributes of the instance. For example, I can easily find all of the instances that are in the Stopped state:

    And I can, of course, combine this with a filter on a tag. I can find all of my stopped instances:

    I can use an inverse search to find everyone else's stopped instances (I simply prefix the value with an exclamation mark):

    I can also use regular expressions to find instances owned by Kelly or Andy:

    And I can do partial matches to compensate for inconsistent naming:

    I can even filter by launch date to find instances that are newer or older than a particular time:

    Finally, the filter information is represented in the console URL so that you can bookmark your filters or share them with your colleagues:

    Filter Now
    This feature is available now and you can start using it today. It works for EC2 instances now; we expect to make it available for other types of EC2 resources before too long.

    -- Jeff;

  • AWS Week in Review - August 25, 2014

    02 Sep 2014 | permalink

    Let's take a quick look at what happened in AWS-land last week:

    Monday, August 25
    Tuesday, August 26
    Wednesday, August 27
    Thursday, August 28
    Friday, August  29

    Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

    -- Jeff;

  • New Resources APIs for the AWS SDK for Java

    28 Aug 2014 in Developers, Java | permalink

    We are launching a preview of a new, resource-style API model for the AWS SDK for Java. I will summarize the preview here, and refer you to the AWS Java Blog for full information!

    The new resource-oriented APIs are designed to be easier to understand and simpler to use. It obviates much of the request-response verbosity present in the existing model and presents a view of AWS that is decidedly object-oriented. Instead of exposing all of the methods of the service as part of a single class, the resource-style API includes multiple classes, each of which represents a particular type of resource for the service. Each class includes the methods needed to interact with the resource and with related resources of other types. Code written to the new API will generally be shorter, cleaner, and easier to comprehend.

    Here is the old-school way to retrieve an AWS Identity and Access Management (IAM) group using the GetGroup function:

    AmazonIdentityManagement iam = new AmazonIdentityManagementClient();
    iam.setRegion(Region.getRegion(Regions.US_WEST_2));
    
    GetGroupRequest getGroupRequest = new GetGroupRequest("NeedNewKeys");
    GetGroupResult getGroupResult = iam.getGroup(getGroupRequest);
    

    And here is the new way:

    IdentityManagement iam = ServiceBuilder.forService(IdentityManagement.class)
      .withRegion(Region.getRegion(Regions.US_WEST_2))
      .build();
    
    Group needNewKeys = iam.getGroup("NeedNewKeys");
    

    The difference between the old and the new APIs becomes even more pronounced when more complex operations are used. Compare the old-school code for marking an outdated access key (oldKey) for an IAM user as inactive:

    UpdateAccessKeyRequest updateAccessKeyRequest = new UpdateAccessKeyRequest()
      .withAccessKeyId(oldKey)
      .withUserName(user.getUserName())
      .withStatus(StatusType.Inactive);
    iam.updateAccessKey(updateAccessKeyRequest);
    

    With the new, streamlined code, the intent is a lot more obvious. There's a lot less in the way of setup code and the method is invoked on the object of interest instead of on the service:

    oldKey.deactivate();
    

    The new API is being launched in preview mode with support for Amazon Elastic Compute Cloud (EC2), AWS Identity and Access Management (IAM), and Amazon Glacier. We plan to introduce resource APIs for other services and other AWS SDKs in the future.

    -- Jeff;

    PS - To learn more about Resource APIs, read the full post on the AWS Java Development Blog .

  • Amazon Zocalo - Now Generally Available

    27 Aug 2014 in Amazon Zocalo | permalink

    Amazon Zocalo has been available in a Limited Preview since early July (see my blog post, Amazon Zocalo - Document Storage and Sharing for the Enterprise to learn more). During the Limited Preview, many AWS users expressed interest in evaluating Zocalo and were admitted in to the Preview on a space-available basis.

    Today we are making Amazon Zocalo generally available to all AWS customers. You can sign up today and start using Zocalo now. There's a 30-day free trial (200 GB of storage per user for up to 50 users); after that you pay $5 per user per month (see the Zocalo Pricing page for more information).

    As part of this move to general availability, we are also announcing that AWS CloudTrail now records calls made to the Zocalo API. This API is currently internal, but we plan to expose it in the future. If you are interested in building applications that work with the Zocalo API, please express your interest by emailing us at zocalo-feedback@amazon.com. We are very interested in learning more about the kinds of applications that you are thinking about building.

    I have become a regular user of Zocalo, and also a big fan! I generally have between 5 and 10 blog post drafts under way at any given time. I write the first draft, upload it to Zocalo, and share it with the Product Manager for initial review. We iterate on the early drafts to smooth out any kinks, and then share it with a wider audience for final review. When multiple reviewers provide feedback on the same document, Zocalo's Feedback tab lets me scan, summarize, and respond to the feedback quickly and efficiently.

    -- Jeff;

  • Enhanced Throughput for Provisioned IOPS (SSD) and General Purpose (SSD) EBS Volumes

    Back in the old, pre-cloud days, updating your data center to use the latest and greatest hardware was expensive, somewhat risky, and resource intensive. You would have to make the capital investment to acquire new hardware based on your usual 3 or 5 year refresh cycle, field test it, and then migrate your systems and applications. The time between "I saw this cool thing and it could benefit our work" and "we are using this cool thing and it is benefitting our work" was often measured in quarters or years. Delays or inefficiencies in this process have the potential to affect the competitive position, health, and overall viability of your organization.

    As I have said before, the cloud changes this model for the better. First of all, your cloud provider has an incentive to bring the newest and most powerful technology to market on a timely basis. Second, the dynamic nature of the cloud makes it easy for you to launch, test, and measure the performance of your existing applications on the new technology without disrupting your production systems.

    General Purpose (SSD) Adoption is Strong
    I would like to share some interesting numbers with you that illustrate the game-changing nature of the Cloud. In mid-June we announced SSD-Backed Elastic Block Storage and made it available in all AWS Regions. We knew that our customers would find this new offering attractive but we were not quite sure (despite plenty of market research and modeling) just how popular it would turn out to be.

    In less than three months, the General Purpose (SSD) EBS storage has grown to the extent that it is now one of the fastest adopted services in the history of AWS! Here are two data points:

    1. Within a few weeks of the launch, over 25% of the EBS customer base was already making use of the new General Purpose (SSD) EBS volumes in some way.
    2. Today, most of our customers are now using General Purpose (SSD) volumes to meet their need for general purpose block storage. In fact, about 90% of the newly created block storage is now on SSD volumes.
    Looking at this another way, the easy and capital-free migration made possible by the cloud has allowed the vast majority of our customers to move to a new generation of storage in a little over two months. Any way you look at it, this is a rapid upgrade cycle!

    Throughput Enhancement
    To celebrate this huge step forward (and because we love to innovate), we are improving data transfer throughput for General Purpose (SSD) and Provisioned IOPS (SSD) volumes. Here's what's new:

    1. The maximum attainable throughput to each volume has been doubled. Each General Purpose (SSD) and Provisioned IOPS (SSD) volume can now sustain up to 128 megabytes per second of read or write traffic.
    2. An I/O request of up to 256 kilobytes is now counted as a single I/O operation (IOP). In other words, a single IOP is now up to 16 times as cost-effective and performant as before (prior this enhancement, each IOP represented at most 16 kilobytes of data transfer). If you attach multiple General Purpose (SSD) or Provisioned IOPS (SSD) volumes to a single c3.8xlarge EC2 instance you can achieve up to 800 megabytes per second of aggregate throughput per instance.
    These changes will improve your I/O performance and can also dramatically reduce your storage costs. If your application has a need for 128 megabytes per second of data transfer, you can now meet this need by provisioning 500 IOPS instead of 8000 IOPS.

    As I noted above, an I/O request for up to 256 kilobytes is now counted as a single I/O operation. In some cases you can configure your application or your operating environment to make large read and write requests. For example, you can configure the size of requests made by Hadoop by altering the dfs.blocksize parameter. If you are building your own applications, you can read or write large blocks.

    As part of this launch we have also updated the EC2 AMIs for Microsoft Windows. The new AMIs ("2014.08.13") will use SSD volumes exclusively and have an updated PV driver for increased performance. The Microsoft Security Updates are current to August 2014, the PowerShell Tools have been updated.

    This enhancement is now in effect in all AWS Regions. If you are using General Purpose (SSD) or Provisioned IOPS (SSD) volumes then you are already reaping the benefits. We expect this enhancement to improve performance on many types of I/O-intensive workloads including those which involve database loads and scans across large tables.

    -- Jeff;

  • AWS Week in Review - August 18, 2014

    25 Aug 2014 | permalink

    Let's take a quick look at what happened in AWS-land last week:

    Monday, August  18
    Tuesday, August 19
    Wednesday, August 20
    Thursday, August 21
    Friday,August 22

    Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

    -- Jeff;

  • AWS Pop-up Loft - Returning in the Fall!

    22 Aug 2014 | permalink

    Earlier this year we opened up the AWS Pop-up Loft for a pilot run of almost four weeks in San Francisco. During that time, many AWS developers dropped in to network, listen, learn, work, and socialize. Some developers came and enjoyed the structured, scheduled events. Others came in with their laptops, found a quiet corner, and spent some time working on their code.

    During my three day stint at The Loft, I met a number of interesting entrepreneurs and spent time learning about their plans to change the world. For example, I spoke with Cosmo Mielke of infino.me to learn more about his citizen science experiment. He's working to use Big Data to understand the interaction between genetics and lifestyle in an effort to prevent diseases and prolong lives.

    I am happy to announce that The Loft will reopen in the fall! Based on the feedback that we received during the pilot in June, we have fine-tuned the model to make it even more valuable for you. Here are some of the things that you will be able to do at The Loft:

    • Meet 1:1 with an AWS technical expert.
    • Learn about AWS through product sessions.
    • Gain hands-on experience through instructor-led Technical Bootcamps.
    • Sharpen your AWS skills through self-paced, hands-on labs.
    • Attend special evening events including talks with successful startups and networking opportunities (past speakers have represented Twilio, Coin, Hearsay Social, CoreOS, and Chef).

    To make sure that you know about all of the goings-on at The Loft, I'd encourage you to sign up for email alerts. We'll let you know when the doors are open, and we'll send you the information that you'll need to plan your visit(s).

    -- Jeff;

  • DISA Authorizes AWS as First Commercial Cloud Approved for Sensitive Workloads

    21 Aug 2014 | permalink

    I am happy to be able to announce that AWS has achieved the first DoD Provisional Authorization under the DoD Cloud Security Model's at security impact levels 3-5! AWS previously received a DoD Provisional Authorization for security impact levels 1-2. This new Authorization covers AWS GovCloud (US) and DoD customers can now move forward with their deployments of applications processing controlled and for official use only unclassified information. As part of the Level 3-5 Authorization, our partners and DoD customers will be able to implement a wide range of DoD requirements necessary to protect their data at these levels, including AWS Direct Connect routing to the DoD's network, comprehensive computer network defense coverage, and Common Access Card (CAC) integration.

    In March, AWS announced its compliance with security impact levels 1-2 for all AWS Regions in the US, demonstrating adherence to hundreds of controls. With this authorization, we have provided a means for DoD customers deploy applications at levels 3-5. DoD customers with prospective Level 3-5 applications should contact the ECSB to begin the deployment process.

    With today's announcement, DoD agencies can leverage the AWS Provisional Authorization for security impact levels 1-2 and AWS GovCloud.s Provisional Authorization at levels 3-5 to evaluate AWS for their unclassified applications and workloads, achieve their own authorizations to use AWS, and transition DoD workloads into the AWS environment. DoD components and federal contractors can immediately request DoD compliance support by submitting a FedRAMP/DoD Compliance Support Request and begin to moving through the authorization process to achieve a DoD ATO for Levels 1-5 with AWS.

    -- Jeff;

  • New SSL Features for Amazon CloudFront - Session Tickets, OCSP Stapling, Perfect Forward Secrecy

    20 Aug 2014 in Amazon CloudFront | permalink

    You probably know that you can use Amazon CloudFront to distribute your content to users around the world with a high degree of security, low latency and high data transfer speed. CloudFront supports the use of secure HTTPS connections from the origin to the edge and from the edge to the client; if you enable this option data travels from the origin to your end users in a secure, encrypted form.

    Today we are making some additional improvements to the performance and security of CloudFront's SSL implementation. These features are enabled automatically and work with the default CloudFront SSL certificate as well as custom (SNI and Dedicated IP) SSL certificates.

    Performance Enhancements
    We have improved the performance of SSL connections with the use of Session Tickets and OCSP Stapling. Both of these features are built in to the SSL protocol and you don't have to make any code or configuration changes in order to use them. In other words, you (and your users) are already benefitting from these improvements.

    SSL Session Tickets - As part of the SSL handshake process, the client and the server exchange multiple packets as part of a negotiation ritual that results in agreement to use a particular encryption model (cipher) and certificate. This process entails multiple round trips and a fair amount of computation on both ends which adds some latency to the connection process. This process has to be repeated if the connection is broken. To avoid some of this rigmarole while keeping the connection secure, CloudFront now implements SSL Session Tickets. After the negotiation is complete, the SSL server creates an encrypted session ticket and returns it to the client. Later, the client can present this ticket to the server as an alternative to a full negotiation when resuming or restarting a connection. The ticket reminds the server of what they have already agreed to as part of an earlier SSL handshake.

    OCSP Stapling - An SSL certificate must be validated before it can be used. The certificate authority (CA) for the certificate must be consulted in order to ensure that the certificate is legitimate and that it has not been revoked. In the absence of support for OCSP Stapling, the client (e.g. a web browser) will take care of this interaction with the CA, once again at the cost of some round trips and the associated latency they bring. CloudFront now implements OCSP Stapling. This approach moves the burden of domain name resolution (to locate the CA) and certificate validation over to CloudFront, where the results can be cached and then attached (stapled, hence the name) to one of the packets in the SSL handshake. The clients no longer need to handle the domain name resolution or certificate validation and benefits from the work done on the server.

    Security Enhancements
    We have added support for Perfect Forward Secrecy and newer SSL ciphers.

    Perfect Forward Secrecy - This feature creates a new private key for each SSL session. In the event that a private key for a session was discovered, it could be used only to decode that session and no other, past or future.

    Newer Ciphers - CloudFront now supports a set of advanced RSA-AES ciphers. The server and the client agree on a cipher automatically as part of the SSL handshake process.

    Available Now
    These new features are available now at no extra charge and you may already be using them today! See the CloudFront Pricing page for more information.

    -- Jeff;

  • Amazon SNS Update - Large Topics and MPNS Authenticated Mode

    19 Aug 2014 in Amazon SNS | permalink

    Amazon Simple Notification Service (SNS) is a fast and flexible push messaging service. You can easily send messages to Apple, Google, Fire OS and Windows devices, including Android devices in China (via Baidu Cloud Push).

    Today we are enhancing SNS with support for large topics (more than 10,000 subscribers) and authenticated delivery to MPNS (Microsoft Push Notification Service).

    Large Topics
    SNS offers two publish modes. First, you can push messages directly to specific mobile devices. Second, you can create an SNS topic, provide your customers with a mechanism to allow them to subscribe to the topic, and then publish messages to the topic with a single API call. This mode is great for broadcasting breaking news, announcing flash deals, and announcing in-game events or new features. You can combine customers from different platforms in the same topic and you can send a specific payload to each platform (for example, one for iOS and another for Android), again in a single call. Suppose you have created the following topic:

    With the ARN for the topic (arn:aws:sns:us-west-2:xxxxxxxxxxxx:amazon-sns) in hand, here's how you publish a message to all of the subscribers:

    $result = $client->publish(array(
        'TopicArn' => 'arn:aws:sns:us-west-2:xxxxxxxxxxxx:amazon-sns',
        // Message is required
        'Message' => 'Hello Subscribers',
        'Subject' => 'Hello'
    ));
    

    Today we are lifting the limit of 10,000 subscriptions per SNS topic; you can now create as many as you need and no longer need to partition large subscription lists across multiple topics. This has been a frequent request from AWS customers that use SNS to build news and media sharing applications.

    There is an administrative limit of 10 million subscriptions per topic, but we'll happily raise it if you expect to have more subscribers for a single topic. Fill out the Contact Us form, select SNS, and we'll take good care of you!

    Authenticated Delivery to MPNS
    Microsoft Push Notification Service (MPNS) is the push notification relay service for Windows Phone devices prior to Windows 8.1. SNS now supports authenticated delivery to MPNS. In this mode, MPNS does not enforce any limitations on the number of notifications that can be sent to a channel in any given day (per the documentation on Windows Phone Push Mode, there's a daily limit of 500 unauthenticated push notifications per channel).

    If you require this functionality for devices that run Windows 8.1 and above, please consider using Amazon SNS for Windows Notification Service (WNS).

    -- Jeff;