Category: Amazon SNS


EC2 Run Command Update – Monitor Execution Using Notifications

by Jeff Barr | on | in Amazon EC2, Amazon SNS, CloudWatch Events |

We launched EC2 Run Command late last year and have enjoyed seeing our customers put it to use in their cloud and on-premises environments. After the launch, we quickly added Support for Linux Instances, the power to Manage & Share Commands, and the ability to do Hybrid & Cross-Cloud Management. Earlier today we made EC2 Run Command available in the China (Beijing) and Asia Pacific (Seoul) Regions.

Our customers are using EC2 Run Command to automate and encapsulate routine system administration tasks. They are creating local users and groups, scanning for and then installing applicable Windows updates, managing services, checking log files, and the like. Because these customers are using EC2 Run Command as a building block, they have told us that they would like to have better visibility into the actual command execution process. They would like to know, quickly and often in detail, when each command and each code block in the command begins executing, when it completes, and how it completed (successfully or unsuccessfully).

In order to support this really important use case, you can now arrange to be notified when the status of a command or a code block within a command changes. In order to provide you with several different integration options, you can receive notifications via CloudWatch Events or via Amazon Simple Notification Service (SNS).

These notifications will allow you to use EC2 Run Command in true building block fashion. You can programmatically invoke commands and then process the results as they arrive. For example, you could create and run a command that captures the contents of important system files and metrics on each instance. When the command is run, EC2 Run Command will save the output in S3. Your notification handler can retrieve the object from S3, scan it for items of interest or concern, and then raise an alert if something appears to be amiss.

Monitoring Executing Using Amazon SNS
Let’s run up a command on some EC2 instances and monitor the progress using SNS.

Following the directions (Monitoring Commands), I created an S3 bucket (jbarr-run-output), an SNS topic (command-status), and an IAM role (RunCommandNotifySNS) that allows the on-instance agent to send notifications on my behalf. I also subscribed my email address to the SNS topic, and entered the command:

And specified the bucket, topic, and role (further down on the Run a command page):

I chose All so that I would be notified of every possible status change (In Progress, Success, Timed Out, Cancelled, and Failed) and Invocation so that I would receive notifications as the status of each instance chances. I could have chosen to receive notifications at the command level (representing all of the instances) by selecting Command instead of Invocation.

I clicked on Run and received a sequence of emails as the commands were executed on each of the instances that I selected. Here’s a sample:

In a real-world environment you would receive and process these notifications programmatically.

Monitoring Execution Using CloudWatch Events
I can also monitor the execution of my commands using CloudWatch Events. I can send the notifications to an AWS Lambda functioon, an SQS queue, or a Amazon Kinesis stream.

For illustrative purposes, I used a very simple Lambda function:

I created a rule that would invoke the function for all notifications issued by the Run Command (as you can see below, I could have been more specific if necessary):

I saved the rule and ran another command, and then checked the CloudWatch metrics a few seconds later:

I also checked the CloudWatch log and inspected the output from my code:

Available Now
This feature is available now and you can start using it today.

Monitoring via SNS is available in all AWS Regions except Asia Pacific (Mumbai) and AWS GovCloud (US). Monitoring via CloudWatch Events is available in all AWS Regions except Asia Pacific (Mumbai), China (Beijing), and AWS GovCloud (US).

Jeff;

 

New – Worldwide Delivery of Amazon SNS Messages via SMS

by Jeff Barr | on | in Amazon SNS, Mobile Development | | Comments

Many applications use the Amazon SNS to deliver SMS messages to individuals or to groups. Because SNS is simple and easy to use, you can add push notifications to your application with a modest amount of code.

Today we are making SNS even more useful by adding support for worldwide delivery of SMS messages. You can now use SNS to send SMS text messages to mobile phone numbers in more than 200 countries.

Along with this change we are also adding several other features to this feature:

  • No Opt-In – Message recipients no longer need to opt-in to SNS before they can receive messages.
  • Additional Regions – You can now send messages with SMS delivery from the US East (Northern Virginia), US West (Oregon), Europe (Ireland), Asia Pacific (Tokyo), Asia Pacific (Sydney), and Asia Pacific (Singapore) Regions.
  • Direct Publishing – You can now send messages to a phone number without first subscribing the number to an SNS topic. If a user replies with “STOP”, they will be added to the opt-out list associated with your AWS account (this feature applies to messages sent to phone numbers located in countries where local regulations require opt-out capabilities).
  • Opt-Out Management – You can now manage opt-out numbers using the AWS Management Console, API, or CLI (learn more).
  • Delivery Status – You can now enable logging of delivery status for SMS messages and access the status in a pair of CloudWatch Log Groups (Successful and Failed).
  • Limit – You can now set spending limits on a per-account, per-month basis. You can also set a per-message spending limit.
  • Usage Reports – You can now access daily SMS usage reports, with information on successful and failed deliveries (learn more).
  • Long/Short Code Pool – Messages sent via SNS will now appear to come from one of several long or short codes maintained by Amazon SNS. Multiple messages sent from one account to a particular phone number will all be sent from the same long or short code.

To learn more about all of these new features, read Amazon SNS Worldwide SMS Delivery on the AWS Mobile Development Blog.

Jeff;

New – Notifications for AWS CodeCommit

by Jeff Barr | on | in Amazon SNS, AWS CodeCommit, AWS Lambda | | Comments

AWS CodeCommit is a fully-managed source control service that makes it easy for you to host a secure and highly scalable private Git repository.Today we are making CodeCommit even more useful by adding support for repository triggers. You can use these triggers to integrate your existing unit tests and deployment tools into your source code management workflow. Because triggers are efficient and scalable, they are more broadly applicable than a model that is built around polling for changes. I believe that you will find these triggers to be helpful as you move toward a development methodology based on  Continuous Integration and Continuous Delivery.

All About Notifications
You can create up to 10 triggers for each of your CodeCommit repositories. The triggers are activated in response to actions on the repository including code pushes, branch/tag creation, and branch/tag deletion. Triggers can be set to run for a specific branch of a repository or for all branches.

Triggers can send a notification to an Amazon Simple Notification Service (SNS) topic or can invoke a AWS Lambda function. Each trigger can also be augmented with custom data (an uninterpreted string) that you can use to distinguish the trigger from others that run for the same event. You can use triggers to subscribe to repository events through email or SMS. You can wire up SNS to SQS and queue up jobs for your CI/CD tools, or you can use SNS to activate webhooks provided by your tools. In any case, the actions you designate will be triggered by the changes in your CodeCommit repository. You can also use Lambda functions to trigger builds, check syntax, capture code complexity metrics, measure developer producitivity (less is more, of course), and so forth. My colleagues have also come up with some off-unusual ideas that you can find at the end of this post!

You can create, view, and manage your triggers from the AWS Management Console, AWS Command Line Interface (CLI), or via the CodeCommit API.  I used the Console. The navigation column on the left now includes an entry for Triggers:

I simply click on Create Trigger to get started. Then I select the event (or events), pick the branch (or all branches), and fill in the details that are needed to publish a notification or invoke a Lambda function:

Here’s how I choose the events and branches of interest:

 

Then I point to my SNS topic or Lambda function (after ensuring that the proper permissions are in place), use Test Trigger to make sure that it all works as expected, and click on Create.

You can use Test Trigger to verify that your IAM permissions are working as expected. For example, here’s an error that I triggered on purpose:

I fixed this by reading How to Allow AWS CodeCommit to run the Function in the documentation!

Available Now
This new functionality is available now and you can start using it today.To learn more, read about Managing Triggers for an AWS CodeCommit Repository.

My colleague Clare Liguori suggested some creative uses for CodeCommit triggers, above and beyond the usual integration with your CI/CD process:

  • Video Deployment – Have your Lambda function check to see if a new video or a new version of an existing video has been committed, and deploy the video to YouTube.
  • Party Time – Automatically throw and cater a party (using APIs for sandwiches, pizza, and beer) when you deploy a new release.
  • Advertise Releases – When a new release is ready, automatically generate and run a Facebook ad and publicize the release on social media.

I am looking forward to hearing about the creative ways that you make use of these triggers within your development process. Leave me a comment and let me know!

Jeff;

 

Amazon SNS – Now With Enhanced Support for iOS 8

by Jeff Barr | on | in Amazon SNS, Mobile Development | | Comments

Amazon Simple Notification Service (SNS) is a fast, flexible, managed push service that makes it simple and cost-effective to push notifications to Apple, Google, Fire OS, and Windows devices, as well as Android devices in China with Baidu Cloud Push.

With yesterday’s launch of iOS 8, Apple has introduced new features that enable some rich new use cases for mobile push notifications — interactive push notifications, third party widgets, and larger (2 KB) payloads. With these new features, you can message and engage your customers even when your app is not currently active.

Today, we are pleased to announce support for the new mobile push capabilities announced with iOS 8. We are publishing a new iOS 8 Sample App that demonstrates how these new features can be implemented with SNS, and have also implemented support for larger 2KB payloads.

Interactive Push Notifications and Widgets
The new iOS 8 Interactive Push Notifications lets apps present custom options in the push banner. These custom options allow users to interact with the app without moving it to the foreground, resulting in a less intrusive and more fluid experience.

Widgets are miniature versions of your mobile app that users can dock to the Notifications Center. Interactive notifications and widgets are great for quick extensions to your core app that extend the experience throughout the day.

Both interactive push notifications and widgets can be implemented by adding additional parameters to your push notifications. You can use SNS platform-specific payloads to specify those additional parameters for your iOS customers.

Suppose you have the following message payload (yes, this is JSON within JSON):


{"APNS":"{\"aps\":{\"alert\":\"MESSAGE\",\"category\":\"MESSAGE_CATEGORY\"} }"}

Your iOS app can set up a notification category using the following code:


UIMutableUserNotificationCategory *messageCategory = [[UIMutableUserNotificationCategory alloc] init];
messageCategory.identifier = @"MESSAGE_CATEGORY";

[messageCategory setActions:@[readAction, ignoreAction, deleteAction]
  forContext:UIUserNotificationActionContextDefault];

Larger (2 KB) Payloads
In the past, the Apple Push Notification service (APNs) would only accept up to 256 bytes. As of yesterday’s launch, APNs can now accept payloads as large as 2 KB. This change was introduced along with iOS 8, but 2 KB payloads will also work for devices running older versions of iOS. You can use the larger payloads to deliver richer messages and instructions to your app, which is particularly useful for silent notifications that get delivered to your app in the background. SNS also supports 2 KB APNs payloads beginning today.

Ready Now
Our customers tell us that one major benefit of SNS is they no longer have to keep up with changes to the APIs of each messaging platform. These SNS enhancements introduced today are an example of new features that become quickly available to our customers without added server-side development.

To learn more about getting started with SNS Mobile Push, and to download our new iOS 8 sample app (which includes the full version of the code example mentioned above), simply click here.

Jeff;

Amazon SNS Update – Large Topics and MPNS Authenticated Mode

by Jeff Barr | on | in Amazon SNS | | Comments

Amazon Simple Notification Service (SNS) is a fast and flexible push messaging service. You can easily send messages to Apple, Google, Fire OS and Windows devices, including Android devices in China (via Baidu Cloud Push).

Today we are enhancing SNS with support for large topics (more than 10,000 subscribers) and authenticated delivery to MPNS (Microsoft Push Notification Service).

Large Topics
SNS offers two publish modes. First, you can push messages directly to specific mobile devices. Second, you can create an SNS topic, provide your customers with a mechanism to allow them to subscribe to the topic, and then publish messages to the topic with a single API call. This mode is great for broadcasting breaking news, announcing flash deals, and announcing in-game events or new features. You can combine customers from different platforms in the same topic and you can send a specific payload to each platform (for example, one for iOS and another for Android), again in a single call. Suppose you have created the following topic:

With the ARN for the topic (arn:aws:sns:us-west-2:xxxxxxxxxxxx:amazon-sns) in hand, here’s how you publish a message to all of the subscribers:


$result = $client->publish(array(
    'TopicArn' => 'arn:aws:sns:us-west-2:xxxxxxxxxxxx:amazon-sns',
    // Message is required
    'Message' => 'Hello Subscribers',
    'Subject' => 'Hello'
));

Today we are lifting the limit of 10,000 subscriptions per SNS topic; you can now create as many as you need and no longer need to partition large subscription lists across multiple topics. This has been a frequent request from AWS customers that use SNS to build news and media sharing applications.

There is an administrative limit of 10 million subscriptions per topic, but we’ll happily raise it if you expect to have more subscribers for a single topic. Fill out the Contact Us form, select SNS, and we’ll take good care of you!

Authenticated Delivery to MPNS
Microsoft Push Notification Service (MPNS) is the push notification relay service for Windows Phone devices prior to Windows 8.1. SNS now supports authenticated delivery to MPNS. In this mode, MPNS does not enforce any limitations on the number of notifications that can be sent to a channel in any given day (per the documentation on Windows Phone Push Mode, there’s a daily limit of 500 unauthenticated push notifications per channel).

If you require this functionality for devices that run Windows 8.1 and above, please consider using Amazon SNS for Windows Notification Service (WNS).

Jeff;

Amazon SNS TTL (Time to Live) Control

by Jeff Barr | on | in Amazon SNS | | Comments

Amazon Simple Notification Service (SNS) is a fast and flexible push messaging service.

Many of the messages that you can send with SNS are relevant or valuable for a limited period of time. Sports scores, weather notifications, and “flash sale” announcements can all get stale in a short period of time. In situations where devices are offline or disconnected, flooding the user with outdated messages when communication is reestablished makes for a poor user experience.

In order to allow you to build great applications that behave well in an environment with real-time information and intermittent connections, Amazon SNS now allows you to set a TTL (Time to Live) value of up to two weeks for each message. Messages that remain undelivered for the given period of time (expressed as a number of seconds since the message was published) will expire and will not be delivered. Read about How to Use Time to Live Values.

Most of the underlying push services support TTL in some fashion, but each one uses a unique set of APIs and data formats. With today’s release, SNS now lets you use a common format and the cross-platform Publish API to define TTL values for iOS, Android, Fire OS, Windows WNS, and Baidu endpoints (Windows MPNS does not support TTL)

You can set the TTL through the SNS API or the AWS Management Console:

This new feature, in conjunction with our recent launch of mobile push support for Windows (phone and desktop) and Baidu Cloud Push (Android), will help you to build helpful, user-friendly applications that reach a broad user base without having to deal with a multitude of messaging providers.

Jeff;

Amazon SNS Mobile Push Update – Support for Baidu Cloud Push (China), MPNS, and WNS

by Jeff Barr | on | in Amazon SNS, Mobile | | Comments

The Amazon Simple Notification Service (SNS) is a fast and flexible push messaging service. You can send messages to a wide variety of mobile devices. You can deliver notifications via SMS text messages or email, and to Amazon SQS queues and HTTP endpoints.

Today we are enhancing SNS with support for Baidu Cloud Push in China, Microsoft Push Notification Service (MPNS) for Windows Phone 7 and higher, and Windows Push Notification Service (WNS) for Windows Store apps and Windows Phone 8.1 and higher.

Support for Three Additional Services
Amazon SNS can already push messages to mobile devices through the Apple Push Notification Service (APNS), Google Cloud Messaging for Android (GCM), and Amazon Device Messaging (ADM) services. Today’s addition of support for Baidu Cloud Push and a pair of Windows services means that you can now send messages to more users and to additional types of devices.

Baidu Cloud Push – Delivering push notifications in China on Android is more complex than other parts of the world. With many different app stores and push services, our customers are now realizing that this is a difficult issue to solve and have asked for additional options. In order to allow app developers to address this large and rapidly growing market and to avoid having to customize their code for every app store, Amazon SNS can now deliver messages to users in China through the Baidu Cloud Push, regardless of the app store that they use.

Messages can originate in any of the public AWS Regions and will be routed appropriately. You will need to create an account, register as a developer, and obtain the appropriate identifiers (userId and channelId) in order to use this service. Full details, including a walkthrough of the Simplified Chinese registration and console pages, are available in the Getting Started With Baidu Cloud Push documentation.

Windows Push Notification Service – Also known as WNS, this service lets you send messages and updates to Windows 8 desktop applications and Windows Phone 8.1 (and newer) apps via the Windows Store. You will need to register your application and generate a unique SID in order to make use of this service. For more information, read the Getting Started With WNS.

Microsoft Push Notification Service for Windows Phones – Also known as MPNS, this service allows you to send messages to Windows Phone 7 (and above) apps. To learn more, read the Getting Started With MPNS documentation.

The SNS Mobile Push Sample App is referenced in the documentation. It will show you how to send push notifications to endpoints of each supported type.

Send a Message Today
These new features are available now and you can start using them today. The Getting Started With Amazon SNS documentation is a good place to start!

The AWS SDKs already support this feature and can be installed directly or through the language-specific package managers such as Maven, NuGet, and RubyGems.

Jeff;

Price Reductions & Expanded Free Tier for the Simple Queue Service (SQS) and the Simple Notification Service (SNS)

by Jeff Barr | on | in Amazon Simple Notification Service, Amazon SNS, Amazon SQS, Price Reduction | | Comments

We launched the Amazon Simple Queue Service (SQS) in 2004 and the Amazon Simple Notification Service (SNS) in 2010.

Our customers have constantly discovered powerful new ways to build more scalable, elastic and reliable applications with these building blocks. We learned, for example, that some customers use SQS as a buffer ahead of databases and other services. Other customers combine SNS + SQS to transmit identical messages to multiple queues.

Over time, we’ve optimized our own systems in order to make SNS and SQS available to even more customers. The goal has always been to charge less for processing the same volume of messages. We did this with the SQS Batch API in 2011 and more recently with long polling for SQS and 64KB payloads for SNS.

Today we are making SQS and SNS an even better value:

  • SQS API prices will decrease by 50%, to $0.50 per million API requests.
  • SNS API prices will decrease by 17%, to $0.50 per million API requests.
  • The SQS and SNS free tiers will each expand to 1 million free API requests per month, up 10x from 100K requests per month.

The new prices take effect on March 1, 2013 and are applicable in all AWS Regions with the exception of the AWS GovCloud (US).

If you’d like to learn more about SQS and SNS, check out this video from AWS re:Invent:

— Jeff (with lots of help from Jon Turow, Senior Product Manager for SQS and SNS);

New Features for Amazon SNS – Delivery Policies and Message Formatting

by Jeff Barr | on | in Amazon SNS | | Comments

We have added two new features to Amazon SNS to give you additional control over the content and delivery of your messages. As a brief reminder, SNS allows you to create named Topics, subscribe to Topics (with delivery via email, HTTP / HTTPS, an SMS message, or to an SQS queue), and to publish messages to Topics.

SNS Delivery Policies
The SNS Delivery Policies give you options to control the delivery rate and error handling for each SNS endpoint. You can, for example, use a Delivery Policy to avoid overwhelming a particular endpoint with a sudden barrage of messages.

Delivery Policies can be set for Topics and for the endpoints associated with a particular Topic. Each Delivery Policy contains a Retry Policy and a Throttle Policy. With this release, the policies are effective for the HTTP and HTTPS Subscription types.

The Retry Policy can specify the following options:

  • minDelayTarget – Minimum delay for a retry.
  • maxDelayTarget – Maximum delay for a retry.
  • numNoDelayRetries  – Number of retries to be done with no delay (as soon as possible).
  • numMinDelayRetries – Number of retries to be done at minDelayTarget intervals before initiating the backoff function.
  • numMaxDelayRetries – Number of retries to be done at maxTargetDelay during the backoff function.
  • backoffFunction – Model for backoff between retries: Linear, Exponential, or Arithmetic.

There are default, minimum, and maxium values for each option; see the SNS documentation for more information.

The Throttle Policy can specify one option:

  • maxReceivesPerSecond – Maximum number of delivery attempts per second per Subscription.

All attempts to deliver a message are based on an “effective Delivery Policy” which combines the default policy, any policy values set for the Topic, and any policy values set for the Subscription endpoint. Values left unspecified at the Subscription level will be inherited from the Topic’s Delivery Policy and then from the default policy.

SNS Message Formatting
This feature gives you the ability to publish a message that contains content that is specific to each type of Subscription. You could, for example, send a short message to an SMS endpoint and a longer message to an email endpoint.

To use this feature, you set the new MessageStructure parameter to “json” when you call the SNS publish function.  The associated message body must contain a JSON object with a default message body and optional message bodies for other protocols:

{
  “default” : “Server busy.” ,
  “email”   : “Dear Jeff, your server is really busy and you should investigate. Best Regards, AWS.” ,
  “sms”     : “ALERT! Server Busy!!” ,
  “https”     : “ServerBusy”
}

The default entry will be used for any protocol (http and sqs in this case) that does not have an entry of its own.

— Jeff;

PS – Please feel free to share this with your friends via the Like button below!

Behind the Scenes of the AWS Jobs Page, or Scope Creep in Action

by Jeff Barr | on | in Amazon CloudWatch, Amazon EC2, Amazon S3, Amazon SDB, Amazon Simple Notification Service, Amazon SimpleDB, Amazon SNS, Amazon SQS, Auto Scaling | | Comments

The AWS team is growing rapidly and we’re all doing our best to find, interview, and hire the best people for each job. In order to do my part to grow our team, I started to list the most interesting and relevant open jobs at the end of my blog posts. At first I searched our main job site for openings. I’m not a big fan of that site; it serves its purpose but the user interface is oriented toward low-volume searching. I write a lot of blog posts and I needed something better and faster.

Over a year ago I decided to scrape all of the jobs on the site and store them in a SimpleDB domain for easy querying. I wrote a short PHP program to do this. The program takes the three main search URLs (US, UK, and Europe/Asia/South Africa) and downloads the search results from each one in turn. Each set of results consists of a list of URLs to the actual job pages (e.g. Mgr – AWS Dev Support).

Early versions of my code downloaded the job pages sequentially. Since there are now 370 open jobs, this took a few minutes to run and I became impatient. I found Pete Warden’s ParallelCurl and adapted my code to use it. I was now able to fetch and process up to 16 job pages at a time, greatly reducing the time spent in the crawl phase.

// Fetch multple job pages concurrently using ParallelCurl
for ( $i = 0 ; $i < count ( $JobLinks ) ; $i ++ )
{
$PC -> startRequest ( $JobLinks [ $i ] [ ‘Link’ ] , ‘JobPageFetched’ , $i ) ;
}
$PC -> finishAllRequests ( ) ;

My code also had to parse the job pages and to handle five different formatting variations. Once the pages were parsed it was easy to write the jobs to a SimpleDB domain using the AWS SDK for PHP.

Now that I had the data at hand, it was time to do something interesting with it. My first attempt at visualization included a tag cloud and some jQuery code to show the jobs that matched a tag:

I was never able to get this page to work as desired. There were some potential scalability issues because all of the jobs were loaded (but hidden) so I decided to abandon this approach.

I gave upon the fancy dynamic presentation and generated a simple static page (stored in Amazon S3, of course) instead, grouping the jobs by city:

My code uses the data stored in the SimpleDB domain to identify jobs that have appeared since the previous run. The new jobs are highlighted in the yellow box at the top of the page.

I set up a cron job on an EC2 instance to run my code once per day. In order to make sure that the code ran as expected, I decided to have it send me an email at the conclusion of the run. Instead of wiring my email address in to the code, I created an SNS (Simple Notification Service) topic and subscribed to it. When SNS added support for SMS last month, I subscribed my phone number to the same topic.

I found the daily text message to be reassuring, and I decided to take it even further. I set up a second topic and published a notification to it for each new job, in human readable, plain-text form.

The next step seemed obvious. With all of this data in hand, I could generate a tweet for each new job. I started to write the code for this and then discovered that I was reinventing a well-rounded wheel! After a quick conversation with my colleague Matt Wood, it turned out that he already had the right mechanism in place to publish a tweet for each new job.

Matt subscribed an SQS queue to my per-job notification topic. He used a CloudWatch alarm to detect a non-empty queue, and used the alarm to fire up an EC2 instance via Auto Scaling. When the queue is empty, a second alarm reduces the capacity of the group, thereby terminating the instance.

Being more clever than I, Matt used an AWS CloudFormation template to create and wire up all of the moving parts:

  “Resources” : {
“ProcessorInstance” : {
“Type” : “AWS::AutoScaling::AutoScalingGroup” ,
“Properties” : {
“AvailabilityZones” : { “Fn::GetAZs” : “” } ,
“LaunchConfigurationName” : { “Ref” : “LaunchConfig” } ,
“MinSize” : “0” ,
“MaxSize” : “1” ,
“Cooldown” : “300” ,
“NotificationConfiguration” : {
“TopicARN” : { “Ref” : “EmailTopic” } ,
“NotificationTypes” : [ “autoscaling:EC2_INSTANCE_LAUNCH” ,
“autoscaling:EC2_INSTANCE_LAUNCH_ERROR” ,
“autoscaling:EC2_INSTANCE_TERMINATE” ,
“autoscaling:EC2_INSTANCE_TERMINATE_ERROR” ]
}
}
} ,

You can also view and download the full template.

The instance used to process the new job positions runs a single Ruby script, and is bootstrapped from a standard base Amazon Linux AMI using CloudFormation.

The CloudFormation template passes in a simple bootstrap script using instance User Data, taking advantage of the cloud-init daemon which runs at startup on the Amazon Linux AMI. This in turn triggers CloudFormations own cfn-init process, which configures the instance for use based on information in the CloudFormation template.

A collection of packages are installed via the yum and rubygems package managers (including the AWS SDK for Ruby), the processing script is downloaded and installed from S3, and a simple, YAML format configuration file is written to the instance which contains keys, Twitter configuration details and queue names used by the processing script.

queue. poll ( :poll_interval => 10 ) do |msg |

notification = TwitterNotification.new(msg.body)

begin
client.update(notification.update)
rescue Exception => e
log.debug “Error posting to Twitter: #{e}”

  else
    log.debug “Posted: #{notification.update}”
end

end

The resulting tweets show up on the AWSCloud Twitter account.

At a certain point, we decided to add some geo-sophistication to the process. My code already identified the location of each job, so it was a simple matter to pass this along to Matt’s code. Given that I am located in Seattle and he’s in Cambridge (UK, not Massachusetts), we didn’t want to coordinate any type of switchover. Instead, I simple created another SNS topic and posted JSON-formatted messages to it. This loose coupling allowed Matt to make the switch at a time convenient to him.

So, without any master plan in place, Matt and I have managed to create a clean system for finding, publishing, and broadcasting new AWS jobs. We made use of the following AWS technologies:

Here is a diagram to show you how it all fits together:

If you want to hook in to the job processing system, here are the SNS topic IDs:

  • Run complete – arn:aws:sns:us-east-1:348414629041:aws-jobs-process
  • New job found (human readable) – arn:aws:sns:us-east-1:348414629041:aws-new-job
  • New job found (JSON) – arn:aws:sns:us-east-1:348414629041:aws-new-job-json

The topics are all set to be publicly readable so you can subscribe to them without any help from me. If you build something interesting, please feel free to post a comment so that I know about it.

The point of all of this is to make sure that you can track the newest AWS jobs. Please follow @AWSCloud take a look at the list of All AWS Jobs.

— Jeff (with lots of help from Matt);