Category: Amazon S3


Amazon S3 – 566 Billion Objects, 370,000 Requests/Second, and Hiring!

Our customers continue to make very heavy use of Amazon S3. We now process up to 370,000 S3 requests per second.

Many of these are PUT requests, representing new data that is flowing in to S3. As of the end of the third quarter of 2011, there are 566 billion (566,000,000,000) objects in S3. Here’s a growth chart:

We’ve doubled the object count in just nine months (the other data points are from Q4). My math skills are a bit rusty but I definitely know exponential growth when I see it!

Designing, building, and running a large-scale distributed service like this isn’t for the faint of heart. We’re very proud of what we have done, but we have plans to do a whole lot more. If you are ready to push the state of the art in this area, consider applying for one of the open positions on the S3 team. Here’s a sampling (these jobs are all based in Seattle):

We also have a number of business development positions open:

— Jeff;

AWS Summer Startups: Discovr

Over the summer months, we’d like to share a few stories from startups around the world: what are they working on and how they are using the cloud to get things done. Today, we’re profiling Filter Squad from Perth, Australia!

Discovr team


In one of Werner Vogels‘ many travels through Australia this summer, he tweeted about a lean startup he had met, Filter Squad. Lean startups, not to be confused with bootstrapped startups, are built by adapting agile software development methodologies to business. Some of the concepts include building a minimum viable product, testing business assumptions with real market data, measuring results, and then quickly iterating or pivoting depending on what that data shows you. The concept of a “lean startup” was coined by entrepreneur & author Eric Ries

I reached out and spoke to Stuart Hall, CTO of Filter Squad, and creator of, among others, the Discovr Apps and Discovr Music for the iPhone & iPad.

Meet Filter Squad
Filter Squad is a startup focused on building apps that find what you like, according to CTO Stuart Hall. They began with a #1 selling iPad/iPhone app called Discovr Music in January 2011 and expanded the discovery product suite to include Discovr App in June 2011, which has been a #1 category application in 17 countries. As the name implies, Discovr Music makes it easy for users to find music they like based on their preferences, while Discovr App recommends apps the user might like based on the ones you’re already using. We have been extremely happy with AWS and we also plan to use it for our future products. We are big fans of products such as Amazon RDS and the Elastic Load Balancer to give a complete app scaling solution with Amazon EC2, says Stuart.  
Take a look at the Discovr Music app review from Fox News:

AWS & Lean Startups
Because we are a small, lean team, we were looking for a hosting solution that was going to be easy for us to setup, be reliable, and be easy to scale up and down throughout our product iterations. We looked at a large number of providers, but AWS stood out immediately for a number of reasons:
  • Low maintenance
  • Easy to scale
  • Simple to setup
  • Provided good redundancy
We couldn’t find anyone else who could match the AWS products and price. The number of other large, successful companies also using the service was very reassuring.
Building a Native iPhone/iPad App on AWS
Native mobile apps often need server-side components to create a rich user experience. For our Discovr Apps and Discover Music apps, we have used the following AWS products: 
  • Amazon EC2 – because we had no idea of the market reaction to the application when we launched, flexibility in adding and removing virtual servers based on demand was key. 
  • Amazon RDS – we needed a database that would also be easy to scale and be easy to maintain. Amazon RDS provided easy scaling, easy replication for slave instances, and a system where minor software updates are handled entirely by AWS.
  • Amazon S3 – S3 provides a great and cheap way to host static resources, one with which we had worked before and found ideal for our use case.
  • Amazon Elastic Load Balancer –  the load balancer is provided straight out of the box: it doesn’t require any installing and it needs very little configuration. The load balancer provides built in health checks and takes out instances that are not behaving. Elastic load balancing has been faultless since we launched.
  • Caching: the only thing missing was a caching solution, which AWS has since launched and we will be soon moving to. This was also a big consideration, the pace at which AWS are iterating and improving their service matches our philosophy to application development.
We are also big fans of New Relic for monitoring our AWS instance performance. 

 

Scaling up Ruby on Rails with AWS
We use Ruby on Rails server side, Objective C, and Java for client side. More details of our stack, including our architecture and test data, can be seen detailed on our blog

Words of Wisdom for Other Startups:
Understand that you can do it from anywhere, you dont have to be based in Silicon Valley, or even a big city. With the help of the internet and web services such as the AWS cloud, anyone can deliver great products from anywhere in the world.
For example were based in Perth, Australia. Its a five hour flight to Sydney and our hometown is most definitely not the tech capital of the world! To sum up:
  • Build a great product, then don’t forget to market it!
  • Treat your customers like precious gold.
  • Make it easy for your customers to talk to you and listen to what they say.
  • Cross-promote your app with other apps that youve also built.

We’ve collected some lessons learned on our blog: how we got 250k downloads in 4 days.

——————————————————

8 Days Left to Enter Your Startup in the AWS Start-up Challenge!
This year’s AWS Start-up Challenge is a worldwide competition with prizes at all levels, including up to $100,000 in cash, AWS credits, and more for the grand prize winner. Learn more and enter today!

You can also follow @AWSStartups on Twitter for startup-related updates.

 -rodica

Facebook Developer Update: Meet RootMusic, Funzio, and 50Cubes

In honor of today’s Facebook Developer Conference, I’d like to recognize the success of our existing Facebook app developers and invite even more developers to kick-start their next Facebook app project with Amazon Web Services.

Quick Numbers
We crunched some numbers and found out that 70% of the 50 most popular Facebook apps leverage one or more AWS services. Many of their developers rely on AWS to provide them with compute, network, storage, database and messaging services on a pay-as-you-go basis. In addition to Zyngas popular FarmVille and CafeWorld, or games from Playfish and Wooga, many of the most exciting and popular Facebook apps are also running on AWS.

Here are a few examples:

RootMusic‘s BandPage app (currently the #1 Music App on Facebook, and #8 overall app on Facebook) helps bands and musicians build fan pages that will attract and hold the interest of an audience. RootMusic enables artists to tap into the passion their fans feel for their art and keep them engaged with an interactive experience. More than 250,000 bands of all shapes and sizes, from Rihanna and Arctic Monkeys, to bands you haven’t heard of yet but may soon discover, have already made RootMusics BandPage their central online space for connecting with their fans. Artists use it to share music, release special edition songs/albums here, share photos, and list events/shows. BandPage now supports 30 million monthly active users from all over the world. Behind all the capabilities that ignite BandPages music fan communities lies a well-thought out, highly-distributed and highly-scalable backend, powered by Amazon Web Services:

In 20 seconds, we can double our server capacity. In a high-growth environment like ours, it’s very important for us to trust that we have the best support to give to the music community around the world. Five years ago, we would have crashed and been down without knowing when we would be back. Now, because of Amazons continued innovation, we can provide the best technology and scale to serve music communities needs around the world, Christopher Tholen, RootMusic CTO.


Funzio‘s Crime City is #7 in the top 10 Facebook apps, and its the highest rated Facebook game to reach 1 million daily users with an average user rating of 4.9 out of 5. Crime City currently has 5.5 million monthly active users, with 10 million monthly active users at its peak. The iPhone version was recently listed among the top 5 games in the Apple Appstore and #1 free game in 11 countries and counting. Crime City sports modern, 3D-like graphics that look great on both Facebook and iPhone, and has a collection of hundreds of virtual items that players can collect.

Powering this incredibly rich user experience across multiple platforms is their business acumen in promoting the app, as well as a strong backend that leverages many AWS products to serve their viral and highly active user base. Funzio uses Amazon EC2 to quickly scale up and down based on demand, Amazon RDS to store game and current state information. They use Amazon CloudFront to optimize the delivery to a global, widely-distributed audience and to meet Facebook’s SSL certificate requirements.

At Funzio, we use AWS exclusively to host the infrastructure for our games. When developing social games, you need to be ready for that traffic burst for a hit game in a moment’s notice. AWS provides us with the flexibility to quickly and efficiently scale our applications at all layers, from increasing database capacity in RDS, to adding more application or caching servers within minutes in EC2. Amazon’s cloud services allow us to focus our efforts on developing quality games and not on worrying about managing our technology operations. – Ram Gudavalli, Funzio CTO.


50Cubes, the creator of Mall World, is a startup that has developed one of the most highly-regarded and longer-running successful female focused social game on Facebook. With over 5 million monthly active users, Mall World has a track record of being not only one of the first but also the top game of its kind for the past 1.5 years and continues to entice users world-wide.

50Cubes powers Mall World and other games they developed with a suite of AWS products. Out of these, they value the Amazon Auto-scaling and EBS features the most these products helps them effortlessly scale up and down their exclusive use of Amazon EC2 instances with user demand. Their database clusters are a mix of MySQL and other key value storage databases, all hosted and managed by the team on Amazon EC2 using EBS for Cloud Storage.

One thing that impresses me the most about AWS services is that they have rapidly iterated and improved their products and services over the past year and half, executing almost like a startup of our scale.” – Fred Jin, 50cubes CTO.

Get Started: Your Facebook App, Powered by AWS
Doug Purdy, Director of Developer Relations at Facebook, said:

AWS is great for Facebook developers you can start small, test and prove your ideas. As your app grows, you can easily scale up your resources to keep your users engaged and connected. AWS allows developers to build highly-available, highly-scalable, cost-efficient apps that provide the type of rich and responsive user experiences that our global audience has grown to expect.

To make it as easy as possible for you to get started, we’ve updated our Building Facebook Apps on AWS page. We have also improved and refreshed our Facebook App AMI. The new AMI uses AWS CloudFormation to install the latest versions of the Facebook PHP SDK and the AWS SDK for PHP at startup time. If you want to learn more about developing AWS applications in PHP, feel free to check out the free chapters of my AWS programming book (or buy a complete copy).

— Jeff;

AWS Summer Startups: ShowNearby

 

Over the summer months, we’d like to share a few stories from startups around the world: what are they working on and how they are using the cloud to get things done. Today, we’re profiling ShowNearby, from Singapore!

ShowNearby team

 
About ShowNearby

ShowNearby is a leading location-based service in Singapore and an early adopter of the Android platform. Unlike many mobile apps out there, ShowNearby started with deployment on Android and then moved on to the iPhone by mid 2010 and Blackberry by fall of 2010. Today, the ShowNearby flagship app is available on Android, iPhone and Blackberry and reports approximately 100 Million mobile searches conducted across all its platforms.

I spoke to Stephen Bylo, Senior Cloud Architect at ShowNearby, who added a bit of color to the experience of running, planning, and meeting the requirements of a popular mobile app. If you’re not from Singapore and would like to see the app, here’s a quick video demo of ShowNearby.
Surviving Our Success with AWS

Due to the success of our application, we had a very big growth in a short period of time. When we launched on the popular platforms of iOS and subsequently BlackBerry, we were blown away by the huge surge of users that started using ShowNearby. In fact in December of 2010, ShowNearby became the top downloaded app in the App store, edging out thousands of other popular free apps in Singapore! It was then that we realized we needed a scalable solution to handle the increasing load and strain on our servers that our existing provider was unable to provide.

Our infrastructure at the time was hosted with a local service provider, but was unable to cope with the high traffic peaks we were facing.We analyzed a few vendors and decided to go ahead with Amazon because of it’s reliability, high availability, range of services and pricing, but mostly because of its solid customer support.

As part of our deployment, we added AWS services incrementally. Currently we use extensively Amazon EC2 instances with auto scaling, Relational Database Service (RDS), Simple Queue Service (SQS), Cloudwatch and Simple Storage Service (S3).

Next item on our list is to focus on automating the deployment of infrastructure environments with cloud formations, as well as optimizing content delivery globally with Cloudfront.

Choosing the Tech Stack That Makes Business Sense

ShowNearby currently leverages on the LAMP stack for most our web services. Delivery of accurate, always available, location based data is ShowNearbys top priority.That is why we chose AWS.

Other important things why to choose cloud/AWS: Speed and agility to create and tear down infrastructure as and when it is needed.  Good and fast network accessibility for our app.  Ability to scale up and out when needed.  Ability to duplicate infrastructure into new regions.

Reaching Automation Nirvana with AWS

We chose to use AWSs Linux based AMI and dynamically build on top of it using well defined, automatic configuration.  Now, every time an instance is started, we are sure the infrastructure is always in a known state.  Admittedly, a lot of hard work is involved to achieve Automation Nirvana, but knowing precisely what works at the end of the day helps us sleep at night.

  • We use Amazon S3 to store infrastructure configuration and user provided content/images. ShowNearbys business is currently in, and marching into new, regions, so S3 is a natural precursor to AWSs CloudFront content distribution service.
  • We use SQS to help process user behaviour and to determine usage patterns.  
  • We use this to provide our dear users with a better, and hopefully, more personalised experience.We use spot instances for early development & testing servers.
  • We use CloudWatch extensively – how could we do without it?
  • We use RDS, for our hosted mySQL databased needs, of course
  • We use the command-line and PHP AWS API tools to a large extent, which provides us increased business agility.

Words of Wisdom for Mobile Startups

We would tell them to find partners who can be good friends at the same time. The race is long and tough, so better do it enjoying every step of the way. There is a window of opportunity in Asia now open to unleash your full potential, show what you are capable of and you’ll be rewarded.

Today, if we need to refresh or update a web application, we restart new instances and flush out the old.  Moving forward, we are looking into reducing the time between releases still further and so, we are working to improve on our already solid infrastructure and configuration management.  Further automation in the form of Chef and/or Puppet or similar is being investigated.

——————————————————

Enter Your Startup in the AWS Start-up Challenge!
This year’s AWS Start-up Challenge is a worldwide competition with prizes at all levels, including up to $100,000 in cash, AWS credits, and more for the grand prize winner. 7 Finalists receive $10,000 in AWS credits and 5 regional semi-finalists receive $2,500 in AWS credits. All eligible entries receive $25 in AWS credits. Learn more and enter today!

You can also follow @AWSStartups on Twitter for updates.

-rodica

AWS Summer Startups: Classle

Over the summer months, we’d like to share a few stories from startups around the world: what are they working on and how they are using the cloud to get things done. Today, we’re profiling Classle, from Chennai, India!

Classle team


I recently read Mark Susters blog on Avoiding Monoculture – which is why Im happy to share with you what Ive learned about Classle, a  startup from India, focused on solving education problems for areas of the world that experience serious resource constraints. Classle has the big goal of changing the world around them by encouraging students and experts to share knowledge and expertise, and using the AWS cloud to facilitate this exchange. 

I reached out to Vaidya Nathan, Founder and CEO of Classle:

About Classle

Classle is a Social Learning infrastructure company with specific focus on Education, Learning and Knowledge communities. Using the main Classle product, Cloud Campus platform, Classle creates and manages private and public social learning environments and offers services based on it.

Classle helps rural students access higher education and reach opportunities unavailable before. Our company partners with a wide network of colleges throughout India, which act as internet-connected “learning nodes” that distribute educational materials to students. When the student goes home for the day with their downloaded lectures and other materials from the library, Classle makes use of mobile technology and SMS-based quizzes to keep students engaged and actively learning. The entire system was designed to work with simple, $10 phones, not smartphones, and the students are entirely addicted to these quizzes – they cant get enough of them.
All these services are provided free of charge to both students and colleges. Classle monetizes by partnering with companies who are looking to hire top talent from among the students, and by selling their cloud-based learning platform for training purposes within companies.

Starting Small and Growing with Business

We are using AWS since our inception in early 2009. Our first steps involved two small Amazon EC2 instances and Amazon EBS to store our database. Over the years, our use has expanded to match our business growth. Our selection criteria covered tactical as well as strategic points. From a tactical perspective, we wanted a quicker time for provisioning, which AWS on-demand instances enabled, and the option to secure our resource needs through Reserved Instances.

At a strategic level, we wanted to provide the best experience for our customers and it was key to build Classle on top of services, products, and infrastructure designed for growth and scale. To date, we have established relationships with over 30 educational organizations and that list is constantly growing. Thanks to AWS, we are effectively competing with some large and strong players in the e-learning space.

Sharing the AWS Lessons:

We are a small, LAMP stack team and we started using AWS in 2009. Currently, the products we are using are below. For reference, we are also happy to share our Classle architecture diagram, which is included in our case study with AWS.

  • Amazon Elastic Compute Cloud (EC2)
  • Elastic Load Balancing (ELB)
  • AutoscalingAmazon Elastic Bock Storage (EBS)
  • Amazon Simple Storage Service (S3)
  • Amazon Reduced Redundancy Storage (RRS)
  • Amazon CloudFront with both streaming and download
  • Amazon Cloud Watch
  • Amazon Relational Database Service (Amazon RDS) with Multi-AZ and Read replication.
  • Amazon SimpleDB
  • Amazon Simple Notification Service (Amazon SNS)
  • Amazon Route 53

Pretty soon, we would be using Amazon Elastic Map-Reduce clusters for our analytics requirements..

Words of Wisdom to Startups

Starting a company is always hard, whether youre from India or anywhere else. However, its worth to keep in mind that its never been easier to go out there and try things out –  with Open Source for robust software and cloud service providers like AWS for infrastructure, you can test your ideas and run a business at very low cost.

Being from in India, where we dont have a strong start-up mentality like in the U.S., certainly poses some unique challenges. There are many more problems to solve, and it is exciting to try and translate the existing limitations into innovations, solutions and hence opportunities.

If I had to boil down my advice, I would say to my fellow entrepreneurs to: venture with confidence, design for scale, start small & architect for growth.

——————————————————

Enter Your Startup in the AWS Start-up Challenge!
This year’s AWS Start-up Challenge is a worldwide competition with prizes at all levels, including up to $100,000 in cash, AWS credits, and more for the grand prize winner. Learn more and enter today!

You can also follow @AWSStartups on Twitter for updates.

 -rodica

New S3 Features for The AWS Management Console

We have added three new features to the Amazon S3 tab of the AWS Management Console:

  1. Easier Access – You no longer need to install Adobe Flash or provide outbound access to port 843 on your network in order to use the S3 tab.
  2. Folder Upload – You can now upload the contents of an entire folder with a single selection using a new Advanced Uploader.
  3. Jump – You can now search for objects or folders by simply typing the first few characters of the name.

Easier Access
The console no longer uses Adobe Flash. You don’t need to install it (problematic in some locked-down corporate environments) and you don’t have to enable outbound access on port 843 (required by Flash). You can now use the S3 tab from behind a regular or transparent proxy.

Folder Upload
The new Advanced Uploader allows you to upload entire folders at once. It also allows you to upload individual objects that are larger than 5 GB. You must enable the uploader (a Java applet) in order to take advantage of these new features. To do so, simply click on the Upload button and then Enable Enhanced Uploader:


Once you have done this you can select one or more folders each time you click on Add Files. You can even click this button more than once if you’d like:

Jump
If you have buckets with lots of objects, you’ll love the new Jump feature. Start entering the prefix of the objects or folders that you are looking for and the console will jump to the items that match or follow what you type:

We’ll do our best to keep making the AWS Management Console even better. To do so, we need your feedback. Please feel free to post your suggestions in the S3 Forum.

— Jeff;

 

 

AWS Direct Connect

The new AWS Direct Connect service allows enterprises to create a connection to an AWS Region via a dedicated network circuit. In addition to enhancing privacy, dedicated circuits will generally result in more predictable data transfer performance and will also increase bandwidth between your data center and AWS. Additionally, users of dedicated circuits will frequently see a net reduction in bandwidth costs.

AWS Direct Connect has one location available today, located at Equinixs Ashburn, Virginia colocation facility. From this location, you can connect to services in the AWS US-East (Virginia) region. Additional AWS Direct Connect locations are planned for San Jose, Los Angeles, London, Tokyo, and Singapore in the next several months.

There are two ways to get started:

  • If you already have your own hardware in an Equinix data center in Ashburn, Virginia, you can simply ask them to create a cross-connect from your network to ours. They can generally get this set up in 72 hours or less.
  • If you don’t have hardware in this data center, you can work with one of the AWS Direct Connect solution providers (our initial list includes AboveNet, Equinix, and Level 3) to procure a circuit to the same datacenter or obtain colocation space. If you procure a circuit, the AWS Direct Connect solution provider will take care of the cross-connect for you.

You can select 1 Gbit or 10 Gbit networking for each connection, and you can create multiple connections for redundancy if you’d like. Each connection can be used to access all AWS services. It can also be used to connect to one or more Virtual Private Clouds.

Billing will be based on the number of ports and the speed of each one. Data transfer out of AWS across the circuit will be billed at $0.02 / GB (2 cents per GB). There is no charge for data transfer in to AWS.

I expect to see AWS Direct Connect used in a number of different scenarios. Here are a few of them:

  • Data Center Replacement – Migrate an existing data center to AWS and then use Direct Connect to link AWS to the corporate headquarters using a known private connection.
  • Custom Hosting – Place some custom network or storage devices in a facility adjacent to an AWS Region, and enjoy high bandwidth low latency access to the devices from AWS.
  • High Volume Data Transfer – Move extremely large amounts of data in and out of HPC-style applications.

In order to make the most of a dedicated high speed connection, you will want to look at a category of software often known as WAN optimization (e.g. Riverbed’s Cloud Steelhead) or high speed file transfer (e.g. Aspera’s On-Demand Direct for AWS). Late last month I saw a demonstration from Aspera. They showed me that that were able to achieve 700 Mbps of data transfer across a 1 Gbps line. At this rate they are able to transfer 5 Terabytes of data to AWS in 17 hours. 

— Jeff;

 

AWS Summer Startups: Mediology

Over the summer months, we’d like to share a few stories from startups around the world: what are they working on and how they are using the cloud to get things done. Today, we’re profiling Mediology Software from India!

The Story
2010 was the first year when we allowed countries from Asia Pacific to enter the start-up challenge. We were very impressed with the quality of entries and, in specific, one of them, Mediology Software, caught our eye and made it to the final round in Palo Alto.

Mediology Software is currently a one-year old start-up based out of India and employing 35 people. Mediology DigitalEdition, their main product, is a SaaS platform that enables print publishers to digitize their content, add interactivity, create workflows, and then distribute the content via web, mobile and e-reading platforms. The system achieves its massive scale for content digitization and delivery using event-centric cloud computing services from AWS.

As an example of the type of work Mediology does, I encourage you to take a look at the case study we recently published, describing how AWS and Mediology teamed up to help CozyCot.com, a website geared to East Asian and South Asian women on a wide range of topics including family, health, beauty, etc. In addition to offering CozyCot a better website hosting and scaling solution through the AWS infrastructure, Mediology has helped them distribute and promote their content through a wide variety of platforms, increasing CozyCots bottom line. 

From the Founders
I caught up with Manish Dhingra and Gaurav Bhatnagar, Co-Founders at Mediology Software, a few days ago, as I was checking on how theyre doing almost a year after being named finalists in the AWS Start-up Challenge.

Since January 2011, we have had some high-profile launches on our DigitalEdition platform. Naturally, the usage of AWS, not just in terms of the instance volume, but across the set of AWS services has enabled us to create a very scalable, yet cost-effective architecture. We’re 100% build and reliant on AWS. For instance, we use EC2, Cloudfront, S3, SES, SimpleDB, RDS, SNS, CloudWatch and IAM, all orchestrated together to enable our SaaS platform, Mediology DigitalEdition.

How Has the AWS Start-up Challenge Helped Mediology?
I asked Manish to tell me how the AWS Start-up Challenge has helped their business. Here’s what he told me:

Consumer and Customer confidence in our solution has definitely taken a giant leap, since we returned from Palo Alto in December 2010. Although the same has also led to higher expectations, our grasp of AWS has enabled us to meet the customer expectations quite easily.

Sharing the Wisdom with other Asia-Pacific Start-ups:

AWS gives you the ability to enable application or solution heavy-lifting. We believe Asia is a growth market and many new age concepts around value-based computing, value-added services (specifically around mobile, which works on the core tenets of SaaS and SOA) will find great traction here. 

The key is to not get fazed during the stealth and growth stages of your start-up. Think of AWS as something that gives the wings to your creativity and enables very effective working-capital utilization. In fact, if the pricing benefits are passed on to the consumers, then there is a great chance of leveling the playing field and being the best at what you do, without compromising on the bottom line.

The AWS Startup Challenge
We’re getting ready to launch this year’s edition of our own annual contest, the AWS Startup Challenge. You can sign up to get notified when we launch it, or you can follow @AWSStartups on Twitter.

— Simone;

Amazon S3 – More Than 449 Billion Objects

As of the end of the second quarter of 2011, Amazon S3 holds more than 449 billion objects and processes up to 290,000 requests per second for them at peak times. Here’s a chart:

No matter how you look at it, that’s a lot of objects! Here are a few ways to put it into perspective:

— Jeff;

 

 

IAM: AWS Identity and Access Management – Now Generally Available

Our customers use AWS in many creative and innovative ways, continuously introducing new use cases and driving us to solve unexpected and complex problems. We are constantly improving our capabilities to make sure that we support a very wide variety of use cases and access patterns.

In particular, we want to make sure that developers at any level of experience and sophistication (from a student in a dorm room to an employee of a multinational corporation) have complete control over access to their AWS resources.

AWS Identity and Access Management (IAM) lets you manage users, groups of users, and access permissions for AWS services and resources. You can also use IAM to centrally manage security credentials such as access keys, passwords, and MFA devices. Effective immediately, IAM is now a Generally Available (GA) service!

Using IAM you can create users (representing a person, an organization, or an application, as desired) within an existing AWS Account. You can also group users to apply the same set of permissions. The groups can represent functional boundaries (development vs. test), organizational boundaries (main office vs. branch office), or job function (manager, tester, developer, or system administrator). Each user can be a member of multiple groups (branch office, manager). For maximum security, newly created users have no permissions. All permission control is accomplished using policy documents containing policy statements which grant or deny access to AWS service actions or resources.

IAM can be accessed through APIs, a command line interface, and through the AWS Management Console (I’ve written a separate post about the console support).

Here are some examples of the IAM command line interface in action. Let’s create a user that can create and manage other users and then use this user to create a couple of additional users. Then we’ll give one user the ability to access Amazon S3.

The iam-userlistbypath command lists all or some of the users in the account:

C:\> iam-userlistbypath
C:\>

There are no default users. Let’s create a user “jeff” using the iam-usercreate command (“/family” is a path that further qualifies the names):

C:\> iam-usercreate -u jeff -p /family/ -k
AKIAIYPZGF3ABUC2LQELQ
bbYJpBtRQr635j8QVsCpstrLMS7Mf+ihsLabqEQL

The -k argument causes iam-usercreate to create an AWS access key (both the access key id and the secret access key) for each user. These keys are the credentials needed to access data controlled by the account. They can be inserted in to any application or tool that currently accepts an access key id and a secret access key. Note: It is important to capture and save the secret access key at this point; there’s no way to retrieve it if you lose it (you can create a new set of credentials if necessary).

We can use iam-userlistbypath to verify that we now have one user:

C:\> iam-userlistbypath
arn:aws:iam::889279108296:user/family/jeff

However, user “jeff” has no access because we have not granted him any permissions. The iam-useraddpolicy command is used to add permissions to a user. The iam-groupaddpolicy command can be used to do the same for a group. Let’s add a policy that gives me (user “jeff”) permission to use the IAM APIs on users under the “/app” path. I might not be the only user in my account that should have this permission so I’ll start by creating a group and granting the permissions to the group and then add “jeff” to the group.

C:\> iam-groupcreate -g admins
C:\> iam-groupaddpolicy -g admins -p manageusers -e Allow -a “iam:*” -r “arn:aws:iam::889279108296:user/app/*”
C:\> iam-groupadduser -g admins -u jeff

I (identifying myself as user “jeff” using the credentials that I just created) can now create and manage users under the “/app” path. Let’s create users for two of my applications (“syndic8” and “backup”) using “/app” as the path. I can use the same command that I used to create user “jeff”:

C:\> iam-usercreate -u backup -p /app -k
AKIAI7LTROW2TTCLIFCH
kgRiohPeBGyY6iDx7qzqSzCyrang6YUo67etcGat
C:\> iam-usercreate -u syndic8 -p /app -k
AKIAUIEGOSESA354WS2A
iXdFDaA15VUImTo2MrmErSvTloTeK4ERNIESw78R

I can list only the application users I created by providing an argument to iam-userlist:

C:\> iam-userlistbypath -p /app/
arn:aws:iam::889279108296:user/app/backup
arn:aws:iam::889279108296:user/app/syndic8

Neither “backup” nor “syndic8” have any permissions yet. I can use the access keys for user “jeff” to grant permission for the “backup” user to use all of the S3 APIs on any of my S3 resources:

C:\> iam-useraddpolicy -u backup -p dobackup -e Allow -a “s3:*” -r “arn:aws:s3::*”

This policy allows the user named “backup” to use all of the S3 APIs on any of my S3 resources, but not to access any other AWS service that my AWS Account has subscribed to.

The iam-listuserpolicies command displays the policies associated with a user; the -v option displays the contents of each policy:

C:\> iam-userlistpolicies -u backup -v
dobackup
{“Version”:”2008-10-17″,”Statement”:[{“Effect”:”Allow”,”Action”:[“s3:*”],”Resource”:[“arn:aws:s3::*”]}]}

So, by giving my user (“jeff”) the appropriate privileges, I can minimize the use of my AWS Account credentials for access to AWS services.

You can think of the AWS Account as you would think about the Unix root (superuser) account. To get full value from IAM you should start using it when you are the only developer and you only have one application, adding users, groups, and policies as your environment becomes more complex. You can protect the AWS Account using an MFA device, and you should always sign your AWS calls using the access keys from a particular user. Once you have fully adopted IAM there should be no reason to use the AWS Account’s credentials to make a call to AWS.

There are a number of other commands (fully documented in the IAM CLI Reference). Like all of the other AWS command-line tools, the IAM tools make use of the IAM APIs, all of which are documented in the IAM API Reference.

The AWS Policy Generator can be used to create policies for use with the IAM command line tools. After the policy is created it must be uploaded — use iam-useruploadpolicy instead of iam-useraddpolicy:

C:\> iam-useruploadpolicy -u jeff -p ec2 -f \temp\ec2_iam_policy.txt

IAM controls access to each service in an appropriate way. You can control access to the actions (API functions) of any supported service. You can also control access to IAM, SimpleDB, SQS, S3, SNS, and Route 53 resources. The integration is done in a seamless fashion; all of the existing APIs continue to work as expected (subject, of course, to the permissions established by the use of IAM) and there is no need to change any of the application code. You may decide to create a unique set of credentials for each application using IAM. If you do this, you’ll need to embed the new credentials in each such application.

IAM currently integrates with Amazon EC2, Amazon RDS, Amazon S3, Amazon SimpleDB, Amazon SNS, Amazon SQS, Amazon VPC, Auto Scaling, Amazon Route 53, Amazon CloudFront, Amazon ElasticMapReduce, Elastic Load Balancing, AWS CloudFormation, Amazon CloudWatch, and Elastic Block Storage. IAM also integrates with itself as you saw in my example, you can use it to give certain users or groups the ability to perform IAM actions such as creation of new users.

The AWS Account retains control of all of the data. Also, all accounting still takes place at the AWS Account level, so all usage within the account will be rolled up in to a single bill.

We have seen a wide variety of third-party tools and toolkits add support for IAM already. For example, the newest version of CloudBerry Explorer already supports IAM. Here’s a screen shot of their Policy Editor:

Here are some other applications and toolkits that also support IAM:

Eric Hammond’s article, Improving Security on EC2 With AWS Identity and Access Management (IAM), shows you how to use IAM to create a user that can create EBS snapshots and nothing more. As Eric says:

The release of AWS Identity and Access Management alleviates one of the biggest concerns security-conscious folks used to have when they started using AWS with a single key that gave complete access and control over all resources. Now the control is entirely in your hands.

The features that I have described above represent our first steps toward our long-term goals for IAM. However, we have a long (and very scenic) journey ahead of us and we are looking for additional software engineers, data engineers, development managers, technical program managers, and product managers to help us get there. If you are interested in a full-time position on the Seattle-based IAM team, please send your resume to aws-platform-jobs@amazon.com

I think you’ll agree that IAM makes AWS an even better choice for any type of deployment. As always, please feel free to leave me a comment or to send us some email.

–Jeff;