AWS Device Farm – Test Mobile Apps on Real Devices

Although the phrase “write once, test everywhere” was coined to describe another technology, I believe that it also applies to mobile applications. The wide variety of carriers, manufacturers, models, operating systems (including versions, patches, and proprietary extensions), screen sizes, input devices, connectivity options, device settings, and permission settings combine to create a test matrix that is seemingly infinite in size. Let’s toss in the device location and the selected locale, just for fun. Add to this the rapid (sometimes daily) release cycles for popular apps and testing becomes an expensive, complex, and frustrating exercise. As a result, many app developers either forgo comprehensive testing or limit their market to a modest subset of the available devices.

New AWS Device Farm
Today we are introducing the new AWS Device Farm.  This new service will help you to improve the quality of your Android and Fire OS apps by quickly and securely testing them in more than 200 unique (as defined by device and operating system) environments.

You can easily upload your app, select the devices that you want to test it on, and initiate a run using the tests of your choice. You’ll get real-time reports as the tests complete, so that you can quickly identify and address issues.

Visit the Farm
Let’s take a tour of the Device Farm. I’ll use the IMDB Movie App for testing. The first step is to create a new project:

When I am ready to test, I simply create a new Run. A Run identifies the application to be tested, the tests to be run, and the devices to test on. If I have already uploaded my app, I can start from there. If not, I simply click on Upload and locate the APK file.

If the package contains more than one application I can choose the one I want (the field labeled Minimum SDK indicates that the binary will run on SDK 14 and above; this is used to decide which devices are compatible with the app):

Then I configure a test for the run. I can use the built-in fuzz test, or I can choose to build a script for Appium, Calabash, or uiautomator. I can also run instrumentation tests using JUnit, Robotium, or Espresso.

I have a Calabash script so I’ll upload and use it:

Now I get to choose the devices that I’d like to test on. AWS Device Farm can check for app compatibility and select all of the compatible devices, or I can create a Device Pool and select the desired devices. I’ll go with the first option:

I have control over the device state. I can upload extra files, install other apps, enable the desired communication channels (radios), set the apparent location for the device, and choose the locale:

Then I review the settings in preparation for my test run:

Testing will commence as soon as I click on Confirm and start run, based on device availability (generally a matter of seconds). The console will display the testing status on a per-device basis:

The window will be updated as the tests proceed. My run identified some issues (failures and warnings); Device Farm grouped the results in order to simplify the presentation:

From there I can dig in and investigate individual issues:

I have access to screen shots created and captured during the test run:

I can also look at performance (CPU & memory utilization, along with thread count) over the course of the test:

I have access to the files generated during the test:

After I study, understand, and remedy the failures, I can create another Run and try again.

I also have the option to automate any or all of the steps above! The Device Farm API contains functions for creating projects & device pools, uploading files, retrieving information about devices, projects, and jobs (including test results), and more. Most importantly, it allows test runs to be scheduled with a single call to ScheduleRun. We also have a plugin for Jenkins, with connections to other CI (Continuous Integration) systems in the works.

Available Now
The AWS Device Farm is available now and you can start testing your apps today (read the AWS Device Farm Developer Guide to learn more).

Pricing is in units of device minutes, basically the duration of a single test run on a particular device. You get 250 minutes at no charge as part of a Free Trial; after that you pay $0.17 per device minute. You can also opt in to our unmetered testing plan; you can perform unlimited testing on any supported Android or FireOS device for a flat monthly fee of $250.

— Jeff;