Category: AWS Device Farm

Last year I wrote about AWS Device Farm and told you how you can use it to Test Mobile Apps on Real Devices. As I described at the time, AWS Device Farm allows you to create a project, identify an application, configure a test, and then run the test against a variety of iOS and Android devices.

Remote Access to Devices
Today we are launching a new feature that provides you with remote access to devices (phones and tablets) for interactive testing. You simply open a new session on the desired device, wait (generally a minute or two) until the device is available, and then interact with the device via the AWS Management Console.

You can gesture, swipe, and interact with devices in real time directly through your web browser as if the device was on your desk or in your hand. This includes installing and running applications!

Here’s a quick demo. I click on Start a new session to begin:

Then I search for a device of the desired type, including the desired OS version, select it, and name my session. I click on Confirm and start session to proceed:

Then I wait for the device to become available (about 30 seconds in this case):

Once the device is available I can see the screen and access it through the Console:

I can interact with the Kindle Fire using my mouse. Perhaps my app is not behaving as expected when the language is set to Latin American Spanish. I can change the Kindle Fire’s settings with a couple of clicks:

I can install my app on the Kindle Fire by clicking on Upload and choosing my APK.

My session can run for up to 60 minutes. After that time it will stop automatically.

Available Now
This new feature is available in beta form now, with a wide selection of Android phones and tablets. We will be adding iOS devices later this year, along with additional control over the device configuration and (virtual) location.

AWS Device Farm comes with a one-time free trial of 250 device minutes. After that you are charged $0.17 per device minute. Or you can pay $250 per slot per month for unmetered access (slots are the units of concurrent execution).

— Jeff;

If you build mobile apps, you know that you have two implementation choices. You can build native or hybrid applications that compile to an executable file. You can also build applications that run within the device’s web browser.

We launched the AWS Device Farm in July with support for testing native and hybrid applications on iOS and Android devices (see my post, AWS Device Farm – Test Mobile Apps on Real Devices, to learn more).

Today we are adding support for testing browser-based applications on iOS and Android devices. Many customers have asked for this option and we are happy to be able to announce it. You can now create a single test run that spans any desired combination of supported devices and makes use of the Appium Java JUnit or Appium Java TestNG frameworks (we’ll add additional frameworks over time; please let us know what you need).

Testing a Web App
I tested a simple web app. It opens amazon.com and searches for the string “Kindle”. I opened the Device Farm Console and created a new project (Test Amazon Site). Then I created a new run (this was my second test, so I called it Web App Test #2):

Then I configured the test by choosing the test type (TestNG) and uploading the tests (prepared for me by one of my colleagues):

The file (chrome-with-screenshot.zip) contains the compiled test and the dependencies (a bunch of JAR files):

Next, I choose the devices. I had already created a “pool” of Android devices, so I used it:

I started the run and then checked in on it a few minutes later:

Then I inspected the output, including screen shots, from a single test:

Available Now
This new functionality is available now and you can start using it today! Read the Device Farm Documentation to learn more.

— Jeff;

Update (August 4, 2015) – Support for testing iOS apps is available now; click here to get started!

We launched AWS Device Farm earlier this month with support for testing apps on Android and Fire OS devices.

I am happy to give you a heads-up that you will soon be able to test your apps on Apple phones and tablets! We plan to launch support for iOS on August 4, 2015 with support for the following test automation frameworks:

• Appium (Java JUnit and Java TestNG).
• Calabash.
• UI Automation.
• XCTest.

You can also use the fuzz test that is built in to Device Farm. This test randomly sends user interface events to devices and reports on the results.

Here are some preliminary screen shots of the new iOS support in action. After you upload your binary to Device Farm, you will have the opportunity to select the app to test:

After you start the test (step 5 in the screen shot above), the test results and the associated screen shots will be displayed as they arrive:

With the new iOS support, you will be able to test your cross-platform titles and get reports (including high-level test results, problem patterns, logs, screenshots, and performances data) that are consistent, regardless of the platform and test framework that you use. If you use a cross-platform test framework such as Appium or Calabash, you can use the same code for Android, FireOS, and iOS tests.

Be Prepared
As I said earlier, iOS support will be available in less than a week. You can get ready now by reading the Device Farm documentation and by creating test suites and scripts using one or more of the frameworks that I mentioned above.

— Jeff;

Although the phrase “write once, test everywhere” was coined to describe another technology, I believe that it also applies to mobile applications. The wide variety of carriers, manufacturers, models, operating systems (including versions, patches, and proprietary extensions), screen sizes, input devices, connectivity options, device settings, and permission settings combine to create a test matrix that is seemingly infinite in size. Let’s toss in the device location and the selected locale, just for fun. Add to this the rapid (sometimes daily) release cycles for popular apps and testing becomes an expensive, complex, and frustrating exercise. As a result, many app developers either forgo comprehensive testing or limit their market to a modest subset of the available devices.

New AWS Device Farm
Today we are introducing the new AWS Device Farm.  This new service will help you to improve the quality of your Android and Fire OS apps by quickly and securely testing them in more than 200 unique (as defined by device and operating system) environments.

You can easily upload your app, select the devices that you want to test it on, and initiate a run using the tests of your choice. You’ll get real-time reports as the tests complete, so that you can quickly identify and address issues.

Visit the Farm
Let’s take a tour of the Device Farm. I’ll use the IMDB Movie App for testing. The first step is to create a new project:

When I am ready to test, I simply create a new Run. A Run identifies the application to be tested, the tests to be run, and the devices to test on. If I have already uploaded my app, I can start from there. If not, I simply click on Upload and locate the APK file.

If the package contains more than one application I can choose the one I want (the field labeled Minimum SDK indicates that the binary will run on SDK 14 and above; this is used to decide which devices are compatible with the app):

Then I configure a test for the run. I can use the built-in fuzz test, or I can choose to build a script for Appium, Calabash, or uiautomator. I can also run instrumentation tests using JUnit, Robotium, or Espresso.

I have a Calabash script so I’ll upload and use it:

Now I get to choose the devices that I’d like to test on. AWS Device Farm can check for app compatibility and select all of the compatible devices, or I can create a Device Pool and select the desired devices. I’ll go with the first option:

I have control over the device state. I can upload extra files, install other apps, enable the desired communication channels (radios), set the apparent location for the device, and choose the locale:

Then I review the settings in preparation for my test run:

Testing will commence as soon as I click on Confirm and start run, based on device availability (generally a matter of seconds). The console will display the testing status on a per-device basis:

The window will be updated as the tests proceed. My run identified some issues (failures and warnings); Device Farm grouped the results in order to simplify the presentation:

From there I can dig in and investigate individual issues:

I have access to screen shots created and captured during the test run:

I can also look at performance (CPU & memory utilization, along with thread count) over the course of the test:

I have access to the files generated during the test:

After I study, understand, and remedy the failures, I can create another Run and try again.

I also have the option to automate any or all of the steps above! The Device Farm API contains functions for creating projects & device pools, uploading files, retrieving information about devices, projects, and jobs (including test results), and more. Most importantly, it allows test runs to be scheduled with a single call to ScheduleRun. We also have a plugin for Jenkins, with connections to other CI (Continuous Integration) systems in the works.

Available Now
The AWS Device Farm is available now and you can start testing your apps today (read the AWS Device Farm Developer Guide to learn more).

Pricing is in units of device minutes, basically the duration of a single test run on a particular device. You get 250 minutes at no charge as part of a Free Trial; after that you pay $0.17 per device minute. You can also opt in to our unmetered testing plan; you can perform unlimited testing on any supported Android or FireOS device for a flat monthly fee of $250.

— Jeff;