Front-End Web & Mobile

Get started with the AWS Device Farm CLI and Calabash Part 1: Creating a Device Farm Run for Android Calabash Test Scripts

AWS Device Farm is an app testing service that enables you to test your native, hybrid, and web apps on real Android and iOS devices that are hosted in the AWS Cloud. A test report containing high-level results, low-level logs, pixel-to-pixel screenshots, and performance data is updated as tests are completed.

Device Farm allows you to upload your own tests or use built-in, script-free compatibility tests. Because testing is automatically performed in parallel, test results are available in minutes. Device Farm supports native and hybrid Android, iOS, and Fire OS apps, including those created with PhoneGap, Titanium, Xamarin, Unity, and other frameworks. It also supports testing web apps in Chrome on Android and Safari on iOS. You can run Built-in-Fuzz tests and Explorer tests or utilize your favorite testing framework including Appium, Calabash, Espresso, and UI Automation. 

The purpose of this walkthrough is to show you how you can quickly submit Calabash test scripts to AWS Device Farm leverageing the AWS Command Line Interface (CLI) for Device Farm. Although the walkthrough uses an Android application and Calabash, you can follow the same steps to test iOS and web apps, as well as use one of the many supported frameworks.

Key Concepts

Before we get started, let’s review a few key AWS Device Farm concepts:

  • PROJECT – A project in Device Farm enables you to organize your tests in whatever way you choose. For example, there can be one project per app title, or there can be one project per platform. You can create as many projects as you need.
  • RUN – A run in Device Farm represents a specific build of your app, with a specific set of tests, to be run on a specific set of devices.
  • DEVICE POOL – A device pool is a pre-configured set of devices that can be used as part of a run. 

Scheduling a Device Farm Run Using the CLI

Prerequisite: Before you can create a run in Device Farm, you must install the AWS Command Line Interface (CLI).

Once you install the CLI, there are five steps to create a Calabash test run in AWS Device Farm:

  1. Create a Device Farm project
  2. Upload your application (Android APK file)
  3. Upload your Calabash test scripts
  4. Create a device pool
  5. Schedule a run

Step 1: Create a Device Farm Project

The first step is to create a Device Farm project. You can also use an existing project if you’ve already created one.

[ec2-user]$  aws devicefarm create-project –name adndroidTest

Output:

[ec2-user@ip-10-0-19-179 ~]$ aws devicefarm create-project –name calabashCLI
{
    “project”: {
        “name”: “calabashCLI”,
        “arn”: “arn:aws:devicefarm:us-west-2:705582597265:project:2c21a412-bb7b-4657-a28c-d7d78b3888f7”,
        “created”: 1442719623.245
    }
}

Note the project ARN as you will reference it when creating your run in the upcoming steps.

Step 2: Create an Upload for Your Android APK File

AWS Device Farm leverages Amazon’s secure and durable S3 service for uploading all objects required for the run.  The upload process utilizes S3 pre-signed URLs to assure a secure upload.

The first step in the upload process is to create your upload request and obtain the S3 pre-signed upload URL:

[ec2-user]$  aws devicefarm create-upload –project-arn arn:aws:devicefarm:us-west-2:705582597265:project:2c21a412-bb7b-4657-a28c-d7d78b3888f7 –name IMDb_5.4.1.105410410_105410410.apk –type ANDROID_APP

The –project-arn is the ARN of your project that you created in Step1: Create a Device Farm Project.

Output:

{
    “upload”: {
        “status”: “INITIALIZED”,
        “name”: “IMDb_5.4.1.105410410_105410410.apk”,
        “created”: 1442719884.445,
        “url”: “https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A705582597265%3Aproject%3A2c21a412-bb7b-4657-a28c-d7d78b3888f7/uploads/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A705582597265%3Aupload%3A2c21a412-bb7b-4657-a28c-d7d78b3888f7/d1f8bebc-a7f5-49a3-881f-f1ffdae90303/IMDb_5.4.1.105410410_105410410.apk?AWSAccessKeyId=AKIAI3BDGZ5S7TQL2OZA&Expires=1442806284&Signature=HZYPvpPBQn%2FTo%2BWmdSx1LW1nOwE%3D”,
        “type”: “ANDROID_APP”,
        “arn”: “arn:aws:devicefarm:us-west-2:705582597265:upload:2c21a412-bb7b-4657-a28c-d7d78b3888f7/d1f8bebc-a7f5-49a3-881f-f1ffdae90303”
    }
}

The pre-signed URL is returned as the url output parameter.

The second step is to upload the actual APK file using the S3 pre-signed URL:

[ec2-user]$  curl -T IMDb_5.4.1.105410410_105410410.apk “https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A705582597265%3Aproject%3A2c21a412-bb7b-4657-a28c-d7d78b3888f7/uploads/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A705582597265%3Aupload%3A2c21a412-bb7b-4657-a28c-d7d78b3888f7/d1f8bebc-a7f5-49a3-881f-f1ffdae90303/IMDb_5.4.1.105410410_105410410.apk?AWSAccessKeyId=AKIAI3BDGZ5S7TQL2OZA&Expires=1442806284&Signature=HZYPvpPBQn%2FTo%2BWmdSx1LW1nOwE%3D“

You can check the status of your APK upload using the get-upload CLI command. Specify the ARN returned in the create-upload request above.

[ec2-user]$  aws devicefarm get-upload –arn arn:aws:devicefarm:us-west-2:705582597265:upload:2c21a412-bb7b-4657-a28c-d7d78b3888f7/d1f8bebc-a7f5-49a3-881f-f1ffdae90303

Output:

{
    “upload”: {
        “status”: “SUCCEEDED”,
        “name”: “IMDb_5.4.1.105410410_105410410.apk”,
        “created”: 1442719884.445,
        “type”: “ANDROID_APP”,
        “arn”: “arn:aws:devicefarm:us-west-2:705582597265:upload:2c21a412-bb7b-4657-a28c-d7d78b3888f7/d1f8bebc-a7f5-49a3-881f-f1ffdae90303”,
        “metadata”: “{“screens”: [“small”, “normal”, “large”, “xlarge”], “sdk_version”: “14”, “package_name”: “com.imdb.mobile”, “native_code”: [], “target_sdk_version”: “18”, “version_name”: “5.4.1.105410410”, “version_code”: “105410410”, “activity_name”: “com.imdb.mobile.HomeActivity”}”
    }
}

Once the status is “SUCCEEDED” you are ready to continue to Step 3.

Step 3: Create an Upload for Your Calabash Test Package

In this step, you will upload the Calabash test scripts package. Please refer to the documentation on how to prepare your Android Calabash test package for upload to Device Farm.

The upload process for Calabash test scripts is similar to the upload of the Android APK file in Step 2.

First create an upload request and obtain the S3 pre-signed upload URL:

[ec2-user]$  aws devicefarm create-upload –project-arn arn:aws:devicefarm:us-west-2:705582597265:project:2c21a412-bb7b-4657-a28c-d7d78b3888f7 –name features.zip –type CALABASH_TEST_PACKAGE

Output:

{
    “upload”: {
        “status”: “INITIALIZED”,
        “name”: “features.zip”,
        “created”: 1442724498.222,
        “url”: “https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A705582597265%3Aproject%3A2c21a412-bb7b-4657-a28c-d7d78b3888f7/uploads/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A705582597265%3Aupload%3A2c21a412-bb7b-4657-a28c-d7d78b3888f7/ccfa27c9-3f66-49ac-abe0-542214cd650c/features.zip?AWSAccessKeyId=AKIAI3BDGZ5S7TQL2OZA&Expires=1442810898&Signature=MOGwu7vUP3q%2B8OrrlSmJ5nVsQe0%3D”,
        “type”: “CALABASH_TEST_PACKAGE”,
        “arn”: “arn:aws:devicefarm:us-west-2:705582597265:upload:2c21a412-bb7b-4657-a28c-d7d78b3888f7/ccfa27c9-3f66-49ac-abe0-542214cd650c”
    }
}

Next, upload the Calabash text package ZIP file to the S3 pre-signed URL:

[ec2-user]$  curl -T features.zip “https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A705582597265%3Aproject%3A2c21a412-bb7b-4657-a28c-d7d78b3888f7/uploads/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A705582597265%3Aupload%3A2c21a412-bb7b-4657-a28c-d7d78b3888f7/ccfa27c9-3f66-49ac-abe0-542214cd650c/features.zip?AWSAccessKeyId=AKIAI3BDGZ5S7TQL2OZA&Expires=1442810898&Signature=MOGwu7vUP3q%2B8OrrlSmJ5nVsQe0%3D”

You can check the status of your Calabash test package upload using the get-upload CLI command. Specify the upload ARN returned in the create-upload request above.

[ec2-user]$  aws devicefarm get-upload –arn arn:aws:devicefarm:us-west-2:705582597265:upload:2c21a412-bb7b-4657-a28c-d7d78b3888f7/ccfa27c9-3f66-49ac-abe0-542214cd650c

Output:
{
    “upload”: {
        “status”: “SUCCEEDED”,
        “name”: “features.zip”,
        “created”: 1442724498.222,
        “type”: “CALABASH_TEST_PACKAGE”,
        “arn”: “arn:aws:devicefarm:us-west-2:705582597265:upload:2c21a412-bb7b-4657-a28c-d7d78b3888f7/ccfa27c9-3f66-49ac-abe0-542214cd650c”,
        “metadata”: “{“valid”: true}”
    }
}

Once the status is “SUCCEEDED” you are ready to continue to Step 4.

Step 4: Create a Device Pool Associated with the Project

The AWS Device Farm fleet currently includes over 200 different Android, Fire OS, and iOS devices (see the current device list). You may create Device Pools that consist of specific set of devices that you would like to test against. Your device pools can consist of static lists of devices or rule-based on multiple parameters including device platform (i.e. Android or iOS), manufacturer, and form factor.

The CLI command below creates a Device Pool consisting of all available Android devices:

[ec2-user]$  aws devicefarm create-device-pool –project-arn arn:aws:devicefarm:us-west-2:705582597265:project:2c21a412-bb7b-4657-a28c-d7d78b3888f7 –name myAndroidDevicePool –rules ‘[{“attribute”: “PLATFORM”, “operator”: “EQUALS”, “value”: “”ANDROID””}]’

Output:

{
    “devicePool”: {
        “rules”: [
            {
                “operator”: “EQUALS”,
                “attribute”: “PLATFORM”,
                “value”: “”ANDROID””
            }
        ],
        “name”: “myAndroidDevicePool”,
        “arn”: “arn:aws:devicefarm:us-west-2:705582597265:devicepool:2c21a412-bb7b-4657-a28c-d7d78b3888f7/e4a54fbd-d094-4130-b6d9-60811820e4d8”
    }
}

Step 5: Schedule a Run

To schedule a run, you will need the following input parameters:

  • project-arn: the ARN of the project from Step1: Create a Device Farm Project
  • app-arn: the ARN of the Application upload from Step 2: Create an Upload for Your Android APK File
  • testPackageArn: the ARN of the Calabash test package upload from Step 3: Create an Upload for Your Calabash Test Package
  • device-pool-arn: the ARN of the device pool created in Step 4: Create a Device Pool Associated with the Project

The following command schedules a run in Device Farm using the specified Android application, Calabash test package, and device pool:

[ec2-user]$  aws devicefarm schedule-run –project-arn arn:aws:devicefarm:us-west-2:705582597265:project:2c21a412-bb7b-4657-a28c-d7d78b3888f7 –app-arn arn:aws:devicefarm:us-west-2:705582597265:upload:2c21a412-bb7b-4657-a28c-d7d78b3888f7/d1f8bebc-a7f5-49a3-881f-f1ffdae90303 –device-pool-arn arn:aws:devicefarm:us-west-2:705582597265:devicepool:2c21a412-bb7b-4657-a28c-d7d78b3888f7/e4a54fbd-d094-4130-b6d9-60811820e4d8 –name firstCLIRun –test ‘{“type”: “CALABASH”,”testPackageArn”:”arn:aws:devicefarm:us-west-2:705582597265:upload:2c21a412-bb7b-4657-a28c-d7d78b3888f7/ccfa27c9-3f66-49ac-abe0-542214cd650c”}’

Output:

{
    “run”: {
        “status”: “SCHEDULING”,
        “name”: “firstCLIRun”,
        “created”: 1442724966.821,
        “totalJobs”: 100,
        “completedJobs”: 0,
        “platform”: “ANDROID_APP”,
        “result”: “PENDING”,
        “billingMethod”: “METERED”,
        “type”: “CALABASH”,
        “arn”: “arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/61c16425-15c8-4fba-a44f-07dcba60f717”,
        “counters”: {
            “skipped”: 0,
            “warned”: 0,
            “failed”: 0,
            “stopped”: 0,
            “passed”: 0,
            “errored”: 0,
            “total”: 0
        }
    }
}

You can check the status of the run by using the get-run CLI command:

[ec2-user]$  aws devicefarm get-run –arn arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/61c16425-15c8-4fba-a44f-07dcba60f717

Output:

{
    “run”: {
        “status”: “COMPLETED”,
        “name”: “firstCLIRun”,
        “created”: 1442724966.821,
        “totalJobs”: 100,
        “completedJobs”: 100,
        “platform”: “ANDROID_APP”,
        “result”: “ERRORED”,
        “billingMethod”: “METERED”,
        “type”: “CALABASH”,
        “arn”: “arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/61c16425-15c8-4fba-a44f-07dcba60f717”,
        “counters”: {
            “skipped”: 37,
            “warned”: 17,
            “failed”: 65,
            “stopped”: 0,
            “passed”: 495,
            “errored”: 65,
            “total”: 679
        }
    }
}

Once the status is “COMPLETED” your tests on all devices have completed and a full, aggregate summary is available. More on that in Part 2 of this series.

Conclusion

As you can see it’s easy to test your application on dozens of real mobile devices using the Device Farm CLI. Join us in Part 2 of this series where you will retrieve the status and artifacts for your Device Farm Run, including an aggregate summary, log files, and screenshots. If you have any questions please leave a comment below or reach out in our developer forum.