AWS Mobile Blog

Get started with the AWS Device Farm CLI and Calabash Part 2: Retrieving Reports and Artifacts

by Epshteyn, Ilya | on | Permalink | Comments |  Share

In Part 1 of the walkthrough, we went through the process of creating an AWS Device Farm run for Calabash-Android test scripts.  In the second and final part of the series, we will go through the CLI commands for obtaining the status of the run and to retrieve all of the artifacts associated with a run including log files and screenshots.

Device Farm Run Components

Before we begin, let’s review the hierarchical structure of a Device Farm run:

  • RUN – A run in Device Farm represents a specific build of your app, with a specific set of tests, to be run on a specific set of devices. A run produces a report that contains information about the test execution. A run contains one or more jobs.
  • JOB – A job is a request to test a single app against a single device. A job contains one or more suites.
  • SUITE – A suite is a hierarchical organization of tests in a test package. A suite contains one or more tests.
  • TEST – A test is an individual test within a suite. This is the most granular entity in Device Farm’s data hierarchy.

 

Retrieving Reports and Artifacts

Using Device Farm CLI, we can obtain the status and result counters for each level of the run hierarchy including run, job, suite, and test. As you’ll see, the data available for each level of the hierarchy is very similar, with the scope of the data becoming more granular as you navigate deeper in the hierarchy.

Step 1: Retrieve the Status and Result Counters for a Device Farm Run

Use the get-run CLI command to retrieve the status and result counters of the run. 

[ec2-user]$  aws devicefarm get-run –arn arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917

Output:

{
    "run": {
        "status": "COMPLETED",
        "name": "firstCLIRun",
        "created": 1443331615.166,
        "totalJobs": 101,
        "completedJobs": 101,
        "platform": "ANDROID_APP",
        "result": "ERRORED",
        "billingMethod": "METERED",
        "type": "CALABASH",
        "arn": "arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917",
        "counters": {
            "skipped": 37,
            "warned": 16,
            "failed": 46,
            "stopped": 0,
            "passed": 514,
            "errored": 66,
            "total": 679
        }
    }
}

The above Run has a total of 101 jobs corresponding to 101 unique devices, all of which have completed.  This Run also has 679 individual tests with 46 failures and 66 errors.


Step 2: Retrieve the Status and Counters for all Jobs within a Run:

Use the list-jobs CLI command to list all jobs within a run along with job-level status and result counters.

[ec2-user]$  aws devicefarm list-jobs –arn arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917

Output (partial):

{
    "jobs": [
        {
            "status": "COMPLETED",
            "name": "ASUS Nexus 7 – 2nd Gen (WiFi)",
            "created": 1443331616.586,
            "result": "PASSED",
            "device": {
                "formFactor": "TABLET",
                "name": "ASUS Nexus 7 – 2nd Gen (WiFi)",
                "resolution": {
                    "width": 1200,
                    "height": 1920
                },
                "image": "NA",
                "platform": "ANDROID",
                "heapSize": 0,
                "memory": 34359738368,
                "model": "Nexus 7 – 2nd Gen",
                "os": "4.3.1",
                "cpu": {
                    "frequency": "MHz",
                    "architecture": "ARMv7",
                    "clock": 1512.0
                },
                "arn": "arn:aws:devicefarm:us-west-2::device:208FE7EE973042EA97DEC2EEF31CD10A",
                "manufacturer": "ASUS"
            },
            "message": "Successful test lifecycle of Setup Test",
        "arn": "arn:aws:devicefarm:us-west-2:705582597265:job:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066",
            "counters": {
                "skipped": 0,
                "warned": 0,
                "failed": 0,
                "stopped": 0,
                "passed": 7,
                "errored": 0,
                "total": 7
            }
        },
        {
            "status": "COMPLETED",
            "name": "Amazon Kindle Fire HDX 7 (WiFi)",
            "created": 1443331615.56,
            "result": "FAILED",
            "device": {
                "formFactor": "TABLET",
                "name": "Amazon Kindle Fire HDX 7 (WiFi)",
                "resolution": {
                    "width": 1920,
                    "height": 1200
                },
                "image": "NA",
                "platform": "ANDROID",
                "heapSize": 0,
                "memory": 17179869184,
                "model": "Kindle Fire HDX 7",
                "os": "4.4.3",
                "cpu": {
                    "frequency": "MHz",
                    "architecture": "ARMv7",
                    "clock": 2150.0
                },
                "arn": "arn:aws:devicefarm:us-west-2::device:BC44B6802F134918BDAB6FB4F38C37CC",
                "manufacturer": "Amazon"
            },
        "message": "I see the text "In Theaters" failed: Timeout waiting for elements: * {text CONTAINS[c] ‘In Theaters’} (Calabash::Android::WaitHelpers::WaitError)",
        "arn": "arn:aws:devicefarm:us-west-2:705582597265:job:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00018",
            "counters": {
                "skipped": 0,
                "warned": 0,
                "failed": 5,
                "stopped": 0,
                "passed": 2,
                "errored": 0,
                "total": 7
            }
        }

Note the hierarchical nature of the ARN.  The ARN type is “job” and the job ID is appended to the Run ARN: arn:aws:devicefarm:us-west-2:705582597265:job:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066


Step 3: Retrieve the Status and Result Counters for all Suites within a Job:

Use the list-suites CLI command to list all Suites within a job along with suite-level status and result counters.

[ec2-user]$  aws devicefarm list-suites –arn arn:aws:devicefarm:us-west-2:705582597265:job:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066

Output (partial)

    "suites": [
        {
            "status": "COMPLETED",
            "name": "Setup Suite",
            "created": 1443331656.183,
            "result": "PASSED",
            "message": "Successful test lifecycle of Setup Test",
            "arn": "arn:aws:devicefarm:us-west-2:705582597265:suite:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00000",
            "counters": {
                "skipped": 0,
                "warned": 0,
                "failed": 0,
                "stopped": 0,
                "passed": 1,
                "errored": 0,
                "total": 1
            }
        },
        {
            "status": "COMPLETED",
            "name": "STARmeter",
            "created": 1443331656.221,
            "result": "PASSED",
            "message": "Given I see the text "In Theaters" passed",
            "arn": "arn:aws:devicefarm:us-west-2:705582597265:suite:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00001",
            "counters": {
                "skipped": 0,
                "warned": 0,
                "failed": 0,
                "stopped": 0,
                "passed": 1,
                "errored": 0,
                "total": 1
            }
        }

The returned ARN type is “suite” and the suite ID is appended to the end: arn:aws:devicefarm:us-west-2:705582597265:suite:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00006
 

Step 4: Retrieving the Status and Result Counters for all Tests within a Suite

Use the list-tests CLI command to list all tests within a suite along with test-level status and result counters.

[ec2-user]$  aws devicefarm list-tests –arn arn:aws:devicefarm:us-west-2:705582597265:suite:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00006

Output:

{
    "tests": [
        {
            "status": "COMPLETED",
            "name": "Teardown Test",
            "created": 1443331656.424,
            "result": "PASSED",
            "message": "Successful test lifecycle of Teardown Test",
        "arn": "arn:aws:devicefarm:us-west-2:705582597265:test:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00006/00000",
            "counters": {
                "skipped": 0,
                "warned": 0,
                "failed": 0,
                "stopped": 0,
                "passed": 1,
                "errored": 0,
                "total": 1
            }
        }
    ]
}

The returned ARN type is “test” and the Test ID is appended to the end of the ARN": "arn:aws:devicefarm:us-west-2:705582597265:test:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00006/00000
 

Step 5: Retrieving the Artifacts for a Run, Job, Suite or Test

Using the list-artifacts CLI command, you can retrieve all of the artifacts associated with any of the levels of the run by specifying the corresponding ARN.  For example:

  • List artifacts for the run by specifying the run ARN:
    [ec2-user]$  aws devicefarm list-artifacts –arn arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917 –type FILE
  • List artifacts for the job by specifying the job ARN:
    [ec2-user]$  aws devicefarm list-artifacts –arn arn:aws:devicefarm:us-west-2:705582597265:job:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066 –type FILE
  • List artifacts for the suite by specifying the suite ARN:
    [ec2-user]$  aws devicefarm list-artifacts –arn arn:aws:devicefarm:us-west-2:705582597265:suite:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00000 –type FILE
  • List artifacts for the individual test by specifying the test ARN:
    [ec2-user]$  aws devicefarm list-artifacts –arn arn:aws:devicefarm:us-west-2:705582597265:test:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00006/00000 –type FILE

Note that there are three types of artifacts available: FILE, LOG and SCREENSHOT.

The output of the list-artifacts CLI command will include a pre-signed S3 URL that you can use to securely download the artifact:

{
    "artifacts": [
        {
            "url": "https://prod-us-west-2-results.s3-us-west-2.amazonaws.com/705582597265/2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00000/00000/3d811922-64d2-4d0c-8195-ffb715300836.logcat?AWSAccessKeyId=AKIAI3BDGZ5S7TQL2OZA&Expires=1446165402&Signature=D02sc66m01XXYtSJV0hVGK4l9aA%3D",
            "extension": "logcat",
            "name": "Logcat",
            "arn": "arn:aws:devicefarm:us-west-2:705582597265:artifact:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00000/00000/00000"
        }
    ]
}

 

Conclusion

In this two-part series we have demonstrated the use of the AWS CLI to submit a Calabash-Android test script to Device Farm for testing against real mobile devices.  We have also demonstrated the use of the CLI to retrieve the status of the test Run as well as to retrieve the reports and artifacts for your test Run. If you have questions, please let us know in the comments section or in our developer forum.