Tag: Android

The defining characteristics of a useful mobile testing service in the cloud are: instant access to a large number of real devices, ability to run parallel tests, excellent reporting, ability to configure the device state, and little to no modifications to your existing test setup.

Working with our most successful mobile customers, we noticed that they use a small set of dedicated physical devices or emulators during the app development and test code authoring phase. Thereafter, when they are closer to the release, they access a larger fleet of devices to get extensive market coverage. In addition, their continuous integration (CI) tools need access to a set of devices to run their regression tests on every check-in and on a nightly or daily basis.

(more…)

“It worked on my machine.”  If you have been a developer for a while, you probably have heard this.  It is a lament for a bug that is hard to diagnose and points to some peculiar setup of the developer’s machine. You can fix this by including continuous integration in your tool arsenal.  When you check in code to your code repository, a build happens automatically with an approved build system. Unit tests are run, and artifacts – the compiled code ready to run on the device and any logs – are uploaded to an accessible location.

(more…)

Recently, we released the AWS Mobile SDK v2.6.0 for Android and iOS. The release has new features that make it easier for iOS and Android developers to integrate AWS backend services into their apps.  The new features include:

• A configuration pattern that makes it easier for developers to get started, and to target production vs. pre-production environments.
• A native authentication user experience, which allows developers to easily add sign-in and sign-up flows powered by Amazon Cognito.
• A new S3 transfer manager that provides easier integration of file upload and download capabilities.

(more…)

Recently, we released v2.6.0 of the AWS Mobile SDK for Android and AWS Mobile SDK for iOS. It includes many improvements. This post focuses on two of these. First, you can integrate a JSON file with the configuration details of your mobile backend. The easiest way to generate this file is with AWS Mobile Hub. You can also generate the file yourself. Using the JSON file means that you don’t need to edit the Info.plist file (for iOS) or create a class of constants (for Android). You can use the same configuration file for both iOS and Android versions of the app.

Second, we integrated two features that were available only on the AWS Mobile Hub custom SDK. IdentityManager is for managing identity. It includes a UI component for each platform. The component makes it easier to develop a sign-in and sign-up screen. TransferManager is for managing file uploads and downloads. As with IdentityManager, it’s designed for integrating into the UI. Since the custom SDK features have been integrated into the standard SDK, we no longer have a custom SDK for AWS Mobile Hub users.

In this post, we walk through adding analytics and authentication to an existing Android app. The example app is a standalone note-taking app, which you can download from GitHub. Start by downloading and compiling the source code. Set up an emulator and run the code to see how it performs. We used the master-detail project so that the application is responsive to tablets and phones. The app was tested in the emulator for a Nexus 5X and a Pixel C on Android 7.1.1 (API level 25).

(more…)

In the first part of this blog, we introduced you to the Amazon DynamoDB Document API. This API is a mechanism for accessing data in DynamoDB that doesn’t require you to map models to the data. Instead, you access the data through standard accessor methods on a standard Document object. In this blog post, we show how to integrate this functionality into a real application. The application for this demonstration is an Android memo app. The mobile backend architecture for the app is shown in the following diagram.

The services used are:

• Amazon Cognito, which authenticated the mobile device to the backend.
• AWS Identity and Access Management (IAM), which defines permissions for the mobile device.
• Amazon DynamoDB, which stores the data for the individual memos.

This infrastructure can be easily set up using AWS Mobile Hub.  Either click the Deploy to AWS Mobile Hub button in our GitHub repository or follow these instructions:

1. Click Create a new Project in the AWS Mobile Hub console.  Provide a name for your project.
2. Click NoSQL Database.
3. Click Enable NoSQL.
4. Click Add a new table.
5. Click Start with an example schema.
6. Select Notes as the example schema.
7. Select Public for the permissions (we won’t add sign-in code to this app).
8. Click Create Table, and then click Create Table in the dialog box.

Even though the table you created has a userID, the data is stored unauthenticated in this example. If you were to use this app in production, use Amazon Cognito to sign the user in to your app, and then use the userID to store the authenticated data.

In the left navigation pane, click Resources. AWS Mobile Hub created an Amazon Cognito identity pool, an IAM role, and a DynamoDB database for your project. Mobile Hub also linked the three resources according to the permissions you selected when you created the table. For this demo, you need the following information:

• The Amazon Cognito identity pool ID (for example, us-east-1:f9d582af-51f9-4db3-8e36-7bdf25f4ee07)
• The name of the Notes table (for example, androidmemoapp-mobilehub-1932532734-Notes)

These are stored in the application code and used when connecting to the database.

Now that you have a mobile app backend, it’s time to look at the frontend. We have a memo app that you can download from GitHub. First, add the required libraries to the dependencies section of the application build.gradle file:

compile 'com.amazonaws:aws-android-sdk-core:2.4.4'
compile 'com.amazonaws:aws-android-sdk-ddb:2.4.4'
compile 'com.amazonaws:aws-android-sdk-ddb-document:2.4.4'

Add the INTERNET, ACCESS_NETWORK_STATE, and ACCESS_WIFI_STATE permissions to AndroidManifest.xml:

<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.ACCESS_WIFI_STATE" />

Set up the connection to the Amazon DynamoDB table by creating an Amazon Cognito credentials provider (for appropriate access permissions), and then creating a DynamoDbClient object. Finally, create a table reference:

// Create a new credentials provider
credentialsProvider = new CognitoCachingCredentialsProvider(
        context, COGNITO_POOL_ID, COGNITO_REGION);

// Create a connection to DynamoDB
dbClient = new AmazonDynamoDBClient(credentialsProvider);

// Create a table reference
dbTable = Table.loadTable(dbClient, DYNAMODB_TABLE);

You can now perform CRUD (Create, Read, Update, Delete) operations on the table:

/**
 * create a new memo in the database
 * @param memo the memo to create
 */
public void create(Document memo) {
    if (!memo.containsKey("userId")) {
        memo.put("userId", credentialsProvider.getCachedIdentityId());
    }
    if (!memo.containsKey("noteId")) {
        memo.put("noteId", UUID.randomUUID().toString());
    }
    if (!memo.containsKey("creationDate")) {
        memo.put("creationDate", System.currentTimeMillis());
    }
    dbTable.putItem(memo);
}

/**
 * Update an existing memo in the database
 * @param memo the memo to save
 */
public void update(Document memo) {
    Document document = dbTable.updateItem(memo, new UpdateItemOperationConfig().withReturnValues(ReturnValue.ALL_NEW));
}

/**
 * Delete an existing memo in the database
 * @param memo the memo to delete
 */
public void delete(Document memo) {
    dbTable.deleteItem(
        memo.get("userId").asPrimitive(),   // The Partition Key
        memo.get("noteId").asPrimitive());  // The Hash Key
}

/**
 * Retrieve a memo by noteId from the database
 * @param noteId the ID of the note
 * @return the related document
 */
public Document getMemoById(String noteId) {
    return dbTable.getItem(new Primitive(credentialsProvider.getCachedIdentityId()), new Primitive(noteId));
}

/**
 * Retrieve all the memos from the database
 * @return the list of memos
 */
public List<Document> getAllMemos() {
    return dbTable.query(new Primitive(credentialsProvider.getCachedIdentityId())).getAllResults();
}

There are two mechanisms for searching the dataset: scan and query. The query() method uses indexed fields within the DynamoDB table to rapidly retrieve the appropriate information. The scan() method is more flexible. It allows you to search on every field, but it can run into performance issues when searching large amounts of data. This results in a worse experience for your users because data retrieval will be slower. For the best experience, index fields that you intend to search often and use the query() method.

The Notes schema in DynamoDB usually segments data on a per-user basis. The app works with both authenticated and unauthenticated users by using the .getCachedIdentityId() method. This method stores the current user identity with every new note that is created.

Android does not allow you to perform network requests on the main UI thread. You must wrap each operation in an AsyncTask. For example:

/**
 * Async Task to create a new memo into the DynamoDB table
 */
private class CreateItemAsyncTask extends AsyncTask<Document, Void, Void> {
    @Override
    protected Void doInBackground(Document... documents) {
        DatabaseAccess databaseAccess = DatabaseAccess.getInstance(EditActivity.this);
        databaseAccess.create(documents[0]);
        return null;
    }
}

You can initiate a save operation by instantiating the appropriate AsyncTask and then calling .execute():

/**
 * Event Handler called when the Save button is clicked
 * @param view the initiating view
 */
public void onSaveClicked(View view) {
    if (memo == null) {
        Document newMemo = new Document();
        newMemo.put("content", etText.getText().toString());
        CreateItemAsyncTask task = new CreateItemAsyncTask();
        task.execute(newMemo);
    } else {
        memo.put("content", etText.getText().toString());
        UpdateItemAsyncTask task = new UpdateItemAsyncTask();
        task.execute(memo);
    }
    // Finish this activity and return to the prior activity
    this.finish();
}

Similarly, you can retrieve a list of memos on an AsyncTask and pass the memos back to a method in the MainActivity to populate the UI:

/**
 * Async Task for handling the network retrieval of all the memos in DynamoDB
 */
private class GetAllItemsAsyncTask extends AsyncTask<Void, Void, List<Document>> {
    @Override
    protected List<Document> doInBackground(Void... params) {
        DatabaseAccess databaseAccess = DatabaseAccess.getInstance(MainActivity.this);
        return databaseAccess.getAllMemos();
    }

    @Override
    protected void onPostExecute(List<Document> documents) {
        if (documents != null) {
            populateMemoList(documents);
        }
    }
}

You can find this sample in our GitHub repository. To run the sample:

1. Open the project in Android Studio.
2. Open the DatabaseAccess.java file.
3. Replace the temporary string for COGNITO_POOL_ID with your Amazon Cognito Identity Pool ID.
4. Replace the temporary string for DYNAMODB_TABLE with your Amazon DynamoDB table name.
5. If necessary, replace the COGNITO_REGION with the region for your Amazon Cognito identity pool ID.
6. Run the app on a real device or the Android emulator.

If you are successful, the app looks like the following:

The Document API is currently in beta. We would like to hear about the use cases you want us to support. You can leave your feedback and issues on our forums and GitHub.

Starting with the AWS Mobile SDK for Android version 2.4.0, you have two options to encrypt client-side objects before sending them to Amazon S3: an AWS Key Management Service (AWS KMS) customer master key (CMK) or a client-side master key.

Customer master key

When using an AWS KMS customer master key (CMK) for client-side data encryption, you provide only a CMK ID. The AWS Mobile SDK for Android sends a request to AWS KMS for a key to encrypt your data. There is no requirement to provide encryption keys to the Amazon S3 encryption client.

While uploading, the data in the object is encrypted and the cipher blob version of the encryption key that is returned from KMS is stored in the object metadata.

When the client calls Amazon S3 to download the object and its metadata, the client uses the cipher blob key to get the plaintext key to decrypt the entire S3 object.

The following example shows how to set up Amazon S3 client-side encryption with a CMK on an Android device using the AWS Mobile SDK for Android.

Initialize KMSEncryptionMaterialsProvider

final KMSEncryptionMaterialsProvider materialProvider = new KMSEncryptionMaterialsProvider(<kmsKey>); 
final CryptoConfiguration cryptoConfiguration = new CryptoConfiguration()
                                                      .withAwsKmsRegion(Region
                                                                   .getRegion(Regions.US_EAST_1));
final AmazonS3 s3Client = new AmazonS3EncryptionClient(<credentialProvider>, 
                                            materialProvider, 
                                            cryptoConfiguration);
s3Client.setRegion(Region.getRegion(Regions.US_EAST_1));

Uploading

To upload an object to Amazon S3, use the AmazonS3EncryptionClient you initialized, and then use either PutObjectRequest or TransferUtility.

Using PutObjectRequest

final PutObjectRequest putObjectRequest = new PutObjectRequest(<bucketName>, <key>, <file>);
final PutObjectResult putObjectResult = s3Client.PutObject(putObjectRequest);

Using TransferUtility

final TransferUtility transferUtility = new TransferUtility(s3Client, <context>); 
final TransferObserver transferObserver =  transferUtility
                                                  .upload(<bucketName>, <key>, <file>);

Downloading

To download an object from Amazon S3, use the AmazonS3EncryptionClient you initialized earlier, and then use either GetObjectRequest or TransferUtility.

Using GetObjectRequest

final InputStream inputStream = s3Client.getObject(<bucketName>, <key>).getObjectContent();

Using TransferUtility:

final File tempDownloadFile = temp.newFile();
final TransferObserver observer = transferUtility
                                       .download(<bucketName>, <key>, <tempDownloadFile>);

Client-side master Key

Your client-side master keys and your unencrypted data are never sent to AWS. It is important that you safely manage your encryption keys. If you lose them, you won’t be able to decrypt your data.

If you want to use your own custom master key, use the following code to create a material provider.

final EncryptionMaterials encryptionMaterials = new EncryptionMaterials(<keypair>);
final AmazonS3 s3Client = new AmazonS3EncryptionClient(<credentialProvider>, encryptionMaterials);

As always you can post your questions and feedback to our GitHub repo or to our AWS Forums.

In Part 1 of the walkthrough, we went through the process of creating an AWS Device Farm run for Calabash-Android test scripts.  In the second and final part of the series, we will go through the CLI commands for obtaining the status of the run and to retrieve all of the artifacts associated with a run including log files and screenshots.

Device Farm Run Components

Before we begin, let’s review the hierarchical structure of a Device Farm run:

• RUN – A run in Device Farm represents a specific build of your app, with a specific set of tests, to be run on a specific set of devices. A run produces a report that contains information about the test execution. A run contains one or more jobs.
• JOB – A job is a request to test a single app against a single device. A job contains one or more suites.
• SUITE – A suite is a hierarchical organization of tests in a test package. A suite contains one or more tests.
• TEST – A test is an individual test within a suite. This is the most granular entity in Device Farm’s data hierarchy.

Retrieving Reports and Artifacts

Using Device Farm CLI, we can obtain the status and result counters for each level of the run hierarchy including run, job, suite, and test. As you’ll see, the data available for each level of the hierarchy is very similar, with the scope of the data becoming more granular as you navigate deeper in the hierarchy.

Step 1: Retrieve the Status and Result Counters for a Device Farm Run

Use the get-run CLI command to retrieve the status and result counters of the run. 

[ec2-user]$  aws devicefarm get-run –arn arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917

Output:

{
    "run": {
        "status": "COMPLETED",
        "name": "firstCLIRun",
        "created": 1443331615.166,
        "totalJobs": 101,
        "completedJobs": 101,
        "platform": "ANDROID_APP",
        "result": "ERRORED",
        "billingMethod": "METERED",
        "type": "CALABASH",
        "arn": "arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917",
        "counters": {
            "skipped": 37,
            "warned": 16,
            "failed": 46,
            "stopped": 0,
            "passed": 514,
            "errored": 66,
            "total": 679
        }
    }
}

The above Run has a total of 101 jobs corresponding to 101 unique devices, all of which have completed.  This Run also has 679 individual tests with 46 failures and 66 errors.

Step 2: Retrieve the Status and Counters for all Jobs within a Run:

Use the list-jobs CLI command to list all jobs within a run along with job-level status and result counters.

[ec2-user]$  aws devicefarm list-jobs –arn arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917

Output (partial):

{
    "jobs": [
        {
            "status": "COMPLETED",
            "name": "ASUS Nexus 7 – 2nd Gen (WiFi)",
            "created": 1443331616.586,
            "result": "PASSED",
            "device": {
                "formFactor": "TABLET",
                "name": "ASUS Nexus 7 – 2nd Gen (WiFi)",
                "resolution": {
                    "width": 1200,
                    "height": 1920
                },
                "image": "NA",
                "platform": "ANDROID",
                "heapSize": 0,
                "memory": 34359738368,
                "model": "Nexus 7 – 2nd Gen",
                "os": "4.3.1",
                "cpu": {
                    "frequency": "MHz",
                    "architecture": "ARMv7",
                    "clock": 1512.0
                },
                "arn": "arn:aws:devicefarm:us-west-2::device:208FE7EE973042EA97DEC2EEF31CD10A",
                "manufacturer": "ASUS"
            },
            "message": "Successful test lifecycle of Setup Test",
        "arn": "arn:aws:devicefarm:us-west-2:705582597265:job:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066",
            "counters": {
                "skipped": 0,
                "warned": 0,
                "failed": 0,
                "stopped": 0,
                "passed": 7,
                "errored": 0,
                "total": 7
            }
        },
        {
            "status": "COMPLETED",
            "name": "Amazon Kindle Fire HDX 7 (WiFi)",
            "created": 1443331615.56,
            "result": "FAILED",
            "device": {
                "formFactor": "TABLET",
                "name": "Amazon Kindle Fire HDX 7 (WiFi)",
                "resolution": {
                    "width": 1920,
                    "height": 1200
                },
                "image": "NA",
                "platform": "ANDROID",
                "heapSize": 0,
                "memory": 17179869184,
                "model": "Kindle Fire HDX 7",
                "os": "4.4.3",
                "cpu": {
                    "frequency": "MHz",
                    "architecture": "ARMv7",
                    "clock": 2150.0
                },
                "arn": "arn:aws:devicefarm:us-west-2::device:BC44B6802F134918BDAB6FB4F38C37CC",
                "manufacturer": "Amazon"
            },
        "message": "I see the text "In Theaters" failed: Timeout waiting for elements: * {text CONTAINS[c] ‘In Theaters’} (Calabash::Android::WaitHelpers::WaitError)",
        "arn": "arn:aws:devicefarm:us-west-2:705582597265:job:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00018",
            "counters": {
                "skipped": 0,
                "warned": 0,
                "failed": 5,
                "stopped": 0,
                "passed": 2,
                "errored": 0,
                "total": 7
            }
        }

Note the hierarchical nature of the ARN.  The ARN type is “job” and the job ID is appended to the Run ARN: arn:aws:devicefarm:us-west-2:705582597265:job:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066

Step 3: Retrieve the Status and Result Counters for all Suites within a Job:

Use the list-suites CLI command to list all Suites within a job along with suite-level status and result counters.

[ec2-user]$  aws devicefarm list-suites –arn arn:aws:devicefarm:us-west-2:705582597265:job:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066

Output (partial)

    "suites": [
        {
            "status": "COMPLETED",
            "name": "Setup Suite",
            "created": 1443331656.183,
            "result": "PASSED",
            "message": "Successful test lifecycle of Setup Test",
            "arn": "arn:aws:devicefarm:us-west-2:705582597265:suite:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00000",
            "counters": {
                "skipped": 0,
                "warned": 0,
                "failed": 0,
                "stopped": 0,
                "passed": 1,
                "errored": 0,
                "total": 1
            }
        },
        {
            "status": "COMPLETED",
            "name": "STARmeter",
            "created": 1443331656.221,
            "result": "PASSED",
            "message": "Given I see the text "In Theaters" passed",
            "arn": "arn:aws:devicefarm:us-west-2:705582597265:suite:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00001",
            "counters": {
                "skipped": 0,
                "warned": 0,
                "failed": 0,
                "stopped": 0,
                "passed": 1,
                "errored": 0,
                "total": 1
            }
        }

The returned ARN type is “suite” and the suite ID is appended to the end: arn:aws:devicefarm:us-west-2:705582597265:suite:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00006
 

Step 4: Retrieving the Status and Result Counters for all Tests within a Suite

Use the list-tests CLI command to list all tests within a suite along with test-level status and result counters.

[ec2-user]$  aws devicefarm list-tests –arn arn:aws:devicefarm:us-west-2:705582597265:suite:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00006

Output:

{
    "tests": [
        {
            "status": "COMPLETED",
            "name": "Teardown Test",
            "created": 1443331656.424,
            "result": "PASSED",
            "message": "Successful test lifecycle of Teardown Test",
        "arn": "arn:aws:devicefarm:us-west-2:705582597265:test:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00006/00000",
            "counters": {
                "skipped": 0,
                "warned": 0,
                "failed": 0,
                "stopped": 0,
                "passed": 1,
                "errored": 0,
                "total": 1
            }
        }
    ]
}

The returned ARN type is “test” and the Test ID is appended to the end of the ARN": "arn:aws:devicefarm:us-west-2:705582597265:test:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00006/00000
 

Step 5: Retrieving the Artifacts for a Run, Job, Suite or Test

Using the list-artifacts CLI command, you can retrieve all of the artifacts associated with any of the levels of the run by specifying the corresponding ARN.  For example:

• List artifacts for the run by specifying the run ARN:
  [ec2-user]$  aws devicefarm list-artifacts –arn arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917 –type FILE

• List artifacts for the job by specifying the job ARN:
  [ec2-user]$  aws devicefarm list-artifacts –arn arn:aws:devicefarm:us-west-2:705582597265:job:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066 –type FILE

• List artifacts for the suite by specifying the suite ARN:
  [ec2-user]$  aws devicefarm list-artifacts –arn arn:aws:devicefarm:us-west-2:705582597265:suite:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00000 –type FILE

• List artifacts for the individual test by specifying the test ARN:
  [ec2-user]$  aws devicefarm list-artifacts –arn arn:aws:devicefarm:us-west-2:705582597265:test:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00006/00000 –type FILE

Note that there are three types of artifacts available: FILE, LOG and SCREENSHOT.

The output of the list-artifacts CLI command will include a pre-signed S3 URL that you can use to securely download the artifact:

{
    "artifacts": [
        {
            "url": "https://prod-us-west-2-results.s3-us-west-2.amazonaws.com/705582597265/2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00000/00000/3d811922-64d2-4d0c-8195-ffb715300836.logcat?AWSAccessKeyId=AKIAI3BDGZ5S7TQL2OZA&Expires=1446165402&Signature=D02sc66m01XXYtSJV0hVGK4l9aA%3D",
            "extension": "logcat",
            "name": "Logcat",
            "arn": "arn:aws:devicefarm:us-west-2:705582597265:artifact:2c21a412-bb7b-4657-a28c-d7d78b3888f7/16b3ce7d-63de-4371-bbc7-fdcb20b3f917/00066/00000/00000/00000"
        }
    ]
}

Conclusion

In this two-part series we have demonstrated the use of the AWS CLI to submit a Calabash-Android test script to Device Farm for testing against real mobile devices.  We have also demonstrated the use of the CLI to retrieve the status of the test Run as well as to retrieve the reports and artifacts for your test Run. If you have questions, please let us know in the comments section or in our developer forum.

AWS Device Farm is an app testing service that enables you to test your native, hybrid, and web apps on real Android and iOS devices that are hosted in the AWS Cloud. A test report containing high-level results, low-level logs, pixel-to-pixel screenshots, and performance data is updated as tests are completed.

Device Farm allows you to upload your own tests or use built-in, script-free compatibility tests. Because testing is automatically performed in parallel, test results are available in minutes. Device Farm supports native and hybrid Android, iOS, and Fire OS apps, including those created with PhoneGap, Titanium, Xamarin, Unity, and other frameworks. It also supports testing web apps in Chrome on Android and Safari on iOS. You can run Built-in-Fuzz tests and Explorer tests or utilize your favorite testing framework including Appium, Calabash, Espresso, and UI Automation. 

The purpose of this walkthrough is to show you how you can quickly submit Calabash test scripts to AWS Device Farm leverageing the AWS Command Line Interface (CLI) for Device Farm. Although the walkthrough uses an Android application and Calabash, you can follow the same steps to test iOS and web apps, as well as use one of the many supported frameworks.

Key Concepts

Before we get started, let’s review a few key AWS Device Farm concepts:

• PROJECT – A project in Device Farm enables you to organize your tests in whatever way you choose. For example, there can be one project per app title, or there can be one project per platform. You can create as many projects as you need.
• RUN – A run in Device Farm represents a specific build of your app, with a specific set of tests, to be run on a specific set of devices.
• DEVICE POOL – A device pool is a pre-configured set of devices that can be used as part of a run. 

Scheduling a Device Farm Run Using the CLI

Prerequisite: Before you can create a run in Device Farm, you must install the AWS Command Line Interface (CLI).

Once you install the CLI, there are five steps to create a Calabash test run in AWS Device Farm:

1. Create a Device Farm project
2. Upload your application (Android APK file)
3. Upload your Calabash test scripts
4. Create a device pool
5. Schedule a run

Step 1: Create a Device Farm Project

The first step is to create a Device Farm project. You can also use an existing project if you’ve already created one.

[ec2-user]$  aws devicefarm create-project –name adndroidTest

Output:

[ec2-user@ip-10-0-19-179 ~]$ aws devicefarm create-project –name calabashCLI
{
    "project": {
        "name": "calabashCLI",
        "arn": "arn:aws:devicefarm:us-west-2:705582597265:project:2c21a412-bb7b-4657-a28c-d7d78b3888f7",
        "created": 1442719623.245
    }
}

Note the project ARN as you will reference it when creating your run in the upcoming steps.

Step 2: Create an Upload for Your Android APK File

AWS Device Farm leverages Amazon’s secure and durable S3 service for uploading all objects required for the run.  The upload process utilizes S3 pre-signed URLs to assure a secure upload.

The first step in the upload process is to create your upload request and obtain the S3 pre-signed upload URL:

[ec2-user]$  aws devicefarm create-upload –project-arn arn:aws:devicefarm:us-west-2:705582597265:project:2c21a412-bb7b-4657-a28c-d7d78b3888f7 –name IMDb_5.4.1.105410410_105410410.apk –type ANDROID_APP

The –project-arn is the ARN of your project that you created in Step1: Create a Device Farm Project.

Output:

{
    "upload": {
        "status": "INITIALIZED",
        "name": "IMDb_5.4.1.105410410_105410410.apk",
        "created": 1442719884.445,
        "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A705582597265%3Aproject%3A2c21a412-bb7b-4657-a28c-d7d78b3888f7/uploads/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A705582597265%3Aupload%3A2c21a412-bb7b-4657-a28c-d7d78b3888f7/d1f8bebc-a7f5-49a3-881f-f1ffdae90303/IMDb_5.4.1.105410410_105410410.apk?AWSAccessKeyId=AKIAI3BDGZ5S7TQL2OZA&Expires=1442806284&Signature=HZYPvpPBQn%2FTo%2BWmdSx1LW1nOwE%3D",
        "type": "ANDROID_APP",
        "arn": "arn:aws:devicefarm:us-west-2:705582597265:upload:2c21a412-bb7b-4657-a28c-d7d78b3888f7/d1f8bebc-a7f5-49a3-881f-f1ffdae90303"
    }
}

The pre-signed URL is returned as the url output parameter.

The second step is to upload the actual APK file using the S3 pre-signed URL:

[ec2-user]$  curl -T IMDb_5.4.1.105410410_105410410.apk "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A705582597265%3Aproject%3A2c21a412-bb7b-4657-a28c-d7d78b3888f7/uploads/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A705582597265%3Aupload%3A2c21a412-bb7b-4657-a28c-d7d78b3888f7/d1f8bebc-a7f5-49a3-881f-f1ffdae90303/IMDb_5.4.1.105410410_105410410.apk?AWSAccessKeyId=AKIAI3BDGZ5S7TQL2OZA&Expires=1442806284&Signature=HZYPvpPBQn%2FTo%2BWmdSx1LW1nOwE%3D"

You can check the status of your APK upload using the get-upload CLI command. Specify the ARN returned in the create-upload request above.

[ec2-user]$  aws devicefarm get-upload –arn arn:aws:devicefarm:us-west-2:705582597265:upload:2c21a412-bb7b-4657-a28c-d7d78b3888f7/d1f8bebc-a7f5-49a3-881f-f1ffdae90303

Output:

{
    "upload": {
        "status": "SUCCEEDED",
        "name": "IMDb_5.4.1.105410410_105410410.apk",
        "created": 1442719884.445,
        "type": "ANDROID_APP",
        "arn": "arn:aws:devicefarm:us-west-2:705582597265:upload:2c21a412-bb7b-4657-a28c-d7d78b3888f7/d1f8bebc-a7f5-49a3-881f-f1ffdae90303",
        "metadata": "{"screens": ["small", "normal", "large", "xlarge"], "sdk_version": "14", "package_name": "com.imdb.mobile", "native_code": [], "target_sdk_version": "18", "version_name": "5.4.1.105410410", "version_code": "105410410", "activity_name": "com.imdb.mobile.HomeActivity"}"
    }
}

Once the status is "SUCCEEDED" you are ready to continue to Step 3.

Step 3: Create an Upload for Your Calabash Test Package

In this step, you will upload the Calabash test scripts package. Please refer to the documentation on how to prepare your Android Calabash test package for upload to Device Farm.

The upload process for Calabash test scripts is similar to the upload of the Android APK file in Step 2.

First create an upload request and obtain the S3 pre-signed upload URL:

[ec2-user]$  aws devicefarm create-upload –project-arn arn:aws:devicefarm:us-west-2:705582597265:project:2c21a412-bb7b-4657-a28c-d7d78b3888f7 –name features.zip –type CALABASH_TEST_PACKAGE

Output:

{
    "upload": {
        "status": "INITIALIZED",
        "name": "features.zip",
        "created": 1442724498.222,
        "url": "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A705582597265%3Aproject%3A2c21a412-bb7b-4657-a28c-d7d78b3888f7/uploads/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A705582597265%3Aupload%3A2c21a412-bb7b-4657-a28c-d7d78b3888f7/ccfa27c9-3f66-49ac-abe0-542214cd650c/features.zip?AWSAccessKeyId=AKIAI3BDGZ5S7TQL2OZA&Expires=1442810898&Signature=MOGwu7vUP3q%2B8OrrlSmJ5nVsQe0%3D",
        "type": "CALABASH_TEST_PACKAGE",
        "arn": "arn:aws:devicefarm:us-west-2:705582597265:upload:2c21a412-bb7b-4657-a28c-d7d78b3888f7/ccfa27c9-3f66-49ac-abe0-542214cd650c"
    }
}

Next, upload the Calabash text package ZIP file to the S3 pre-signed URL:

[ec2-user]$  curl -T features.zip "https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A705582597265%3Aproject%3A2c21a412-bb7b-4657-a28c-d7d78b3888f7/uploads/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A705582597265%3Aupload%3A2c21a412-bb7b-4657-a28c-d7d78b3888f7/ccfa27c9-3f66-49ac-abe0-542214cd650c/features.zip?AWSAccessKeyId=AKIAI3BDGZ5S7TQL2OZA&Expires=1442810898&Signature=MOGwu7vUP3q%2B8OrrlSmJ5nVsQe0%3D"

You can check the status of your Calabash test package upload using the get-upload CLI command. Specify the upload ARN returned in the create-upload request above.

[ec2-user]$  aws devicefarm get-upload –arn arn:aws:devicefarm:us-west-2:705582597265:upload:2c21a412-bb7b-4657-a28c-d7d78b3888f7/ccfa27c9-3f66-49ac-abe0-542214cd650c

Output:
{
    "upload": {
        "status": "SUCCEEDED",
        "name": "features.zip",
        "created": 1442724498.222,
        "type": "CALABASH_TEST_PACKAGE",
        "arn": "arn:aws:devicefarm:us-west-2:705582597265:upload:2c21a412-bb7b-4657-a28c-d7d78b3888f7/ccfa27c9-3f66-49ac-abe0-542214cd650c",
        "metadata": "{"valid": true}"
    }
}

Once the status is "SUCCEEDED" you are ready to continue to Step 4.

Step 4: Create a Device Pool Associated with the Project

The AWS Device Farm fleet currently includes over 200 different Android, Fire OS, and iOS devices (see the current device list). You may create Device Pools that consist of specific set of devices that you would like to test against. Your device pools can consist of static lists of devices or rule-based on multiple parameters including device platform (i.e. Android or iOS), manufacturer, and form factor.

The CLI command below creates a Device Pool consisting of all available Android devices:

[ec2-user]$  aws devicefarm create-device-pool –project-arn arn:aws:devicefarm:us-west-2:705582597265:project:2c21a412-bb7b-4657-a28c-d7d78b3888f7 –name myAndroidDevicePool –rules ‘[{"attribute": "PLATFORM", "operator": "EQUALS", "value": ""ANDROID""}]’

Output:

{
    "devicePool": {
        "rules": [
            {
                "operator": "EQUALS",
                "attribute": "PLATFORM",
                "value": ""ANDROID""
            }
        ],
        "name": "myAndroidDevicePool",
        "arn": "arn:aws:devicefarm:us-west-2:705582597265:devicepool:2c21a412-bb7b-4657-a28c-d7d78b3888f7/e4a54fbd-d094-4130-b6d9-60811820e4d8"
    }
}

Step 5: Schedule a Run

To schedule a run, you will need the following input parameters:

• project-arn: the ARN of the project from Step1: Create a Device Farm Project
• app-arn: the ARN of the Application upload from Step 2: Create an Upload for Your Android APK File
• testPackageArn: the ARN of the Calabash test package upload from Step 3: Create an Upload for Your Calabash Test Package
• device-pool-arn: the ARN of the device pool created in Step 4: Create a Device Pool Associated with the Project

The following command schedules a run in Device Farm using the specified Android application, Calabash test package, and device pool:

[ec2-user]$  aws devicefarm schedule-run –project-arn arn:aws:devicefarm:us-west-2:705582597265:project:2c21a412-bb7b-4657-a28c-d7d78b3888f7 –app-arn arn:aws:devicefarm:us-west-2:705582597265:upload:2c21a412-bb7b-4657-a28c-d7d78b3888f7/d1f8bebc-a7f5-49a3-881f-f1ffdae90303 –device-pool-arn arn:aws:devicefarm:us-west-2:705582597265:devicepool:2c21a412-bb7b-4657-a28c-d7d78b3888f7/e4a54fbd-d094-4130-b6d9-60811820e4d8 –name firstCLIRun –test ‘{"type": "CALABASH","testPackageArn":"arn:aws:devicefarm:us-west-2:705582597265:upload:2c21a412-bb7b-4657-a28c-d7d78b3888f7/ccfa27c9-3f66-49ac-abe0-542214cd650c"}’

Output:

{
    "run": {
        "status": "SCHEDULING",
        "name": "firstCLIRun",
        "created": 1442724966.821,
        "totalJobs": 100,
        "completedJobs": 0,
        "platform": "ANDROID_APP",
        "result": "PENDING",
        "billingMethod": "METERED",
        "type": "CALABASH",
        "arn": "arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/61c16425-15c8-4fba-a44f-07dcba60f717",
        "counters": {
            "skipped": 0,
            "warned": 0,
            "failed": 0,
            "stopped": 0,
            "passed": 0,
            "errored": 0,
            "total": 0
        }
    }
}

You can check the status of the run by using the get-run CLI command:

[ec2-user]$  aws devicefarm get-run –arn arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/61c16425-15c8-4fba-a44f-07dcba60f717

Output:

{
    "run": {
        "status": "COMPLETED",
        "name": "firstCLIRun",
        "created": 1442724966.821,
        "totalJobs": 100,
        "completedJobs": 100,
        "platform": "ANDROID_APP",
        "result": "ERRORED",
        "billingMethod": "METERED",
        "type": "CALABASH",
        "arn": "arn:aws:devicefarm:us-west-2:705582597265:run:2c21a412-bb7b-4657-a28c-d7d78b3888f7/61c16425-15c8-4fba-a44f-07dcba60f717",
        "counters": {
            "skipped": 37,
            "warned": 17,
            "failed": 65,
            "stopped": 0,
            "passed": 495,
            "errored": 65,
            "total": 679
        }
    }
}

Once the status is "COMPLETED" your tests on all devices have completed and a full, aggregate summary is available. More on that in Part 2 of this series.

Conclusion

As you can see it’s easy to test your application on dozens of real mobile devices using the Device Farm CLI. Join us in Part 2 of this series where you will retrieve the status and artifacts for your Device Farm Run, including an aggregate summary, log files, and screenshots. If you have any questions please leave a comment below or reach out in our developer forum.

AWS Device Farm is a service that allows you to test your Android, Fire OS, and iOS apps on real devices (not emulators or simulators) in the AWS Cloud. You can upload your apps to the cloud and run Fuzz tests to simulate random activity or leverage the built-in app explorer, which crawls your app and captures screenshots as well as performance metrics and logs. To take further advantage of the service you’ll need to leverage one of the supported test automation frameworks. That’s where this series of blog posts comes in.

In Part One and Part Two of this series you created a basic Android application and configured it to use Espresso for running automated tests and taking screenshots. Now that you have your development and build environment setup you can upload the app to AWS Device Farm.

Upload into Device Farm

Log onto your AWS Account and under Mobile Services select Device Farm. You may be prompted to change to the US West (Oregon) Region.

Click "Getting Started" or the "Create a new project" button if you have an existing Project. Call the Project "Espresso Test."

Go into the Project and select "Create a new Run." Choose the "Upload" button to browse to the APK file that Android Studio generated.

Depending on your operating system, the output location for the APK file will be different. In this simple example we have built an unsigned application for testing purposes on OSX so the default location will be something like:

/Users/username/AndroidStudioProjects/SimpleDemoApp/app/build/outputs/apk

Notice in this directory there will be an "app-debug.apk." This is the file we will upload here but take note of the "app-debug-androidTest-unaligned.apk" file as it will be used next.

Once you upload this file and it is processed by Device Farm you will see a message similar to below.

Press next and select "Instrumentation" as the test type. Again, press the "Upload" button. However, this time select the "app-debug-androidTest-unaligned.apk" file. This is where Android Studio has placed the test cases you created in the project structure before. Once the file uploads successfully you will see the Test Package information and Runner displayed.

If you wanted to you have the capability to apply a test filter and run a single test case rather than an entire suite. This can be very helpful if you have resolved an issue and don’t need to retest an entire suite during development. For this tutorial we will leave the filter blank.

On the next screen you will see a list of devices that your application has compatibility. If you wish you can create a custom device pool or accept the default list of devices.

Press "Next step" and take note of the device state options. If you wanted to on this screen you could modify things such as latitude/longitude, locale, radio states, or install other applications – perhaps if you would like to test dependencies on other apps.

One more thing worth pointing out is your ability to “Add extra data." A good use case for this would be an application, which is quite large and leverages an APK Expansion File. If you have an application that makes use of this capability then this would be the time for you to upload your zip file to the Test Run.

Click "Review and start run." Then, assuming everything looks okay, click "Confirm and start run" to begin testing your application. After a moment you will be able to click on the run and see some initial results. Click on one of the devices such as a Samsung Galaxy S6 (Verizon).

Note the package names in the Test Suite. Click on "com.example.simpledemoapp.EspressoTest" and you will see the "testEspresso" method you created earlier as well as some logging details in the lower pane. Clear the "Source" filter by clicking the "X" next to "Harness" and you will see all of the messages including the logcat messages from your code listed. To filter more specifically you can use wildcards in the "Message" field as seen below.

You can also download the raw logs by clicking on the "Files" tab at the top of the screen and process these files using the tools of your choice.

Finally click the "Screenshots" tab and you will see an image that looks like what you saw earlier when running the sample application locally in your emulator.

Conclusion

That concludes this series on getting started with Android test automation using Espresso and AWS Device Farm. Hopefully you now have the foundation to write your own tests that exercise the functionality and business logic of your Android apps. For additional pointers and sample code for Espresso and other frameworks, check out our sample Android app and tests on GitHub.  If you have questions or comments please join us in the comment section below.