Production quality assurance (QA) workflows require more than individual test execution. You must organize tests into regression suites that run as a batch, and integrate them into continuous integration and continuous delivery (CI/CD) pipelines so that test results gate deployments automatically.

In a previous post, we introduced QA Studio, a reference solution for agentic QA automation built with Amazon Nova Act. We showed how to define individual use cases in natural language, run them on demand through AI-powered visual navigation, and inspect execution artifacts with full trajectory visibility.

In this post, we extend that foundation to demonstrate how QA Studio addresses batch regression testing and pipeline integration through test suites that organize and parallelize execution, and a command-line interface that brings agentic testing into automated CI/CD pipelines.

Test suites for organized regression testing

With QA Studio, you can group individual use cases, each validating a specific user journey, into collections called test suites that run together. These test suites support structured regression testing across functional areas.

Suites execute as a batch with parallel processing: when a suite runs, each use case executes independently on its own Amazon Elastic Container Service (Amazon ECS) on AWS Fargate worker task. Because each use case runs on its own Fargate worker task, a suite of 20 tests can execute concurrently rather than sequentially. This reduces total suite duration relative to serial execution.

You can organize suites by functional area, release stage, or testing purpose. Examples include smoke tests that validate critical paths on every deployment, regression suites that run across the full application, and integration tests that verify cross-feature workflows before release.

Creating and managing test suites

You create test suites in the QA Studio web interface by providing a name, description, and optional tags, then add existing use cases to the suite. Each use case retains its own configuration, including starting URL, variables, secrets, and headers. When the suite executes, these configurations apply independently to each use case.

Figure 1 — Test suite detail page showing use cases and execution history

Suite execution and results

When a suite executes, QA Studio creates individual execution records for each use case and dispatches them to the worker queue. The suite execution page provides an aggregate view: how many use cases succeeded, failed, or are still running. You can drill into individual execution results to review trajectory logs, screenshots, and session recordings for any failed test.

Each suite maintains its own execution history, giving you a longitudinal view of regression stability. Consistent passes build confidence in the tested functionality, while intermittent failures highlight areas that need attention.

Figure 2 — Suite execution results showing aggregate status and individual use case outcomes

CI/CD integration with the QA Studio CLI

The QA Studio web interface works well for interactive test creation and on-demand execution. CI/CD pipelines require a different interface: command-line execution with structured output, non-interactive authentication, and exit codes that integrate with pipeline orchestrators.

The QA Studio CLI (qa-studio) provides this interface. It connects to the same API backend as the web application. But instead of dispatching tests to Fargate workers, it runs them with Amazon Nova Act on the machine where the CLI executes, such as a CI/CD runner. Results are reported back to the QA Studio deployment.

Installing and authenticating

The QA Studio CLI is part of the project’s GitHub repository. Clone the repository, then install the CLI as a Python package with its optional runner dependencies:

pip install -e "./qa-studio-cli[runner]"

For CI/CD environments, the CLI supports OAuth 2.0 client credentials authentication. You create an OAuth client in the QA Studio web interface with the required scopes (api/suite.read, api/suite.write, api/executions.read, api/executions.write, api/usecases.read, api/usecases.execute), then configure the credentials as pipeline environment variables:

export OAUTH_CLIENT_ID="your-client-id"
export OAUTH_CLIENT_SECRET="your-client-secret"
export OAUTH_TOKEN_ENDPOINT="https://your-cognito-domain.auth.region.amazoncognito.com/oauth2/token"

The CLI automatically requests and caches access tokens, refreshing them when they expire. No interactive browser login is required.

Running tests and suites

The qa-studio run command executes individual use cases or entire test suites:

# Run a single test
qa-studio run --usecase-id test-123

# Run a test suite
qa-studio run --suite-id suite-456

Environment and variable overrides

CI/CD pipelines often need to run the same tests against different environments. The CLI supports several override mechanisms that modify test behavior without changing the test definitions stored in QA Studio.

The --base-url flag replaces the domain of the starting URL while preserving the path and query parameters. A single test can then target development, staging, or production environments:

# Run against staging
qa-studio run --suite-id suite-456 --base-url https://staging.example.com

# Run against production
qa-studio run --suite-id suite-456 --base-url https://production.example.com

The --var flag overrides template variables defined in the use case. Variables referenced in test steps using {{VariableName}} syntax are substituted at runtime. This supports environment-specific configuration without duplicating test definitions:

# Override credentials for a specific environment
qa-studio run --usecase-id test-123 \
    --var username=staging_user \
    --var password=staging_pass \
    --var api_key=staging_key_123

The --region flag controls which AWS Region the browser runs in, and --model-id selects the Amazon Nova Act model version:

qa-studio run --usecase-id test-123 \
    --region eu-central-1 \
    --model-id nova-act-v1.0

Headers and secrets

Use cases can define custom HTTP headers that are sent with every request during test execution. This is useful for authentication tokens, feature flags, or custom identifiers that the application under test requires. Headers are configured in the use case settings and applied automatically during both web interface and CLI execution.

Secrets provide secure storage for sensitive values like passwords, API keys, or tokens. Secrets are stored in AWS Secrets Manager, encrypted at rest. QA Studio is designed so that secret values aren’t written to execution logs or history records. Test steps reference secrets by name, and the actual values are retrieved at runtime. This separation means CI/CD pipelines can execute tests that require credentials without exposing those credentials in pipeline configuration or logs.

Exit codes and pipeline integration

The CLI uses exit codes that map directly to pipeline success and failure states:

This three-state model allows pipelines to distinguish between test failures (exit code 1) and infrastructure problems (exit code 2). You can configure different notification or retry strategies for each case.

The --format flag controls output formatting. The default json format provides structured output for programmatic consumption. The human format provides a readable summary for pipeline logs:

qa-studio run --suite-id suite-456 --format human

During execution, the CLI creates execution records in QA Studio, updates step statuses in real time, and uploads artifacts including trajectory logs and session recordings. You can monitor CLI-triggered executions from the web interface alongside manually triggered runs, maintaining a unified execution history regardless of how tests were initiated.

Figure 3 — CLI execution output showing test results

CI/CD platform examples

The following examples demonstrate QA Studio integration with common CI/CD tools. Each example assumes OAuth client credentials and AWS credentials are stored as pipeline secrets.

GitHub Actions

name: QA Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  smoke-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install QA Studio CLI
        run: pip install -e "./qa-studio-cli[runner]"

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1

      - name: Run Smoke Tests
        env:
          OAUTH_CLIENT_ID: ${{ secrets.OAUTH_CLIENT_ID }}
          OAUTH_CLIENT_SECRET: ${{ secrets.OAUTH_CLIENT_SECRET }}
          OAUTH_TOKEN_ENDPOINT: ${{ secrets.OAUTH_TOKEN_ENDPOINT }}
        run: |
          qa-studio run \
            --suite-id ${{ vars.SMOKE_TEST_SUITE_ID }} \
            --base-url https://staging.example.com \
            --format human

      - name: Upload Artifacts
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: test-artifacts
          path: ~/.qa-studio/artifacts/

The if: always() condition on the artifact upload step verifies that test recordings and logs are preserved even when tests fail, providing the debugging context that you need to investigate failures.

GitLab CI

stages:
  - test

smoke-tests:
  stage: test
  image: python:3.11
  before_script:
    - pip install -e "./qa-studio-cli[runner]"
  script:
    - |
      qa-studio run \
        --suite-id $SMOKE_TEST_SUITE_ID \
        --base-url https://staging.example.com \
        --format human
  variables:
    AWS_DEFAULT_REGION: us-east-1
  artifacts:
    when: always
    paths:
      - ~/.qa-studio/artifacts/
    expire_in: 7 days
  only:
    - main
    - merge_requests

regression-tests:
  stage: test
  image: python:3.11
  before_script:
    - pip install -e "./qa-studio-cli[runner]"
  script:
    - |
      qa-studio run \
        --suite-id $REGRESSION_SUITE_ID \
        --timeout 7200 \
        --format human
  variables:
    AWS_DEFAULT_REGION: us-east-1
  artifacts:
    when: always
    paths:
      - ~/.qa-studio/artifacts/
    expire_in: 7 days
  only:
    - schedules

GitLab CI variables (OAUTH_CLIENT_ID, OAUTH_CLIENT_SECRET, OAUTH_TOKEN_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) should be configured as protected and masked CI/CD variables in the project settings. The regression-tests job uses the GitLab schedule trigger, running only when triggered by a pipeline schedule rather than a code push.

Jenkins

pipeline {
    agent any

    environment {
        AWS_DEFAULT_REGION = 'us-east-1'
    }

    stages {
        stage('Setup') {
            steps {
                sh '''
                python3.11 -m venv venv
                . venv/bin/activate
                pip install -e "./qa-studio-cli[runner]"
                '''
            }
        }

        stage('Smoke Tests') {
            steps {
                withCredentials([
                    string(credentialsId: 'oauth-client-id', variable: 'OAUTH_CLIENT_ID'),
                    string(credentialsId: 'oauth-client-secret', variable: 'OAUTH_CLIENT_SECRET'),
                    string(credentialsId: 'oauth-token-endpoint', variable: 'OAUTH_TOKEN_ENDPOINT'),
                    string(credentialsId: 'aws-access-key-id', variable: 'AWS_ACCESS_KEY_ID'),
                    string(credentialsId: 'aws-secret-access-key', variable: 'AWS_SECRET_ACCESS_KEY')
                ]) {
                    sh '''
                    . venv/bin/activate
                    qa-studio run \
                        --suite-id ${SMOKE_TEST_SUITE_ID} \
                        --base-url https://staging.example.com \
                        --format human
                    '''
                }
            }
        }
    }

    post {
        always {
            archiveArtifacts artifacts: '~/.qa-studio/artifacts/**/*',
                allowEmptyArchive: true
        }
    }
}

Jenkins uses the withCredentials block to inject secrets into the build environment without exposing them in console output. The post.always block archives test artifacts regardless of build outcome.

Conclusion

Test suites and CI/CD integration extend QA Studio from an interactive test creation tool into a platform for continuous quality assurance. Test suites organize regression coverage into manageable collections with parallel execution. The CLI brings agentic test execution into automated pipelines with environment overrides, secure credential handling, and exit codes that map to pipeline success and failure states.

These capabilities build on the foundation described in our previous post: natural language test definitions, AI-powered visual navigation, and end-to-end trajectory visibility. Together, they demonstrate how agentic QA automation with Amazon Nova Act can integrate into existing software delivery workflows, providing automated quality feedback without requiring you to maintain framework-specific test code.

In a future post, we plan to explore how agentic test automation can extend to mobile applications.

The QA Studio reference solution, including test suites and CLI integration, is available on GitHub. For deployment instructions and detailed documentation, see the project README.

About the authors