Accelerating software delivery with QA automation using Amazon Nova Act – Part 2

0 1 7 minutes read

Accelerating software delivery with QA automation using Amazon Nova Act – Part 2

Manufacturing quality assurance (QA) workflows require more than performing individual tests. You should organize tests into regression suites that work as a batch, and integrate them into continuous integration and continuous delivery (CI/CD) pipelines so that the test results gateway can be used automatically.

In a previous post, we introduced QA Studio, a reference QA automation solution built with Amazon Nova Act. We've shown how they can describe each use case in natural language, do it on demand with AI-powered virtual navigation, and check for use artifacts with full trajectory visibility.

In this post, we extend that foundation to show how QA Studio handles batch regression testing and integration pipelines with test suites that program and coordinate execution, and a command-line interface that brings agent testing to automated CI/CD pipelines.

With QA Studio, you can combine individual use cases, each validating a specific user journey, into collections called test suites that work together. These test environments support systematic regression testing across all functional areas.

Suites work as a group that is processed in parallel: when the suite is running, each use case runs independently in Amazon Elastic Container Service (Amazon ECS) on an AWS Fargate worker. Because each use case runs on its own Fargate worker task, a set of 20 tests can run concurrently instead of sequentially. This reduces the duration of the suite associated with serial implementation.

You can organize suites by workspace, release stage, or test purpose. Examples include smoke testing that validates critical methods for every implementation, rollbacks that run across a full application, and integration testing that validates the workflow of a cross-feature before release.

Creating and managing checkpoints

You create checkpoints in QA Studio's web interface by providing a name, description, and optional tags, then add existing use cases to the suite. Each use case maintains its own configuration, including the initial URL, variables, secrets, and headers. When the suite starts, this configuration applies independently to each use case.

Figure 1 – Test details page showing usage scenarios and execution history

Suite execution and results

When the suite is signed, QA Studio creates individual usage records for each use case and sends them to the worker queue. The suite's execution page provides an aggregated view: how many use cases have succeeded, failed, or are still running. You can drill into the results of each execution to review trajectory logs, screenshots, and session recordings for any failed tests.

Each suite maintains its own execution history, giving you a long-term view of regression stability. Consistent passes build confidence in the performance tested, while occasional failures highlight areas that need attention.

A suite of signature effects that show combined status and individual usage status results

Figure 2 — Sequential signaling results showing combined status and individual consumption results

QA Studio's web interface is efficient for creating interactive tests and doing what's needed. CI/CD pipelines require a different interface: command-line implementation with structured output, asynchronous validation, and exit codes integrated with pipeline orchestrators.

QA Studio CLI (qa-studio) provides this interface. It connects to the same backend API as the web application. But instead of sending the tests to the Fargate staff, it runs them through Amazon Nova Act on the machine where the CLI is running, like a CI/CD runner. Results are reported back using QA Studio.

It installs and validates

QA Studio CLI is part of the GitHub project repository. Close the terminal, and install the CLI as a Python package with its optional runtime dependencies:

pip install -e "./qa-studio-cli[runner]"

In CI/CD environments, the CLI supports authentication of OAuth 2.0 client credentials. You create an OAuth client in the QA Studio web interface with the required scopes (api/suite.read, api/suite.write, api/executions.read, api/executions.write, api/usecases.read, api/usecases.execute), and configure validation as a pipe environment variable:

export OAUTH_CLIENT_ID="your-client-id"
export OAUTH_CLIENT_SECRET="your-client-secret"
export OAUTH_TOKEN_ENDPOINT="

The CLI automatically requests and stores access tokens, renewing them when they expire. No interactive browser login is required.

Running tests and suites

I qa-studio run command using the usage conditions for individual or all test suites:

# Run a single test
qa-studio run --usecase-id test-123

# Run a test suite
qa-studio run --suite-id suite-456

Nature and dynamics

CI/CD pipelines often need to perform similar tests in different locations. The CLI supports several output methods that change test behavior without changing the test definitions stored in QA Studio.

I --base-url tag replaces the domain of the starting URL while preserving the path and query parameters. A single test can identify development, staging, or production areas:

# Run against staging
qa-studio run --suite-id suite-456 --base-url 

# Run against production
qa-studio run --suite-id suite-456 --base-url

I --var tag emits template variables defined in the implementation context. The variables specified in the test steps are used {{VariableName}} syntax is replaced at runtime. This supports environment-specific configuration without repeating test definitions:

# Override credentials for a specific environment
qa-studio run --usecase-id test-123 
    --var username=staging_user 
    --var password=staging_pass 
    --var api_key=staging_key_123

I --region flag controls which AWS Region the browser is running in, too --model-id selects the version of the Amazon Nova Act model:

qa-studio run --usecase-id test-123 
    --region eu-central-1 
    --model-id nova-act-v1.0

Topics and secrets

The use of contexts can define custom HTTP headers that are sent with every request during test execution. This is useful for authentication tokens, feature flags, or custom identifiers required by the application under test. Headers are configured in the use case settings and are used automatically during both web interface and CLI execution.

Secrets provide secure storage for sensitive values such as passwords, API keys, or tokens. Secrets are stored in AWS Secret Manager, encrypted at rest. QA Studio is designed in such a way that private values can be written to logs or historical records. Steps to check the secrets of the reference by name, and the actual values are returned at runtime. This separation means that CI/CD pipelines can perform the tests they need to validate without revealing that information in the pipeline configuration or logs.

Outlet codes and plumbing connections

The CLI uses exit codes that indicate pipe success and failure:

Exit the code	Explanation	Pipeline behavior
0	All tests passed	The pipeline continues
1	One or more tests failed	Pipeline fails (test failure)
2	CLI error (auth, configuration, API)	Pipeline fails (infrastructure error)

This three-state model allows pipelines to distinguish between test failures (exit code 1) and infrastructure problems (exit code 2). You can prepare a different notification or try again techniques in each case.

I --format flag controls the formatting of the output. Default json format provides structured output for program use. I human format provides a readable summary of pipeline logs:

qa-studio run --suite-id suite-456 --format human

During execution, the CLI creates execution logs in QA Studio, updates step statuses in real time, and loads artifacts including trajectory logs and session recordings. You can monitor CLI-triggered executions from the web interface as well as manually-triggered runs, keeping a consolidated execution history regardless of how the test was initiated.

CLI output showing test results

Figure 3 – CLI output showing test results

The following examples show the integration of QA Studio with common CI/CD tools. Each example assumes OAuth client credentials and AWS credentials are stored as pipeline secrets.

GitHub Actions

name: QA Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  smoke-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install QA Studio CLI
        run: pip install -e "./qa-studio-cli[runner]"

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1

      - name: Run Smoke Tests
        env:
          OAUTH_CLIENT_ID: ${{ secrets.OAUTH_CLIENT_ID }}
          OAUTH_CLIENT_SECRET: ${{ secrets.OAUTH_CLIENT_SECRET }}
          OAUTH_TOKEN_ENDPOINT: ${{ secrets.OAUTH_TOKEN_ENDPOINT }}
        run: |
          qa-studio run 
            --suite-id ${{ vars.SMOKE_TEST_SUITE_ID }} 
            --base-url  
            --format human

      - name: Upload Artifacts
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: test-artifacts
          path: ~/.qa-studio/artifacts/

I if: always() the artifact loading step condition ensures that test recordings and logs are preserved even when tests fail, providing the debugging context you need to investigate failures.

GitLab CI

stages:
  - test

smoke-tests:
  stage: test
  image: python:3.11
  before_script:
    - pip install -e "./qa-studio-cli[runner]"
  script:
    - |
      qa-studio run 
        --suite-id $SMOKE_TEST_SUITE_ID 
        --base-url  
        --format human
  variables:
    AWS_DEFAULT_REGION: us-east-1
  artifacts:
    when: always
    paths:
      - ~/.qa-studio/artifacts/
    expire_in: 7 days
  only:
    - main
    - merge_requests

regression-tests:
  stage: test
  image: python:3.11
  before_script:
    - pip install -e "./qa-studio-cli[runner]"
  script:
    - |
      qa-studio run 
        --suite-id $REGRESSION_SUITE_ID 
        --timeout 7200 
        --format human
  variables:
    AWS_DEFAULT_REGION: us-east-1
  artifacts:
    when: always
    paths:
      - ~/.qa-studio/artifacts/
    expire_in: 7 days
  only:
    - schedules

GitLab CI variables (OAUTH_CLIENT_ID, OAUTH_CLIENT_SECRET, OAUTH_TOKEN_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) must be configured as protected CI/CD variables and covered in the project settings. I regression-tests The task uses GitLab's schedule trigger, which only works when the pipeline schedule is triggered rather than pushing the code.

Jenkins

pipeline {
    agent any

    environment {
        AWS_DEFAULT_REGION = 'us-east-1'
    }

    stages {
        stage('Setup') {
            steps {
                sh '''
                python3.11 -m venv venv
                . venv/bin/activate
                pip install -e "./qa-studio-cli[runner]"
                '''
            }
        }

        stage('Smoke Tests') {
            steps {
                withCredentials([
                    string(credentialsId: 'oauth-client-id', variable: 'OAUTH_CLIENT_ID'),
                    string(credentialsId: 'oauth-client-secret', variable: 'OAUTH_CLIENT_SECRET'),
                    string(credentialsId: 'oauth-token-endpoint', variable: 'OAUTH_TOKEN_ENDPOINT'),
                    string(credentialsId: 'aws-access-key-id', variable: 'AWS_ACCESS_KEY_ID'),
                    string(credentialsId: 'aws-secret-access-key', variable: 'AWS_SECRET_ACCESS_KEY')
                ]) {
                    sh '''
                    . venv/bin/activate
                    qa-studio run 
                        --suite-id ${SMOKE_TEST_SUITE_ID} 
                        --base-url  
                        --format human
                    '''
                }
            }
        }
    }

    post {
        always {
            archiveArtifacts artifacts: '~/.qa-studio/artifacts/**/*',
                allowEmptyArchive: true
        }
    }
}

Jenkins uses the withCredentials prevent entering secrets into the build environment without displaying them in the console output. I post.always block inspection artifacts regardless of build output.

Test suites and CI/CD integration extend QA Studio from an interactive test creation tool to a continuous quality assurance platform. Test suites organize input regressions into manageable clusters with common signatures. The CLI brings the use of agent testing to automated pipelines with native output, secure data management, and exit codes that indicate pipeline success and failure conditions.

These capabilities build on the foundation described in our previous post: natural language test descriptions, AI-powered visual navigation, and end-to-end route visibility. Together, they show how agentic QA automation and Amazon Nova Act can integrate with existing software delivery workflows, providing automated quality feedback without requiring you to maintain framework-specific testing code.

In future posts, we plan to explore how agent testing automation can extend to mobile applications.

The QA Studio reference solution, including test environments and CLI integration, is available on GitHub. For usage instructions and detailed documentation, see the project's README.