Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.testdriver.ai/llms.txt

Use this file to discover all available pages before exploring further.

Parallel testing allows you to run multiple tests simultaneously, significantly reducing the time required to execute your test suite. By leveraging GitHub Actions’ matrix strategy, you can dynamically distribute your tests across multiple jobs, ensuring efficient and scalable test execution.

​
Why use parallel testing?

  1. Faster Execution: Run multiple tests at the same time to reduce overall test duration.
  2. Scalability: Easily scale your testing efforts as your test suite grows.
  3. Dynamic Distribution: Automatically distribute tests across jobs using GitHub’s matrix strategy.
  4. Cost Efficiency: Optimize resource usage by running tests in parallel.
  5. Reliability: Avoid failures from going beyond your plan’s max instances.

​
Setting Up parallel testing with a matrix strategy

1
Ensure your test files are stored in a directory (for example, testdriver/acceptance/) and follow a consistent naming convention (for example, login.yaml, signup.yaml, etc.).
2

Define the GitHub Action Workflow

Here’s an example of a GitHub Action workflow that uses the matrix strategy to run tests in parallel:
name: Parallel Testing with TestDriver

on:
  push:
    branches:
      - main
  pull_request:
  workflow_dispatch:

jobs:
  gather-test-files:
    name: Gather Test Files
    runs-on: ubuntu-latest
    outputs:
      test_files: ${{ steps.test_list.outputs.files }}
    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Find all test files
        id: test_list
        run: |
          FILES=$(ls ./testdriver/acceptance/*.yaml)
          FILENAMES=$(basename -a $FILES)
          FILES_JSON=$(echo "$FILENAMES" | jq -R -s -c 'split("\n")[:-1]')
          echo "files=$FILES_JSON" >> $GITHUB_OUTPUT

  run-tests:
    name: Run Tests in Parallel
    needs: gather-test-files
    strategy:
      matrix:
        test: ${{ fromJson(needs.gather-test-files.outputs.test_files) }}
      fail-fast: false
      max-parallel: 8

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Run test in headless mode
        run: npx testdriverai@latest run testdriver/acceptance/${{ matrix.test }}
        env:
          FORCE_COLOR: 3
          TD_API_KEY: ${{ secrets.TD_API_KEY }}
          TD_WEBSITE: https://testdriver-sandbox.vercel.app
          TD_THIS_FILE: ${{ matrix.test }}

​
Explanation of the workflow

  1. gather-test-files Job:
    • Collects all test files in the testdriver/acceptance/ directory.
    • Uses basename to get just the filenames without the full path.
    • Outputs the list of test files as a JSON array for the matrix strategy.
  2. run-tests Job:
    • Uses the matrix strategy to dynamically create a job for each test file.
    • Runs each test file in parallel using npx testdriverai@latest with the --headless flag.
    • The max-parallel setting limits the number of tests that can run in parallel to 8 (the max instances for the Medium Team Plan).
  3. Matrix Strategy:
    • The matrix.test variable represents each test filename.
    • Each job runs a single test file from the testdriver/acceptance/ directory.
  4. Environment Variables:
    • TD_API_KEY: Your TestDriver API key for authentication.
    • TD_WEBSITE: The target website URL for testing if its a web application.
    • TD_THIS_FILE: The current test file being executed, this will also be the title of the test recording.
  5. Fail-Fast:
    • Set to false to ensure all tests run even if one fails.

​
Benefits of using the matrix strategy

  1. Dynamic Test Distribution: Automatically adapts to the number of test files.
  2. Scalable: Easily handles large test suites by distributing tests across multiple jobs.
  3. Efficient Resource Usage: Runs tests in parallel, reducing idle time.

​
Example output

When this workflow runs:
  • Each test file in testdriver/acceptance/ (for example, login.yaml, signup.yaml) is executed in its own job.
  • The results of all tests are displayed in the GitHub Actions dashboard with clear matrix job names.
  • Failed tests can be easily identified and debugged individually.

​
Best practices

  1. Organize Test Files: Use a consistent naming convention for test files to simplify management.
  2. Monitor Test Results: Review the GitHub Actions dashboard to identify and debug failing tests.
  3. Optimize Test Files: Ensure each test file is self-contained and doesn’t depend on the execution of other tests.
  4. Use Fail-Fast Judiciously: Enable fail-fast: true only if you want to stop all tests when one fails.

​
Conclusion

Parallel testing with the GitHub Action matrix strategy is a powerful way to speed up your TestDriver test suite. By dynamically distributing tests across multiple jobs, you can ensure efficient execution and scalability, making it easier to maintain high-quality software.