Local Runners

Follow the Local Agent Setup to run on your own machine for free. You'll use your own computer to create and run tests.

Hosted Linux Runners

We recommend building and running your tests with our linuxGitHub Actions. You can run multiple tests in parallel and deploy them to CI/CD with our GitHub Action Setup.

Hosted Windows Runners

If you're building a desktop app, need more coverage, more power, or want to test more complex flows, we recommend using our hosted Windows Runners.

Application Support

TestDriver opeates a full desktop environment, so it can run any application.

Testing Features

TestDriver is AI first.

Test Coverage

Testdriver has more coverage than selector based frameworks.

Debugging Features

Debugging features are powered by Dashcam.io.

Web Browser Support

TestDriver is browser agnostic and supports any verson of any browser.

Operating System Support

TestDriver currently supports Mac and Windows!

With TestDriver’s AI-powered QA Agents, creating and deploying robust UI tests has never been faster or easier. Here's how to get up and running in no time.

Generate your first test suite

The fastest way to create your first set of tests is by running the demo. TestDriver will automatically log into your application, generate a dozen UI, and push them directly to your GitHub repository.

Follow the guide here to generate your first test suite.

Write your own tests with the Computer-Use agent

Getting started with TestDriver is simple. Just download the TestDriver Agent to enable test generation directly from your environment.

Need Help? Watch the Setup Video

Follow along with our step-by-step YouTube setup guide to see how easy it is to install the agent, connect your app, and start generating tests—no guesswork needed.

Set Up Your First TestDriver Project

To get started, install the testdriverai package globally using Yarn. This package enables you to generate tests effortlessly using natural language commands.

yarn add global testdriverai@beta

Next, create a new directory for your test project and initialize it with testdriverai init.

mkdir testdriver-first-project
cd testdriver-first-project
testdriverai init

This will launch the interactive setup process:

Beginning setup...
✔ Use TestDriver Runners? (Recommended) … yes
? API KEY (from https://app.testdriver.ai/team) ›

TestDriver Runners are the fastest way to get started, but you'll need an TestDriver Pro plan. See Getting an API Key.

You can use your own computer as a runner for free. See Local Agent Setup

How To Get an API Key

Subscribe to TestDriver Pro in the TestDriver Dashboard to access your API key. Every Pro plan comes with $100 of free credits to help you hit the ground running.

Paste your API key into the prompt when asked, then follow the remaining steps in the setup to complete your project configuration.

Build tests with the Computer-Use Agent

Now comes the fun part—building your tests! To get started, trigger the Computer-Use Agent by running:

testdriverai

This launches TestDriver in interactive mode. The agent will begin listening for commands and automatically generate test steps based on your input. All generated commands will be saved to testdriver/testdriver.yaml

Give Instructions in Plain English

Now, just tell TestDriver what you want it to do. For our example, try click new task.

> click new task

What Happens Next

TestDriver will analyze your screen, interpret your instruction, and generate a full test script for you. It can see your UI, control the mouse and keyboard, and automate complex workflows—all through simple commands.

See the yml TestDriver generated? That's our own schema. You can learn more about it in the Test Steps.

> click new task

    thinking...

    To accomplish the goal of clicking "New Task" on the
    webpage, we need to focus on the browser window and then
    click the "New Task" button.

    Here is the YAML code to achieve this:

    commands:
      - command: focus-application
        name: Google Chrome
      - command: hover-text
        text: New Task
        description: button to create a new task
        action: click

command='focus-application' name='Google Chrome'
command='hover-text' text='New Task' description='button to create a new task' action='click'

    thinking...

    Based on the provided coordinates and text, the element
    that matches "New Task" with the description "button to
    create a new task" is at coordinates (864, 328).

    Here is the YAML code to click on that element:

    commands:
      - command: click
        x: 864
        y: 328
        click: single
        button: left

command='click' x=864 y=328 click='single' button='left'

Keep going!

You can continue instructing TestDriver with natural commands. It will visually inspect your app and generate the next steps for you. Try:

> enter a task title
> enter a task summary
> click create task
> delete the task

Test It Locallay Before Deployment

Before pushing to GitHub, it’s a good idea to validate your test locally. After you’ve saved the test using /save, run:

testdriverai run testdriver/testdriver.yml

Deploy tests to GitHub

Ready to ship your tests? Check out our CI/CD integration guides to learn how to deploy tests to GitHub and automate them in your pipeline.

The fastest way to get started with TestDriver

TestDriver will crawl your app and generate tests. It interacts with each element and continues to come up with new ideas.

TestDriver can generate 1,000s of tests automatically. Tests are opened as Pull Requests. All you need ot do is review the created tests and merge them into your test suite!

Demo

The easiest way to generate tests for your app is to use the form below. This will generate up to 10 tests for you in a new GitHub repository.

Reviewing Pull Requests

TestDriver will generate tests for your app. Each pull request description will contain:

2. The test result

Debugging Pull Requests

Making Changes

TestDriver may try multiple times to accomplish a goal.

Not every test will complete the objective entirely, but the steps that do complete successfully make great regression tests and it usually only takes a little modification to make a successful test!

We recommend:

1. Deleting the test steps that did not complete successfully.

Deploying Your Test (Requires Upgrade)

This repository is set up to automatically run any merged regression tests! You'll need an API key to do so, and you can only get one by upgrading your account to Pro or higher.

Where to go from here?

Executes tasks based on user input using natural language processing. This command is invoked when the user input does not start with a / command. The system interprets the input and attempts to carry out the task specified by the user.

Example Usage

Prompting Tips

The agent is selecting the wrong thing!

This is the most common issues encountered with our agent, here are some possible reasons.

You're asking the AI to interact with elements it can not see

TestDriver uses the context from your prompt and the computer screen to make a decision of what commands to run. You should only prompt the AI to interact with elements it can currently see.

Incorrect Prompt

A common example of this is interacting with a dropdown. We often see users prompt the agent to interact with a dropdown and choose a state.

Recommended prompts

Instead, simply treat these as two separate prompts. This allows the UI to render and gives the AI the opportunity to parse the new screen data.

You're asking the AI to click on elements it does not understand

The TestDriver agent relies on visual understanding, not functional. Like any user, the AI does not understand what the function of a button will be. It can only guess.

Incorrect Prompt

Correct Prompts

Describing Images Properly

If you're uncertain of how to describe an icon, simply ask ChatGPT-4o what it would call it, and use that as your input.

The AI can not find small images

Small, isolated images smaller than 15x15px appear like "noise" to the AI and may not be clickable. However, you can use the match-imagecommand to select these using manually made screenshots.

No matter what I do, TestDriver will not select my element.

The AI has trouble selecting some specific elements, like empty gray boxes, some substrings, or conditions where there is a lot of similar text close together.

Deploying your TestDriver tests to CI/CD requires an API key!

1. Head over to https://app.testdriver.ai to get an API key

2. Upgrade your account to TestDriver Pro (Every upgrade comes with $100 in free usage credits!)

3. Copy the API key from your developer portal

We know our customers need QA coverage yesterday. While other QA services offer coverage delivered months later for hundreds of thousands of dollars, we'll set you for free during our 30-day trial!

Pricing

We think you're going to love TestDriver, which is why for a limited time our team will build 30 custom tests for you as a part of our risk-free, 30 day trail.

Afterward, plans start at just $995/m and include 12,000 runner minutes per month. That's enough minutes to run those 30 tests 2 - 5 times per day. See Promotion Details for more.

Promotion Details

• Test any publicly available desktop app, chrome extension, mobile app, or website

• Choose from 250 AI generated tests within 7 days.

• Run tests 3 - 5 times every day.

• AI Quality reports delivered to your email.

• AI fixes and maintains tests.

• Integrates with GitHub.

• Full access to our enterprise test dashboards

See Contract Details for more.

How does it work?

1. Book an Onboarding call so we can learn about the specific flows you want to test

2. We'll set up Generate a Test Suite to explore your app and generate 100s of tests.

3. Your new tests will be deployed to our GitHub Actions.

4. Our support will work with you to create tests for any features TestDriver might have missed.

5. Our support team will help train your engineers to create additional tests

Onboarding Process

Contract Details

• Get 30 custom tests free during a 30-day trial when subscribing to $995/m "30x30" plan

• The tests are yours! You are free to modify, duplicate, or distribute tests however you wish

• Payment method is required to begin trial

• Contract renews monthly

• Cancel any time, for any reason

• Credits do not "roll over." Unused credits expire every month

• Additional usage is billed at standard rates

You can use your own computer as a TestDriver runner. TestDriver will 'alt-tab' between the terminal and your computer.

Just tell TestDriver what to test on your local machine and TestDriver will look at the screen, generate some commands to control the computer, and execute them.

Extra Setup

Running tests on your own machine requires a bit of extra setup. You'll need to install some dependencies and give the terminal permission to see your screen and file system access.

NodeJS

You'll need NodeJS version 20 to get started. Windows users will need a couple extra tools.

choco install python visualstudio2022-workload-vctools -y

choco install nodejs-lts --version="20.17.0"

Set-ExecutionPolicy RemoteSigned -Scope CurrentUser

C:\Users\<your-username>\AppData\Local\Microsoft\WindowsApps 

winget install python

winget install --id=Microsoft.VisualStudio.2022.Community --override "--add Microsoft.VisualStudio.Workload.VCTools" --accept-package-agreements --accept-source-agreements

winget install --id=OpenJS.NodeJS.LTS --version="20.17.0"

Set-ExecutionPolicy RemoteSigned -Scope CurrentUser

brew install nvm

nvm install 20.17.0

Install TestDriver via NPM

Install testdriverai via NPM. This will make testdriverai available as a global command.

npm install testdriverai -g

Set up the project

Run testdriverai init in a new folder.

testdriverai init

This will walk you through setting up a local project.

Set up your test environment

Before we get started, let's set up your machine to collaborate with TestDriver.

Display

TestDriver isn't like any framework you've used before. TestDriver makes decisions based on what it can see on your display!

For now, set up your environment with a browser window and your terminal side by side like so:

When you enter commands into TestDriver, the current terminal window will minimize and the focus-window command will bring Chrome or other applications to the foreground.

Application State

The application you want to test should be visible before you run the testdriverai command.

For our example, make a new incognito window in Chrome and load our test webpage:

https://testdriverai.github.io/example-react-todo/

Running in CI/CD

CI/CD pipelines enhances software quality and release velocity. Adding TestDriver to your CI/CD, you can catch defects early, prevent regressions, and gain rapid feedback on code changes.

Our GitHub Action Setup allows you to run the tests you designed on your local computer in a virtual machine as part of your CI/CD workflow.

The TestDriver GitHub action will execute tests when you mention it in a comment, on a schedule, or anywhere else in your CI/CD workflow.

Continuous Integration

TestDriver can be integrated with GitHub Actions to automatically run tests whenever code is pushed to a repository. This ensures that new changes do not break existing functionality.

Automated Code Reviews

TestDriver can be configured to run specific test suites when a pull request is created. This ensures that any code changes meet the project's quality standards before being merged.

Increased Test Coverage

TestDriver can automatically generate and run test cases across multiple environments, providing detailed feedback on test coverage and potential issues.

Improved Developer Experience

Developers get immediate feedback on the success or failure of their changes within the pull request itself, reducing the time spent on manual code reviews.

Installing the GitHub action

Get your API Key

In order to execute your TestDriver actions on our VMs you'll need to add your API key as a GitHub secret. If you don't see an API key, you'll need ot upgrade your account.

Create your workflow

Now it's time to create your first TestDriver workflow.

In .github/workflows/testdriver.yml add the following code.

Output

How it works

1. The GitHub action is triggered via the conditions supplied via on

2. The key value is used to authenticate you

3. An ephemeral virtual machine is spawn on our infrastructure

4. The code from the current branch will be cloned on to the VM

7. prompt is parsed as a markdown list.

8. Each list item from promp is fed into TestDriver one by one

9. TestDriver summarizes the test and sets it's exit code depending on it's pass or failed state

11. The VM is destroyed and all data is wiped

Deploy the test

Save the file, make a new branch, push to your repository, and create a new pull request.

This will trigger a new TestDriver execution.

The, simply parallelize your test executions by calling the TestDriver action multiple time as a part of two different jobs.

Here's an example of testing a matrix of files.

Prerun scripts are Bash commands executed on a TestDriver VM before each test within a CI/CD pipeline. Their primary purpose is to establish the state of a machine. This ensure it is consistent before every test execution.

You can configure prerun script to install necessary dependencies, build your application, set specific configurations, and more.

This crucial step helps to speed up the setup of an environment, prepare for a test suite to run, and prevent test failures due to environment inconsistencies and promote reproducible builds, ultimately enhancing the overall test suite's effectiveness.

Example

This is an example of how to download Arc Browser and use it instead of Chrome.

To configure the operating system, simply supply the os param to the GitHub action. The current available operating systems are windows and mac.

Prerun files

Note that prerun files are executed on these machines themselves. You must use powershell on Windows and bash on mac.

Testing multiple browsers and operating systems

While TestDriver is incredibly smart, using AI matching methods all the time can be slow. Nobody wants to block developers from merging while waiting for tests to run!

Here are some tips for improving TestDriver performance.

Use Parallel Testing

Covered in the previous section, use runand Parallel Testing to split your actions into multiple files and run them.

Use ai matching method

The most common actions like hover-text, wait-for-text , and scroll-until-text use an optimized matching algorithm.

This algorithm uses text similarity to quickly compute the most similar text to what appears in your yml. This is about 40% faster than the ai method!

Usse `async` Asserts

The assert method has property async: true which allows you to create non blocking test assertions costing you almost no time!

The TestDriver action outputs the following variables. You can chain mlutliple actions together to post TestDriver results as comments, send an email on failure, or upload them to 3rd party test reporting software.

Example

Here's an example of creating a comment on the PR after every execution.

name: TestDriver.ai

permissions:
  actions: read
  contents: read
  statuses: write
  pull-requests: write

on:
  pull_request:

jobs:
  test:
    name: "TestDriver"
    runs-on: ubuntu-latest
    id: run-testdriver
    steps:
      - uses: dashcamio/testdriver@main
        version: v4.0.0
        key: ${{secrets.TESTDRIVER_API_KEY}}
        with:
          prompt: |
            1. /run /Users/ec2-user/actions-runner/_work/testdriver/testdriver/.testdriver/test.yml
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          FORCE_COLOR: "3"
    - name: Create comment on PR
      if: ${{ always() }}
      uses: peter-evans/create-or-update-comment@v3
      with:
        issue-number: ${{steps.get_issue_number.outputs.result}}
        body: |
          ${{ needs.run-testdriver.outputs.summary }}
          ${{ needs.run-testdriver.outputs.markdown }}

Test Generation Parallel Testing Importing Tests Desktop Apps Secure Log In

TestDriver will worry about generating and maintaining tests for the most part. However, if you'd like to edit tests or gain a better understanding of what's going on you can find all of the commands in this secion.

As for YML format, here is an example of a valid yml file:

version: 4.0.0
steps:
  - prompt: enter fiber.google.com in url
    commands:
      - command: focus-application
        name: Google Chrome
      - command: hover-text
        text: Search Google or type a URL
        description: main google search
        action: click
      - command: type
        text: fiber.google.com
      - command: press-keys
        keys:
          - enter
  - prompt: enter a fake address and check availability
    commands:
      - command: focus-application
        name: Google Chrome
      - command: hover-text
        text: Enter your address
        description: address input field
        action: click
      - command: type
        text: 123 Fake Street
      - command: hover-text
        text: ZIP
        description: ZIP code input field
        action: click
      - command: type
        text: 12345
      - command: hover-text
        text: Check availability
        description: check availability button
        action: click
  - prompt: assert a familiy appears on screen
    commands:
      - command: focus-application
        name: Google Chrome
      - command: assert
        expect: a family appears on screen

This workflow uses parallel testing to test several URLs in a short amount of time

This workflow:

• Runs on each push to the "main" branch & every day at midnight

• Defines a matrix of tests to be ran on individual, non-sequential, ephemeral VMs

• Runs the matrix of tests

  • Each test in the matrix will have Dashcam installed, and Chrome opened to the respective URL defined in the URL value in the matrix

name: TestDriver.ai

permissions:
  actions: read
  contents: read
  statuses: write
  pull-requests: write

on:
  push:
    branches: ["main"]
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: "0 0 * * *"

jobs:
  testdriver_matrix:
    name: "TestDriver Matrix"
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        test:
          - name: "Functions testdriver"
            file: "testdriver/functions.yml"
            url: "https://example.com"
          - name: "Chat testdriver"
            file: "testdriver/chat_test.yml"
            url: "https://www.exmaple.com/demos/chat/"
          - name: "P2P dynamic payment testdriver"
            file: "testdriver/p2p_dynamic_payment.yml"
            url: "https://www.exmaple.com/demos/fintech/"
          - name: "Dynamic matchmaking testdriver"
            file: "testdriver/skill_based_matchmaking.yml"
            url: "https://www.example.com/demos/skill-based-matchmaking-dashboard/"
          - name: "Support search testdriver"
            file: "testdriver/support_search.yml"
            url: "https://support.example.com/hc/en-us"
          - name: "Ask AI testdriver"
            file: "testdriver/ask_ai.yml"
            url: "https://www.example.com"
          - name: "Edit settings testdriver"
            file: "testdriver/edit_account_settings.yml"
            url: "https://www.example.com"
          - name: "Debug console testdriver"
            file: "testdriver/debug_console.yml"
            url: "https://www.example.com"
    steps:
      - uses: testdriverai/action@main
        with:
          key: ${{ secrets.TESTDRIVER_API_KEY }}
          prompt: |
            1. /run ${{ matrix.test.file }}
          prerun: |
            cd $env:TEMP
            npm init -y
            npm install dashcam-chrome
            Start-Process "C:/Program Files/Google/Chrome/Application/chrome.exe" -ArgumentList "--start-maximized", "--load-extension=$(pwd)/node_modules/dashcam-chrome/build", "${{ matrix.test.url }}"
            exit
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          FORCE_COLOR: "3"

Every TestDriver CI test is recorded with Dashcam, the first TestDriver product we built to help debug remote test executions.

Dashcam captures the video of the desktop test execution, the browser logs, network requests, prerun script output, and of course the TestDriver test execution. You can play it all back in the TestDriver dashboard!

Dashcam dashes are transmitted over HTTS, encrypted at test, and have "google style" sharing permissions. Sensitive data in logs is masked.

When a test finishes, a Dashcam clip (a dash for short) is into GitHub automatically. This enables developers to see a test artifact and collaborate directly from within GitHub.

You an learn more about Dashcam on the Dashcam documentation page.

This workflow will generate tests using the exploratory prompts found in the promptvalue of the GitHub Actionsconfiguration.

The promptvalue takes a markdown list as input.

name: TestDriver.ai

permissions:
  actions: read
  contents: read
  statuses: write
  pull-requests: write
  
on:
  push:
    branches: ["main"]
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: "0 0 * * *"

jobs:
  test:
    name: "TestDriver"
    runs-on: ubuntu-latest
    steps:
      - uses: testdriverai/action@main
        with:
          key: ${{secrets.TESTDRIVER_API_KEY}}
          prompt: |
            1. Search for cat pictures
            2. Download the first image to the desktop
            3. Assert the cat picture is saved to the desktop
          prerun: |
            cd $env:TEMP
            npm init -y
            npm install dashcam-chrome
            Start-Process "C:/Program Files/Google/Chrome/Application/chrome.exe" -ArgumentList "--start-maximized", "--load-extension=$(pwd)/node_modules/dashcam-chrome/build", "${{ env.WEBSITE_URL }}"
            exit
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          FORCE_COLOR: "3"
          WEBSITE_URL: "https://example.com" # Define the website URL here

This command is useful for interacting with elements that the AI has trouble locating. The testdriverai package will take a screenshot of the desktop and search for the location of the image within the screenshot.

Screenshot should be stored in testdriver/screenshots/(mac/linux/windows)/PATH.png . TestDriver will dynamically resolve images based on the current platform.

The screenshot template matching logic looks for the most similar image within the screenshot and not exact matches. If the match is not above ~80%, it will search additional scales. Otherwise it fails

To create screenshots from remote tests, download the video of the test and open it "full" or "actual" size within your computer. Then use a screenshot tool like Cleanshot X to create a screenshot of the target element. Do the best you can to center the clickable element within the screenshot.

Example Usage

command: match-image
relativePath: button.png
action: click

Example Usage

command: hover-text
text: Sign Up
description: link in the header
action: click

This workflow defines TD_USERNAME and TD_PASSWORD in env: to existing repository secrets of the same name. These TD_USERNAME and TD_PASSWORD can then be used in any YAML passed into the workflow. Secrets must begin with TD_ in order for Testdriver to be able to see them.

Example YAML steps using secrets could look like:

steps:
  - prompt: input email ${TD_TEST_USERNAME} then press continue
    commands:
      - command: hover-text
        text: Email address
        description: email input field label
        action: click
      - command: type
        text: ${TD_USERNAME}
      - command: hover-text
        text: CONTINUE
        description: continue button below the email input
        action: click
  - prompt: input password ${TD_TEST_PASSWORD} then click continue
    commands:
      - command: hover-text
        text: Password
        description: password input field label
        action: click
      - command: type
        text: ${TD_PASSWORD}
      - command: hover-text
        text: CONTINUE
        description: continue button below the password input
        action: click

This workflow:

• Runs on each push to the "main" branch & every day at midnight

• Defines variables that point to repository secrets

• Runs the test provided

name: TestDriver.ai

permissions:
  actions: read
  contents: read
  statuses: write
  pull-requests: write

on:
  push:
    branches: ["main"]
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: "0 0 * * *"

jobs:
  test:
    name: "TestDriver"
    runs-on: ubuntu-latest
    steps:
      - name: Run TestDriver.ai Action
        uses: testdriverai/action@main
        with:
          key: ${{ secrets.TESTDRIVER_API_KEY }}
          prompt: |
            1. /run testdriver/test.yml
          prerun: |
            cd $env:TEMP
            npm init -y
            npm install dashcam-chrome
            Start-Process "C:/Program Files/Google/Chrome/Application/chrome.exe" -ArgumentList "--start-maximized", "--load-extension=$(pwd)/node_modules/dashcam-chrome/build", "${{ env.WEBSITE_URL }}"
            exit
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          FORCE_COLOR: "3"
          WEBSITE_URL: "example.com"
          TD_USERNAME: ${{ secrets.TD_USERNAME }}
          TD_PASSWORD: ${{ secrets.TD_PASSWORD }}

Example Usage

command: hover-image
description: search icon in the webpage content
action: click

Example Usage

command: focus-application
name: Google Chrome

This workflow's prerun script downloads an installer, then installs the software before running the tests

This workflow:

• Runs on each push to the "main" branch & every day at midnight

• Downloads an installer from the provided URL

• Runs the installer and install the software

• Run the provided test

  • Functionalities like 'automatic test matrix population' can be added to this workflow

name: TestDriver.ai

permissions:
  actions: read
  contents: read
  statuses: write
  pull-requests: write
  
on:
  push:
    branches: ["main"]
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: "0 0 * * *"

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: testdriverai/action@main
        with:
          key: ${{ secrets.TESTDRIVER_API_KEY }}
          prompt: |
            1. /run testdriver/test.yml
          prerun: |
            # Check IPv6 addresses (optional)
            Get-NetIPAddress -AddressFamily IPv6

            # URL for the installer
            $installerUrl = "https://example.com/windows/ExampleInstaller.exe"
            # Location to save the installer
            $installerPath = "$env:USERPROFILE\Downloads\ExampleInstaller.exe"

            # Download the installer
            Write-Host "Downloading installer..."
            Invoke-WebRequest -Uri $installerUrl -OutFile $installerPath

            # Check if the download was successful
            if (Test-Path $installerPath) {
                Write-Host "Download successful. Running the installer..."
                Start-Process -FilePath $installerPath -ArgumentList '/silent' -Wait
                Start-Sleep -Seconds 10
            } else {
                Write-Host "Failed to download the 
                
                installer."
            }

        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          FORCE_COLOR: "3"

Example

version: 4.2.10
session: 67a3fdacd06c99b9c179b566
steps:
  - prompt: exec
    commands:
      - command: exec
        cli: pwd
        silent: false
        output: my_var
  - prompt: type ls
    commands:
      - command: type
        text: ${OUTPUT.my_var}

Example Output

❯ node index.js run testdriver/testdriver.yml
Spawning GUI...
Howdy! I'm TestDriver v4.2.13
Working on /Users/ianjennings/Development/testdriverai/testdriver/testdriver.yml

This is beta software!
Join our Discord for help
https://discord.com/invite/cWDFW8DzPm

running /Users/ianjennings/Development/testdriverai/testdriver/testdriver.yml...

exec
command='exec' cli='pwd' output='my_var'
/Users/ianjennings/Development/testdriverai


type ls
command='type' text='/Users/ianjennings/Development/testdriverai
'
/Users/ianjennings/Development/testdriverai

Weekly Email Reports

The TestDriver Performance agent will email you weekly with high level summaries of your software quality. Get alerted of new bugs and detect feature regressions.

Every email includes:

• an single performance score

• a high level summary of test failures

• test passing rates per test file

Dashboard

Also included is a high level summary of test passing rates and coverage by feature.

Performance

Measure test completion duration over time.

Example Usage

command: if
condition: the active window is "Google Chrome"
then:
  - command: hover-text
    text: Search Google or type a URL
    description: main google search
    action: click
  - command: type
    text: monster trucks
    description: search for monster trucks
else:
  - command: focus-application
    name: Google Chrome

Example Usage

command: press-keys
keys: [command, space]

Example Usage

command: remember
description: My dog's name
value: Roofus

Example Usage

command: run
file: path/to/another-script.yaml

This workflow creates a matrix of tests dynamically based off of the YAML files available in a directory (in this case, and most, the /testdriver directory)

This workflow:

• Runs on each push to the "main" branch & every day at midnight

• Dynamically gathers .yml files from the /testdriver directory to create a matrix of tests

• Runs the matrix of tests in parallel

  • Each test in the matrix will download the Arc browser installer

name: TestDriver.ai

permissions:
  actions: read
  contents: read
  statuses: write
  pull-requests: write

on:
  push:
    branches: ["main"]
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: "0 0 * * *" 

jobs:
  gather-test-files:
    name: Gather Test Files
    runs-on: ubuntu-latest
    outputs:
      test_files: ${{ steps.test_list.outputs.files }}
    steps:
      - name: Check out repository
        uses: actions/checkout@v2

      - name: Find all test files and extract filenames
        id: test_list
        run: |
          FILES=$(ls ./testdriver/*.yml)
          FILENAMES=$(basename -a $FILES)
          FILES_JSON=$(echo "$FILENAMES" | jq -R -s -c 'split("\n")[:-1]')
          echo "::set-output name=files::$FILES_JSON"

  test:
    needs: gather-test-files
    runs-on: ubuntu-latest
    strategy:
      matrix:
        test: ${{ fromJson(needs.gather-test-files.outputs.test_files) }}
      fail-fast: false
    name: ${{ matrix.test }}
    steps:
      - name: Check out repository
        uses: actions/checkout@v2

      - name: Display filename being tested
        run: |
          echo "Running job for file: ${{ matrix.test }}"
      
      - uses: testdriverai/action@main
        with:
          key: ${{ secrets.TESTDRIVER_API_KEY }}
          prompt: | 
            1. /run testdriver/${{ matrix.test }} 
          prerun: |
            cd $env:TEMP
            npm init -y
            npm install dashcam-chrome
            Start-Process "C:/Program Files/Google/Chrome/Application/chrome.exe" -ArgumentList "--start-maximized", "--load-extension=$(pwd)/node_modules/dashcam-chrome/build", "${{ env.WEBSITE_URL }}"
            exit
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          FORCE_COLOR: "3"
          WEBSITE_URL: "example.com"

TestDriver isn't like any test framework you've used before - it's more like your own QA employee with their own development environment.

1. Tell TestDriver what to do in natural language

2. TestDriver looks at the screen and uses mouse and keyboard emulation to accomplish the goal

Advantages

TestDriver then uses AI vision and hardware emulation to simulate real user on their own computer. This has three main advantages:

• Easier set up: No need to add test IDs or craft complex selectors

• Less Maintenance: Tests don't break when code changes

• More Power: TestDriver can test any application and control any OS setting

Just tell TestDriver what to do

Use our CLI to tell TestDriver what to do, like so:

> open google chrome
> navigate to airbnb.com
> search for destinations in austin tx
> click check in
> select august 8

Possibilities

As you can imagine, a specialized QA agent with it's own computer is extremely powerful. TestDriver can:

• Test any user flow on any website in any browser

• Clone, build, and test any desktop app

• Render multiple browser windows and popups like 3rd party auth

• Test <canvas>, <iframe>, and <video> tags with ease

• Use file selectors to upload files to the browser

• Resize the browser

• Test chrome extensions

• Test integrations between applications

The problem with current approach to end-to-end testing

End-to-end is commonly described as the most expensive and time-consuming test method. Right now we write end-to-end tests using complex selectors that are tightly coupled with the code.

You've probably seen selectors like this:

const e = await page.$('div[class="product-card"] >> text="Add to Cart" >> nth=2');

This tight coupling means developers need to spend time to understand the codebase and maintain the tests every time the code changes. And code is always changing!

End-to-end is about users, not code

In end-to-end testing the business priority is usability. All that really matters is that the user can accomplish the goal.

TestDriver uses human language to define test requirements. Then our simulated software tester figures out how to accomplish those goals.

These high level instructions are easier to create and maintain because they are loosely coupled from the codebase. We're describing a high level goal, not a low level interaction.

The tests will still continue to work even when the junior developer changes .product-card to .product.card or the designers change Add to Cart to Buy Now . The concepts remain the same so the AI will adapt.

How exactly does this work?

TestDriver's AI is a fine tuned model developed over the course of more than a year with the help of computer vision experts, custom research tooling, and a few million dollars in funding (thanks VCs).

TestDriver uses a combination of reinforcement learning and computer vision. The context from successful text executions inform future executions. Here's an example of the context our model considers when locating a text match:

You'll likely want TestDriver to log in to your app as a test user, but you wouldn't want to expose that password to the world.

Thankfully TestDriver provides a way to securely use secrets. Secrets will be masked from all test output, including Debugging Test Runs logs.

Replace the hardcoded secrets within your Test Steps

Open your test file (.yml) in a code editor and replace your secrets with ${TD_YOUR_SECRET} .

Here is an example of Test Stepswith the secret syntax.

version: 4.1.35
steps:
  - prompt: sign in with username and password
    commands:
      - command: focus-application
        name: Google Chrome
      - command: hover-text
        text: Email or phone
        description: email input field
        action: click
      - command: type
        text: ${TD_USERNAME}
      - command: hover-text
        text: Next
        description: next button after entering email
        action: click
      - command: hover-text
        text: Password
        description: password input field
        action: click
      - command: type
        text: ${TD_PASSWORD}

Add the secrets to your GitHub repository

First, configure the secrets within your GitHub repository.

Follow the guide here for detailed instructions on how to add a secrete to your GitHub repo or organization.

Supply the secrets to the GitHub Actions

When using the GitHub Actions to spawn tests, supply your TD_SECRET within the env: value provided to the GitHub action.

env:
    TD_USERNAME: ${{ secrets.TD_USERNAME }}
    TD_PASSWORD: ${{ secrets.TD_PASSWORD }}

permissions:
  actions: read
  contents: read
  statuses: write
  pull-requests: write
  
jobs:
  test:
    name: "TestDriver"
    runs-on: ubuntu-latest
    steps:
      - uses: testdriverai/action@main
        with:
          key: ${{ secrets.TESTDRIVER_API_KEY }}
          prompt: | 
            1. /run tests/signin.yml
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          FORCE_COLOR: "3"
          TD_USERNAME: ${{ secrets.TD_USERNAME }}
          TD_PASSWORD: ${{ secrets.TD_PASSWORD }}

Example Usage

Example Usage

Example Usage

These are a list of commands you can run when you see the > after running testdriverai . Click any of the subpages to learn more. We recommend reading them in this order:

Example Usage

Example Usage

Arguments

Example Usage

Example Usage

• testdriverai [file]

• testdriverai init

• testdriverai run [file]

Arguments

Example Usage

command: scroll-until-image
description: Submit at the bottom of the form
direction: down

Use the assert to command generate an assertion. This will take a screenshot and use it to identify some criteria that ensures the task was complete.

This will "assert" that there's no error message, just like a user would see. The generated command will look like this:

When TestDriver runs this test, it will look at the screen and verify that the value of expect is true. If it is not true, the test will fail and exit immediately.

Many asserts can slow down a test. Use async: true to speed things up.

Run the initcommand to trigger the testdriverai interactive setup.

This is the core testdriver command that will launch the testdriver agent.

Supply a file command to specify output location

This defaults to testdriver/testdriver.yml

Source

The TestDriver agent is open-source and available on NPM. You can browser the source and see all the data collected and how everything works.

API

The TestDriver agent does not contain any AI models within it. Instead, it uploads desktop context to our API which uses that context to make decisions about what actions to perform.

Our API makes use of OpenAI models behind the scenes. You can learn more about OpenAI and privacy in their privacy center.

Desktop Context Collected

During execution the TestDriver agent uploads the following information to our API

• User input prompts

• The active window and other windows that may be open (including application and window titles)

• System information

• The mouse position

• Screenshots of the desktop

With the exception of desktop screenshots, desktop context is persisted into our database.

Desktop Screenshots

TestDriver frequently takes screenshots of the desktop to provide our AI with decisions making context. You will not be prompted. Desktop screenshots are uploaded to our API for processing but are not persisted.

The TestDriver Agent will only take screenshots of the primary display. For complete privacy, we recommend running TestDriver within a virtual machine on your desktop.

TestDriver can not operate without visual context. Do not install TestDriver if you do not want to capture images of the desktop.

Active Window

Information about the open windows on the desktop is reported by the active-window module.

System Information

Information about the computer system running testdriver is reported by the systeminformation module.

User Prompts

The prompts you input to TestDriver are uploaded to our API and persisted in a database. We store this data to provide our AI with a history of context.

Additional Analytics

When running testdriver init you'll be asked if you'd like to share additional analytics. Sharing usage analytics is opt-in, this extra data will not be collected unless explicitly set in your environment.

If you would like to disable additional analytics, you can set TD_ANALYTICS within your environment.

TD_ANALYTICS=false

Rate Limiting and Other Restrictions

While the TestDriver Agent is free, we do reserve the right to rate limit or restrict usage by IP address for any reason.

/generatewill look at the display and generate lists of exploratory prompts that can be used to generate tests. Each exploratory test looks like a simple markdown file with a list inside.

Check out this test-search-function.mdfor example:

This command generates a YAML file with the history of executed commands and tasks.

To run a test you've previously created, use the /run command.

TestDriver will run the test plan back performing each command.

This command will exit the program upon execution. Any failures will be output and the program will exit with code 1.

If Testdriver doesn't do what you expect, you can easily remove newly generated commands with the /undo. You can undo as many times as you like.

For example given the following test:

- step: 
    - command: scroll-until-text
      text: Add to cart
- step: 
    - command: hover-text
      text: Add to cart
      action: click

Calling /undo will cause the last part to be undone and look like this:

- step: 
    - command: scroll-until-text
      text: Add to cart

Runs the specified file.

The command will return exit code 0if the test is successful and 1if a failure.

Tests that execute via our GitHub Action are recorded and reported via Dashcam (another application developed by TestDriver). You can find more information on the Dashcam docs.

Dashcam and TestDriver share the same API and web application back end. This web application includes the following privacy and security features:

The TestDriver action is open source. You can find the source below.

Ephemeral Virtual Machine Runners

TestDriver tests are executed on private virtual machines managed by Amazon EC2. These VMs are ephemeral meaning they only exist for the lifetime of the test execution. After the test completes, the VM is destroyed and the hard disk is wiped.

Secrets

We recommend storing any private information as a secret in your GitHub repository. Learn more about storing secrets here.

Any secrets supplied within Prerun Scripts or prompts will be transmitted over SSL to our API. Secrets supplied to agent prompts will be persisted (see Agent), but prerun scripts are not persisted.

A common workflow is to use Prerun Scripts to access a private staging website via basic auth. This will allow you to securely log into staging without persisting sensitive data on our servers.

Production

Testing production resources does not require any private information from your team. Simply provide the tests to TestDriver and point toward your publicly available endpoints. TestDriver does not need access to any private information.

Staging

Depending on your implementation, TestDriver may need secure information to test staging environments. See Secretsabove for more information on securely implementing tests on staging.

Development

TestDriver can clone feature branches and build code on it's virtual machines using a similar workflow to GitHub runners.

In order for TestDriver to test development branches of private codebases, it's necessary to supply a GitHub Token within the GitHub action. This personal access token is used to clone the codebase on the VM.

env:
  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

This token is transmitted over SSL and is not persisted. Learn more about managing the privacy of GitHub access tokens here.

If you're on mac, you might see an error related to "screen capture permissions" when getting started. This means that the application invoking the `testdriverai` command does not have permission to capture.

You must enable screen capture permissions for this application

This might be a little confusing at first. It's not testdriverai that needs the permissions, it's the terminal or IDE that's calling testdriverai that needs them. This is probably:

• Mac Terminal

• iTerm

• VS Code

• Or something similar

Enable Screen Capture Permissions for TestDriver

To enable screen capture permissions, do the following.

• Open System Settings (or System Preferences on older versions).

• Navigate to Privacy & Security > Screen Recording.

• Find and enable permissions for Terminal (or any terminal-based app like iTerm2, if you're using it).

• If the desired application is unavailable in the list, click on "+" symbol below, you will get a finder pop up, here find the app that you are currently using under Application section and click on open.

• You may need to restart the terminal for the changes to take effect.