TestDriver isn't like any test framework you've used before - it's more like your own QA employee with their own development environment.

1. Tell TestDriver what to do in natural language

2. TestDriver looks at the screen and uses mouse and keyboard emulation to accomplish the goal

Advantages

TestDriver then uses AI vision and hardware emulation to simulate real user on their own computer. This has three main advantages:

• Easier set up: No need to add test IDs or craft complex selectors

• Less Maintenance: Tests don't break when code changes

• More Power: TestDriver can test any application and control any OS setting

Just tell TestDriver what to do

Use our CLI to tell TestDriver what to do, like so:

> open google chrome
> navigate to airbnb.com
> search for destinations in austin tx
> click check in
> select august 8

Possibilities

As you can imagine, a specialized QA agent with it's own computer is extremely powerful. TestDriver can:

• Test any user flow on any website in any browser

• Clone, build, and test any desktop app

• Render multiple browser windows and popups like 3rd party auth

• Test <canvas>, <iframe>, and <video> tags with ease

• Use file selectors to upload files to the browser

• Resize the browser

• Test chrome extensions

• Test integrations between applications

The problem with current approach to end-to-end testing

End-to-end is commonly described as the most expensive and time-consuming test method. Right now we write end-to-end tests using complex selectors that are tightly coupled with the code.

You've probably seen selectors like this:

const e = await page.$('div[class="product-card"] >> text="Add to Cart" >> nth=2');

This tight coupling means developers need to spend time to understand the codebase and maintain the tests every time the code changes. And code is always changing!

End-to-end is about users, not code

In end-to-end testing the business priority is usability. All that really matters is that the user can accomplish the goal.

TestDriver uses human language to define test requirements. Then our simulated software tester figures out how to accomplish those goals.

These high level instructions are easier to create and maintain because they are loosely coupled from the codebase. We're describing a high level goal, not a low level interaction.

The tests will still continue to work even when the junior developer changes .product-card to .product.card or the designers change Add to Cart to Buy Now . The concepts remain the same so the AI will adapt.

How exactly does this work?

TestDriver's AI is a fine tuned model developed over the course of more than a year with the help of computer vision experts, custom research tooling, and a few million dollars in funding (thanks VCs).

TestDriver uses a combination of reinforcement learning and computer vision. The context from successful text executions inform future executions. Here's an example of the context our model considers when locating a text match:

With TestDriver’s AI-powered QA Agents, creating and deploying robust UI tests has never been faster or easier. Here's how to get up and running in no time.

Generate your first test suite

The fastest way to create your first set of tests is by running the demo. TestDriver will automatically log into your application, generate a dozen UI, and push them directly to your GitHub repository.

Follow the guide here to generate your first test suite.

Write your own tests with the Computer-Use agent

Getting started with TestDriver is simple. Just download the TestDriver Agent to enable test generation directly from your environment.

Need Help? Watch the Setup Video

Follow along with our step-by-step YouTube setup guide to see how easy it is to install the agent, connect your app, and start generating tests—no guesswork needed.

Set Up Your First TestDriver Project

To get started, install the testdriverai package globally using Yarn. This package enables you to generate tests effortlessly using natural language commands.

yarn add global testdriverai@beta

Next, create a new directory for your test project and initialize it with testdriverai init.

mkdir testdriver-first-project
cd testdriver-first-project
testdriverai init

This will launch the interactive setup process:

Beginning setup...
✔ Use TestDriver Runners? (Recommended) … yes
? API KEY (from https://app.testdriver.ai/team) ›

TestDriver Runners are the fastest way to get started, but you'll need an TestDriver Pro plan. See Getting an API Key.

You can use your own computer as a runner for free. See Local Agent Setup

How To Get an API Key

Paste your API key into the prompt when asked, then follow the remaining steps in the setup to complete your project configuration.

Build tests with the Computer-Use Agent

Now comes the fun part—building your tests! To get started, trigger the Computer-Use Agent by running:

testdriverai

This launches TestDriver in interactive mode. The agent will begin listening for commands and automatically generate test steps based on your input. All generated commands will be saved to testdriver/testdriver.yaml

Give Instructions in Plain English

Now, just tell TestDriver what you want it to do. For our example, try click new task.

> click new task

What Happens Next

TestDriver will analyze your screen, interpret your instruction, and generate a full test script for you. It can see your UI, control the mouse and keyboard, and automate complex workflows—all through simple commands.

See the yml TestDriver generated? That's our own schema. You can learn more about it in the Test Steps.

> click new task

    thinking...

    To accomplish the goal of clicking "New Task" on the
    webpage, we need to focus on the browser window and then
    click the "New Task" button.

    Here is the YAML code to achieve this:

    commands:
      - command: focus-application
        name: Google Chrome
      - command: hover-text
        text: New Task
        description: button to create a new task
        action: click

command='focus-application' name='Google Chrome'
command='hover-text' text='New Task' description='button to create a new task' action='click'

    thinking...

    Based on the provided coordinates and text, the element
    that matches "New Task" with the description "button to
    create a new task" is at coordinates (864, 328).

    Here is the YAML code to click on that element:

    commands:
      - command: click
        x: 864
        y: 328
        click: single
        button: left

command='click' x=864 y=328 click='single' button='left'

Keep going!

You can continue instructing TestDriver with natural commands. It will visually inspect your app and generate the next steps for you. Try:

> enter a task title
> enter a task summary
> click create task
> delete the task

Test It Locallay Before Deployment

Before pushing to GitHub, it’s a good idea to validate your test locally. After you’ve saved the test using /save, run:

testdriverai run testdriver/testdriver.yml

Deploy tests to GitHub

The fastest way to get started with TestDriver

TestDriver will crawl your app and generate tests. It interacts with each element and continues to come up with new ideas.

TestDriver can generate 1,000s of tests automatically. Tests are opened as Pull Requests. All you need ot do is review the created tests and merge them into your test suite!

Demo

The easiest way to generate tests for your app is to use the form below. This will generate up to 10 tests for you in a new GitHub repository.

Reviewing Pull Requests

TestDriver will generate tests for your app. Each pull request description will contain:

1. The Interactive Commands used to generate the test (description)

2. The test result

The pull request will also contain a Test Steps. Here is an example of a generated test

Debugging Pull Requests

Click on the Dashcam preview to see a video of the AI completing the test as well as the AI logs, console logs, and network requests. For more on reviewing test results, see #Debugging Test Runs.

Making Changes

TestDriver may try multiple times to accomplish a goal.

Not every test will complete the objective entirely, but the steps that do complete successfully make great regression tests and it usually only takes a little modification to make a successful test!

We recommend:

1. Deleting the test steps that did not complete successfully.

2. Modifying the test steps if required. See Test Steps for details on each TestDriver command

Deploying Your Test (Requires Upgrade)

This repository is set up to automatically run any merged regression tests! You'll need an API key to do so, and you can only get one by upgrading your account to Pro or higher.

Where to go from here?

• Read the GitHub Actions guide for more information on integrating with GitHub.

• Scan the Action Output to learn how to send emails and Slack notifications on test failures

• Check out the Interactive Commands for details on the high level commands TestDriver uses to generate tests or learn about regression tests in Test Steps

• Follow the Local Agent Setup guide to create and debug tests on your local machine

Deploying your TestDriver tests to CI/CD requires an API key!

2. Upgrade your account to TestDriver Pro (Every upgrade comes with $100 in free usage credits!)

3. Copy the API key from your developer portal

Running in CI/CD

CI/CD pipelines enhances software quality and release velocity. Adding TestDriver to your CI/CD, you can catch defects early, prevent regressions, and gain rapid feedback on code changes.

The TestDriver GitHub action will execute tests when you mention it in a comment, on a schedule, or anywhere else in your CI/CD workflow.

Continuous Integration

TestDriver can be integrated with GitHub Actions to automatically run tests whenever code is pushed to a repository. This ensures that new changes do not break existing functionality.

Automated Code Reviews

TestDriver can be configured to run specific test suites when a pull request is created. This ensures that any code changes meet the project's quality standards before being merged.

Increased Test Coverage

TestDriver can automatically generate and run test cases across multiple environments, providing detailed feedback on test coverage and potential issues.

Improved Developer Experience

Developers get immediate feedback on the success or failure of their changes within the pull request itself, reducing the time spent on manual code reviews.

Application Support

TestDriver opeates a full desktop environment, so it can run any application.

Testing Features

TestDriver is AI first.

Test Coverage

Testdriver has more coverage than selector based frameworks.

Debugging Features

Web Browser Support

TestDriver is browser agnostic and supports any verson of any browser.

Operating System Support

TestDriver currently supports Mac and Windows!

Installing the GitHub action

Get your API Key

In order to execute your TestDriver actions on our VMs you'll need to add your API key as a GitHub secret. If you don't see an API key, you'll need ot upgrade your account.

Create your workflow

Now it's time to create your first TestDriver workflow.

In .github/workflows/testdriver.yml add the following code.

Output

How it works

1. The GitHub action is triggered via the conditions supplied via on

2. The key value is used to authenticate you

3. An ephemeral virtual machine is spawn on our infrastructure

4. The code from the current branch will be cloned on to the VM

7. prompt is parsed as a markdown list.

8. Each list item from promp is fed into TestDriver one by one

9. TestDriver summarizes the test and sets it's exit code depending on it's pass or failed state

11. The VM is destroyed and all data is wiped

Deploy the test

Save the file, make a new branch, push to your repository, and create a new pull request.

This will trigger a new TestDriver execution.

Prerun scripts are Bash commands executed on a TestDriver VM before each test within a CI/CD pipeline. Their primary purpose is to establish the state of a machine. This ensure it is consistent before every test execution.

You can configure prerun script to install necessary dependencies, build your application, set specific configurations, and more.

This crucial step helps to speed up the setup of an environment, prepare for a test suite to run, and prevent test failures due to environment inconsistencies and promote reproducible builds, ultimately enhancing the overall test suite's effectiveness.

Example

This is an example of how to download Arc Browser and use it instead of Chrome.

# permissions and other setup here

jobs:
  test:
    name: "TestDriver"
    runs-on: ubuntu-latest
    steps:    
    # Download an exe for this test
    - uses: testdriverai/action@main
      with:
        prerun: |
          Get-NetIPAddress -AddressFamily IPv6
          # URL for the Arc browser installer
          $installerUrl = "https://releases.arc.net/windows/ArcInstaller.exe"
          # Location to save the installer
          $installerPath = "$env:USERPROFILE\Downloads\ArcInstaller.exe"
          # Download the Arc browser installer
          Write-Host "Downloading Arc browser installer..."
          Invoke-WebRequest -Uri $installerUrl -OutFile $installerPath
          # Check if the download was successful
          if (Test-Path $installerPath) {
              Write-Host "Download successful. Running the installer..."
              Start-Process -FilePath $installerPath -ArgumentList '/silent' -Wait
              Start-Sleep -Seconds 10
          } else {
              Write-Host "Failed to download the Arc browser installer."
          }

🔧 Product Capabilities

• What is TestDriver? TestDriver is an AI-powered testing platform that simulates user interactions to automate end-to-end testing for web, desktop, and mobile applications.

• How does TestDriver work? It interprets high-level prompts, interacts with interfaces like a user would, and verifies expected outcomes using assertions and visual validation.

• What platforms does TestDriver support? TestDriver supports Windows, Mac, Linux desktop apps, web browsers, and mobile interfaces (via emulator or device farm).

• Can it be used for exploratory testing? Yes. TestDriver can autonomously navigate the application to discover potential issues or generate new test cases.

• Can it test desktop applications? Yes. It supports testing native desktop applications by simulating mouse and keyboard input and identifying UI elements.

• Can it test mobile apps? Yes, via mobile emulators or integration with device farms.

🤖 Test Creation and Generation

• Can TestDriver generate tests automatically? Yes, it explores the app and creates test cases based on UI flows and user interactions.

• Can I create tests from natural language prompts? Yes. You can write high-level instructions in plain language, and TestDriver will interpret and build tests from them.

• Can it generate tests from user stories or documentation? Yes. It can use minimal descriptions to produce complete test cases.

• Can it turn recorded user sessions into tests? Yes, in supported environments, TestDriver can generate test steps from interaction logs or screen recordings.

🛠️ Test Maintenance and Resilience

• What happens when the UI changes? TestDriver adapts using AI—if a button or label changes, it can often infer the correct action without breaking.

• Do I need to rewrite tests often? No. TestDriver reduces maintenance by handling common UI changes automatically.

• How does it handle flaky tests? It retries failed actions, assigns confidence scores, and logs inconsistencies so you can investigate root causes.

• How are tests updated over time? You can regenerate them using updated prompts or manually edit the test specs.

🚨 Failures, Debugging, and Feedback

• How does TestDriver report test failures? It provides detailed logs, screenshots, console output, and visual diffs.

• What happens when a test fails? It stops execution, flags the failing step, and provides context for debugging.

• Can I view why a test failed? Yes. You can view step-by-step logs, network traffic, DOM state, and video playback of the test run.

• Can it automatically retry failed actions? Yes. You can configure retry behavior for individual steps or full tests.

🚀 Performance and Parallelism

• Can I run tests in parallel? Yes. TestDriver supports parallel execution using multiple VMs or containers.

• Can I track performance metrics during testing? Yes. It can log CPU, memory, load times, and frame rates to help catch performance regressions.

🔍 Advanced Testing Features

• Can it validate non-deterministic output? Yes. It uses AI assertions to verify outcomes even when outputs vary (e.g., generated text or dynamic UIs).

• Can it test workflows with variable inputs? Yes. It supports data-driven tests using parameterized inputs.

• Can it test file uploads and downloads? Yes. TestDriver can interact with file pickers and validate uploaded/downloaded content.

• Can it generate tests for PDFs or document output? Yes. It can open and verify generated files for expected text or formatting.

• Can I trigger tests based on pull requests or merges? Yes. You can integrate TestDriver with your CI to trigger runs via GitHub Actions or other CI/CD tools.

🧩 Integration and Setup

• Does it integrate with CI/CD tools? Yes. TestDriver integrates with pipelines like GitHub Actions, GitLab CI, and CircleCI.

• Can I integrate TestDriver with Jira, Slack, etc.? Yes. You can receive alerts and sync test results with third-party tools via API/webhooks.

• Does it support cloud and local environments? Yes. You can run tests locally or in the cloud using ephemeral VMs for clean state testing.

• Does it work with existing test frameworks? It can complement or convert some existing test cases into its format, though full conversion depends on compatibility.

📊 Test Coverage and Effectiveness

• How does TestDriver measure test coverage? It tracks UI paths, element interaction frequency, and application state changes to infer coverage.

• Can it suggest missing test scenarios? Yes. Based on interaction patterns and user behavior, it can propose additional test cases.

• Can it analyze test stability over time? Yes. You can view trends in pass/fail rates and test execution consistency.

🔒 Security and Compliance

• Is it safe to test sensitive data? Yes. TestDriver supports variable obfuscation, secure containers, and test data isolation.

• Can I avoid using production data in tests? Yes. You can configure mock data, sanitize logs, and use test-specific environments.

🧠 AI Behavior and Prompting

• How does the AI understand what to test? It uses language models to interpret your goals, element names, and interface cues to perform tasks.

• Can I adjust how the AI interprets my prompt? Yes. You can rewrite prompts, add constraints, or review and tweak auto-generated steps.

• Can I see what the AI is doing behind the scenes? Yes. You can inspect the resolved steps, see element matches, and adjust test flows before execution.

📦 Use Cases and Scenarios

• Can I use TestDriver to test new features? Yes. It’s great for validating changes, ensuring no regressions, and verifying rollout configurations.

• Can it test seasonal or time-based behaviors? Yes. You can schedule tests or run them under specific date/time settings to verify time-sensitive logic.

To configure the operating system, simply supply the os param to the GitHub action. The current available operating systems are windows and mac.

Prerun files

Note that prerun files are executed on these machines themselves. You must use powershell on Windows and bash on mac.

Testing multiple browsers and operating systems

name: Test action

permissions:
  actions: read
  contents: read
  statuses: write
  pull-requests: write

on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled]

jobs:
  test-action:
    name: Test action
    runs-on: ubuntu-latest
    strategy:
      matrix:
        os: [windows, mac]
        browser: [chrome, firefox]
    steps:
      - name: Set up Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '16'

      - name: Install Dashcam Chrome
        run: |
          npm init -y
          npm install dashcam-chrome

      - uses: replayableio/testdriver-action@main
        with:
          prompt: |
            1. open youtube
            2. find a cat video
            3. quit the browser
            4. /summarize
          os: ${{ matrix.os }}
          prerun: |
            if [ "${{ matrix.browser }}" == "chrome" ]; then
              if [ "${{ matrix.os }}" == "windows" ]; then
                # Install Google Chrome on Windows
                $ProgressPreference = 'SilentlyContinue'
                Invoke-WebRequest -Uri "https://dl.google.com/chrome/install/latest/chrome_installer.exe" -OutFile "$env:TEMP\chrome_installer.exe"
                Start-Process -FilePath "$env:TEMP\chrome_installer.exe" -ArgumentList "/silent", "/install" -Wait
              else
                # Install Google Chrome on macOS
                brew install --cask google-chrome
              fi
            else
              if [ "${{ matrix.os }}" == "windows" ]; then
                # Install Firefox on Windows
                $ProgressPreference = 'SilentlyContinue'
                Invoke-WebRequest -Uri "https://download.mozilla.org/?product=firefox-latest&os=win64&lang=en-US" -OutFile "$env:TEMP\firefox_installer.exe"
                Start-Process -FilePath "$env:TEMP\firefox_installer.exe" -ArgumentList "/S" -Wait
              else
                # Install Firefox on macOS
                brew install --cask firefox
              fi
            fi
            if [ "${{ matrix.browser }}" == "chrome" ]; then
              if [ "${{ matrix.os }}" == "windows" ]; then
                Start-Process "C:/Program Files/Google/Chrome/Application/chrome.exe" -ArgumentList "--start-maximized","--load-extension=$(pwd)/node_modules/dashcam-chrome/build" &
              else
                /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --start-maximized --load-extension=$(pwd)/node_modules/dashcam-chrome/build &
              fi
            else
              if [ "${{ matrix.os }}" == "windows" ]; then
                Start-Process "C:/Program Files/Mozilla Firefox/firefox.exe" -ArgumentList "--start-maximized" &
              else
                /Applications/Firefox.app/Contents/MacOS/firefox --start-maximized &
              fi
            fi
          key: ${{ secrets.TESTDRIVER_API_KEY }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          FORCE_COLOR: "3"

You can use your own computer as a TestDriver runner. TestDriver will 'alt-tab' between the terminal and your computer.

Just tell TestDriver what to test on your local machine and TestDriver will look at the screen, generate some commands to control the computer, and execute them.

Extra Setup

Running tests on your own machine requires a bit of extra setup. You'll need to install some dependencies and give the terminal permission to see your screen and file system access.

NodeJS

You'll need NodeJS version 20 to get started. Windows users will need a couple extra tools.

Install TestDriver via NPM

npm install testdriverai -g

Set up the project

Run testdriverai init in a new folder.

testdriverai init

This will walk you through setting up a local project.

Set up your test environment

Before we get started, let's set up your machine to collaborate with TestDriver.

Display

TestDriver isn't like any framework you've used before. TestDriver makes decisions based on what it can see on your display!

For now, set up your environment with a browser window and your terminal side by side like so:

When you enter commands into TestDriver, the current terminal window will minimize and the focus-window command will bring Chrome or other applications to the foreground.

Application State

The application you want to test should be visible before you run the testdriverai command.

For our example, make a new incognito window in Chrome and load our test webpage:

https://testdriverai.github.io/example-react-todo/

You'll likely want TestDriver to log in to your app as a test user, but you wouldn't want to expose that password to the world.

Open your test file (.yml) in a code editor and replace your secrets with ${TD_YOUR_SECRET} .

Add the secrets to your GitHub repository

First, configure the secrets within your GitHub repository.

Follow the guide here for detailed instructions on how to add a secrete to your GitHub repo or organization.

The TestDriver action outputs the following variables. You can chain mlutliple actions together to post TestDriver results as comments, send an email on failure, or upload them to 3rd party test reporting software.

Example

Here's an example of creating a comment on the PR after every execution.

Local Runners

Hosted Linux Runners

Hosted Windows Runners

If you're building a desktop app, need more coverage, more power, or want to test more complex flows, we recommend using our hosted Windows Runners.

While TestDriver is incredibly smart, using AI matching methods all the time can be slow. Nobody wants to block developers from merging while waiting for tests to run!

Here are some tips for improving TestDriver performance.

Use Parallel Testing

Covered in the previous section, use runand Parallel Testing to split your actions into multiple files and run them.

Use ai matching method

The most common actions like hover-text, wait-for-text , and scroll-until-text use an optimized matching algorithm.

This algorithm uses text similarity to quickly compute the most similar text to what appears in your yml. This is about 40% faster than the ai method!

Usse `async` Asserts

The assert method has property async: true which allows you to create non blocking test assertions costing you almost no time!

Test Generation Parallel Testing Importing Tests Desktop Apps Secure Log In

Dashcam captures the video of the desktop test execution, the browser logs, network requests, prerun script output, and of course the TestDriver test execution. You can play it all back in the TestDriver dashboard!

Dashcam dashes are transmitted over HTTS, encrypted at test, and have "google style" sharing permissions. Sensitive data in logs is masked.

When a test finishes, a Dashcam clip (a dash for short) is into GitHub automatically. This enables developers to see a test artifact and collaborate directly from within GitHub.

This workflow's prerun script downloads an installer, then installs the software before running the tests

This workflow:

• Runs on each push to the "main" branch & every day at midnight

• Downloads an installer from the provided URL

• Runs the installer and install the software

• Run the provided test

  • Functionalities like 'automatic test matrix population' can be added to this workflow

Weekly Email Reports

The TestDriver Performance agent will email you weekly with high level summaries of your software quality. Get alerted of new bugs and detect feature regressions.

Every email includes:

• an single performance score

• a high level summary of test failures

• test passing rates per test file

Dashboard

Also included is a high level summary of test passing rates and coverage by feature.

Performance

Measure test completion duration over time.

This workflow defines TD_USERNAME and TD_PASSWORD in env: to existing repository secrets of the same name. These TD_USERNAME and TD_PASSWORD can then be used in any YAML passed into the workflow. Secrets must begin with TD_ in order for Testdriver to be able to see them.

Example YAML steps using secrets could look like:

This workflow:

• Runs on each push to the "main" branch & every day at midnight

• Defines variables that point to repository secrets

• Runs the test provided

TestDriver will worry about generating and maintaining tests for the most part. However, if you'd like to edit tests or gain a better understanding of what's going on you can find all of the commands in this secion.

As for YML format, here is an example of a valid yml file:

version: 4.0.0
steps:
  - prompt: enter fiber.google.com in url
    commands:
      - command: focus-application
        name: Google Chrome
      - command: hover-text
        text: Search Google or type a URL
        description: main google search
        action: click
      - command: type
        text: fiber.google.com
      - command: press-keys
        keys:
          - enter
  - prompt: enter a fake address and check availability
    commands:
      - command: focus-application
        name: Google Chrome
      - command: hover-text
        text: Enter your address
        description: address input field
        action: click
      - command: type
        text: 123 Fake Street
      - command: hover-text
        text: ZIP
        description: ZIP code input field
        action: click
      - command: type
        text: 12345
      - command: hover-text
        text: Check availability
        description: check availability button
        action: click
  - prompt: assert a familiy appears on screen
    commands:
      - command: focus-application
        name: Google Chrome
      - command: assert
        expect: a family appears on screen

Example

version: 4.2.10
session: 67a3fdacd06c99b9c179b566
steps:
  - prompt: exec
    commands:
      - command: exec
        cli: pwd
        silent: false
        output: my_var
  - prompt: type ls
    commands:
      - command: type
        text: ${OUTPUT.my_var}

Example Output

❯ node index.js run testdriver/testdriver.yml
Spawning GUI...
Howdy! I'm TestDriver v4.2.13
Working on /Users/ianjennings/Development/testdriverai/testdriver/testdriver.yml

This is beta software!
Join our Discord for help
https://discord.com/invite/cWDFW8DzPm

running /Users/ianjennings/Development/testdriverai/testdriver/testdriver.yml...

exec
command='exec' cli='pwd' output='my_var'
/Users/ianjennings/Development/testdriverai


type ls
command='type' text='/Users/ianjennings/Development/testdriverai
'
/Users/ianjennings/Development/testdriverai

Executes tasks based on user input using natural language processing. This command is invoked when the user input does not start with a / command. The system interprets the input and attempts to carry out the task specified by the user.

Example Usage

Prompting Tips

The agent is selecting the wrong thing!

This is the most common issues encountered with our agent, here are some possible reasons.

You're asking the AI to interact with elements it can not see

TestDriver uses the context from your prompt and the computer screen to make a decision of what commands to run. You should only prompt the AI to interact with elements it can currently see.

Incorrect Prompt

A common example of this is interacting with a dropdown. We often see users prompt the agent to interact with a dropdown and choose a state.

Recommended prompts

Instead, simply treat these as two separate prompts. This allows the UI to render and gives the AI the opportunity to parse the new screen data.

You're asking the AI to click on elements it does not understand

The TestDriver agent relies on visual understanding, not functional. Like any user, the AI does not understand what the function of a button will be. It can only guess.

Incorrect Prompt

Correct Prompts

Describing Images Properly

If you're uncertain of how to describe an icon, simply ask ChatGPT-4o what it would call it, and use that as your input.

The AI can not find small images

Small, isolated images smaller than 15x15px appear like "noise" to the AI and may not be clickable. However, you can use the match-imagecommand to select these using manually made screenshots.

No matter what I do, TestDriver will not select my element.

The AI has trouble selecting some specific elements, like empty gray boxes, some substrings, or conditions where there is a lot of similar text close together.

Example Usage

command: focus-application
name: Google Chrome

We know our customers need QA coverage yesterday. While other QA services offer coverage delivered months later for hundreds of thousands of dollars, we'll set you for free during our 30-day trial!

Pricing

We think you're going to love TestDriver, which is why for a limited time our team will build 30 custom tests for you as a part of our risk-free, 30 day trail.

Promotion Details

• Test any publicly available desktop app, chrome extension, mobile app, or website

• Choose from 250 AI generated tests within 7 days.

• Run tests 3 - 5 times every day.

• AI Quality reports delivered to your email.

• AI fixes and maintains tests.

• Integrates with GitHub.

How does it work?

4. Our support will work with you to create tests for any features TestDriver might have missed.

5. Our support team will help train your engineers to create additional tests

Onboarding Process

Contract Details

• Get 30 custom tests free during a 30-day trial when subscribing to $995/m "30x30" plan

• The tests are yours! You are free to modify, duplicate, or distribute tests however you wish

• Payment method is required to begin trial

• Contract renews monthly

• Cancel any time, for any reason

• Credits do not "roll over." Unused credits expire every month

Rather than execute your tests sequentially, you can make use of the run command to share common setup and teardown test plans.

The, simply parallelize your test executions by calling the TestDriver action multiple time as a part of two different jobs.

name: TestDriver.ai

permissions:
  actions: read
  contents: read
  statuses: write
  pull-requests: write

on:
  pull_request: # run on every PR event
  schedule:
    - cron: '0 * * * *' # run every hour
  push:
    branches:
      - main # run on merge to the main branch
  workflow_dispatch:

jobs:
  test1:
    name: "TestDriver Test 1"
    runs-on: ubuntu-latest
    steps:
      - uses: dashcamio/testdriver@main
        version: v4.0.0
        key: ${{secrets.TESTDRIVER_API_KEY}}
        with:
          prompt: |
            1. /run /Users/ec2-user/actions-runner/_work/testdriver/testdriver/.testdriver/test-1.yml
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          FORCE_COLOR: "3"

  test2:
    name: "TestDriver Test 2"
    runs-on: ubuntu-latest
    steps:
      - uses: dashcamio/testdriver@main
        version: v4.0.0
        key: ${{secrets.TESTDRIVER_API_KEY}}
        with:
          prompt: |
            1. /run /Users/ec2-user/actions-runner/_work/testdriver/testdriver/.testdriver/test-2.yml
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          FORCE_COLOR: "3"

Here's an example of testing a matrix of files.

name: TestDriver.ai / Run / Regressions

on:
  push:
    branches: ["main"]
  pull_request:
  workflow_dispatch:

permissions:
  actions: read
  contents: read
  statuses: write
  pull-requests: write

jobs:
  gather-test-files:
    name: Setup Test Matrix (./testdriver/*.yml)
    runs-on: ubuntu-latest
    outputs:
      test_files: ${{ steps.test_list.outputs.files }}
    steps:
      - name: Check out repository
        uses: actions/checkout@v2
        with:
          ref: ${{ github.event.ref }}
      - name: Find all test files and extract filenames
        id: test_list
        run: |
          FILES=$(ls ./testdriver/*.yml)
          FILENAMES=$(basename -a $FILES)
          FILES_JSON=$(echo "$FILENAMES" | jq -R -s -c 'split("\n")[:-1]')
          echo "::set-output name=files::$FILES_JSON"

  test:
    needs: gather-test-files
    runs-on: ubuntu-latest
    strategy:
      matrix:
        test: ${{ fromJson(needs.gather-test-files.outputs.test_files) }}
      fail-fast: false
    name: ${{ matrix.test }}
    steps:
      - name: Check out repository
        uses: actions/checkout@v2
        with:
          ref: ${{ github.event.ref }}

      - name: Display filename being tested
        run: |
          echo "Running job for file: ${{ matrix.test }}"
      
      - uses: testdriverai/action@main
        with:
          key: ${{ secrets.TESTDRIVER_API_KEY }}
          prompt: 1. /run testdriver/${{ matrix.test }} 
          prerun: |
            cd $env:TEMP
            npm init -y
            npm install dashcam-chrome
            Start-Process "C:/Program Files/Google/Chrome/Application/chrome.exe" -ArgumentList "--start-maximized", "--load-extension=$(pwd)/node_modules/dashcam-chrome/build", "${{ env.WEBSITE_URL }}"
            exit
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          FORCE_COLOR: "3"
          

This workflow creates a matrix of tests dynamically based off of the YAML files available in a directory (in this case, and most, the /testdriver directory)

This workflow:

• Runs on each push to the "main" branch & every day at midnight

• Dynamically gathers .yml files from the /testdriver directory to create a matrix of tests

• Runs the matrix of tests in parallel

  • Each test in the matrix will download the Arc browser installer

name: TestDriver.ai

permissions:
  actions: read
  contents: read
  statuses: write
  pull-requests: write

on:
  push:
    branches: ["main"]
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: "0 0 * * *" 

jobs:
  gather-test-files:
    name: Gather Test Files
    runs-on: ubuntu-latest
    outputs:
      test_files: ${{ steps.test_list.outputs.files }}
    steps:
      - name: Check out repository
        uses: actions/checkout@v2

      - name: Find all test files and extract filenames
        id: test_list
        run: |
          FILES=$(ls ./testdriver/*.yml)
          FILENAMES=$(basename -a $FILES)
          FILES_JSON=$(echo "$FILENAMES" | jq -R -s -c 'split("\n")[:-1]')
          echo "::set-output name=files::$FILES_JSON"

  test:
    needs: gather-test-files
    runs-on: ubuntu-latest
    strategy:
      matrix:
        test: ${{ fromJson(needs.gather-test-files.outputs.test_files) }}
      fail-fast: false
    name: ${{ matrix.test }}
    steps:
      - name: Check out repository
        uses: actions/checkout@v2

      - name: Display filename being tested
        run: |
          echo "Running job for file: ${{ matrix.test }}"
      
      - uses: testdriverai/action@main
        with:
          key: ${{ secrets.TESTDRIVER_API_KEY }}
          prompt: | 
            1. /run testdriver/${{ matrix.test }} 
          prerun: |
            cd $env:TEMP
            npm init -y
            npm install dashcam-chrome
            Start-Process "C:/Program Files/Google/Chrome/Application/chrome.exe" -ArgumentList "--start-maximized", "--load-extension=$(pwd)/node_modules/dashcam-chrome/build", "${{ env.WEBSITE_URL }}"
            exit
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          FORCE_COLOR: "3"
          WEBSITE_URL: "example.com"

Example Usage

command: run
file: path/to/another-script.yaml

Example Usage

command: hover-image
description: search icon in the webpage content
action: click

Example Usage

command: assert
expect: the video is playing

This command is useful for interacting with elements that the AI has trouble locating. The testdriverai package will take a screenshot of the desktop and search for the location of the image within the screenshot.

Screenshot should be stored in testdriver/screenshots/(mac/linux/windows)/PATH.png . TestDriver will dynamically resolve images based on the current platform.

The screenshot template matching logic looks for the most similar image within the screenshot and not exact matches. If the match is not above ~80%, it will search additional scales. Otherwise it fails

To create screenshots from remote tests, download the video of the test and open it "full" or "actual" size within your computer. Then use a screenshot tool like Cleanshot X to create a screenshot of the target element. Do the best you can to center the clickable element within the screenshot.

Example Usage

command: match-image
relativePath: button.png
action: click

Example Usage

command: press-keys
keys: [command, space]

Example Usage

command: type
text: Hello World

Example Usage

command: if
condition: the active window is "Google Chrome"
then:
  - command: hover-text
    text: Search Google or type a URL
    description: main google search
    action: click
  - command: type
    text: monster trucks
    description: search for monster trucks
else:
  - command: focus-application
    name: Google Chrome

Example Usage

command: remember
description: My dog's name
value: Roofus

Example Usage

command: hover-text
text: Sign Up
description: link in the header
action: click

These are a list of commands you can run when you see the > after running testdriverai . Click any of the subpages to learn more. We recommend reading them in this order:

1. Prompting

2. /assert

3. /undo

4. /save

5. /run

6. /generate

This workflow uses parallel testing to test several URLs in a short amount of time

This workflow:

• Runs on each push to the "main" branch & every day at midnight

• Defines a matrix of tests to be ran on individual, non-sequential, ephemeral VMs

• Runs the matrix of tests

  • Each test in the matrix will have Dashcam installed, and Chrome opened to the respective URL defined in the URL value in the matrix

name: TestDriver.ai

permissions:
  actions: read
  contents: read
  statuses: write
  pull-requests: write

on:
  push:
    branches: ["main"]
  pull_request:
  workflow_dispatch:
  schedule:
    - cron: "0 0 * * *"

jobs:
  testdriver_matrix:
    name: "TestDriver Matrix"
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        test:
          - name: "Functions testdriver"
            file: "testdriver/functions.yml"
            url: "https://example.com"
          - name: "Chat testdriver"
            file: "testdriver/chat_test.yml"
            url: "https://www.exmaple.com/demos/chat/"
          - name: "P2P dynamic payment testdriver"
            file: "testdriver/p2p_dynamic_payment.yml"
            url: "https://www.exmaple.com/demos/fintech/"
          - name: "Dynamic matchmaking testdriver"
            file: "testdriver/skill_based_matchmaking.yml"
            url: "https://www.example.com/demos/skill-based-matchmaking-dashboard/"
          - name: "Support search testdriver"
            file: "testdriver/support_search.yml"
            url: "https://support.example.com/hc/en-us"
          - name: "Ask AI testdriver"
            file: "testdriver/ask_ai.yml"
            url: "https://www.example.com"
          - name: "Edit settings testdriver"
            file: "testdriver/edit_account_settings.yml"
            url: "https://www.example.com"
          - name: "Debug console testdriver"
            file: "testdriver/debug_console.yml"
            url: "https://www.example.com"
    steps:
      - uses: testdriverai/action@main
        with:
          key: ${{ secrets.TESTDRIVER_API_KEY }}
          prompt: |
            1. /run ${{ matrix.test.file }}
          prerun: |
            cd $env:TEMP
            npm init -y
            npm install dashcam-chrome
            Start-Process "C:/Program Files/Google/Chrome/Application/chrome.exe" -ArgumentList "--start-maximized", "--load-extension=$(pwd)/node_modules/dashcam-chrome/build", "${{ matrix.test.url }}"
            exit
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          FORCE_COLOR: "3"

Arguments

Example Usage

command: scroll-until-image
description: Submit at the bottom of the form
direction: down

Example Usage

command: scroll
direction: down

/generatewill look at the display and generate lists of exploratory prompts that can be used to generate tests. Each exploratory test looks like a simple markdown file with a list inside.

Check out this test-search-function.mdfor example:

1. Click on the search icon.
2. Type "real-time chat" into the search bar.
3. Assert that search results are relevant and displayed.

Arguments

Example Usage

command: wait
timeout: 5000

• testdriverai [file]

• testdriverai init

• testdriverai run [file]

If Testdriver doesn't do what you expect, you can easily remove newly generated commands with the /undo. You can undo as many times as you like.

For example given the following test:

Calling /undo will cause the last part to be undone and look like this:

Run the initcommand to trigger the testdriverai interactive setup.

testdriverai init

TestDriver will walk you through .envcustomization and clone sample workflow files to deploy tests. See GitHub Actions.

Spawning GUI...
Howdy! I'm TestDriver v4.2.7
Working on /Users/ianjennings/demo-setup/testdriver/testdriver.yml

This is beta software!
Join our Discord for help
https://discord.com/invite/cWDFW8DzPm

Warning! TestDriver sends screenshots of the desktop to our API.
https://docs.testdriver.ai/security-and-privacy/agent

Welcome to the Testdriver Setup!

This is a preview of the Testdriver.ai
Please report any issues in our Discord server:
https://discord.com/invite/cWDFW8DzPm

Beginning setup...

✔ Enable desktop notifications? … yes
✔ Minimize terminal app? … yes
✔ Enable text to speech narration? … yes
✔ Send anonymous analytics? … yes
✔ Where should we append these values? … .env

Writing .env...

Downloading latest workflow files...

Writing .github
Writing .github/workflows
Writing .github/workflows/testdriver.yml

Testdriver setup complete!

Create a new test by running:
testdriverai testdriver/test.yml

Source

The TestDriver agent is open-source and available on NPM. You can browser the source and see all the data collected and how everything works.

API

The TestDriver agent does not contain any AI models within it. Instead, it uploads desktop context to our API which uses that context to make decisions about what actions to perform.

Desktop Context Collected

During execution the TestDriver agent uploads the following information to our API

• User input prompts

• The active window and other windows that may be open (including application and window titles)

• System information

• The mouse position

• Screenshots of the desktop

With the exception of desktop screenshots, desktop context is persisted into our database.

Desktop Screenshots

TestDriver frequently takes screenshots of the desktop to provide our AI with decisions making context. You will not be prompted. Desktop screenshots are uploaded to our API for processing but are not persisted.

The TestDriver Agent will only take screenshots of the primary display. For complete privacy, we recommend running TestDriver within a virtual machine on your desktop.

TestDriver can not operate without visual context. Do not install TestDriver if you do not want to capture images of the desktop.

Active Window

System Information

User Prompts

The prompts you input to TestDriver are uploaded to our API and persisted in a database. We store this data to provide our AI with a history of context.

Additional Analytics

When running testdriver init you'll be asked if you'd like to share additional analytics. Sharing usage analytics is opt-in, this extra data will not be collected unless explicitly set in your environment.

If you would like to disable additional analytics, you can set TD_ANALYTICS within your environment.

TD_ANALYTICS=false

Rate Limiting and Other Restrictions

While the TestDriver Agent is free, we do reserve the right to rate limit or restrict usage by IP address for any reason.

Example Usage

command: wait-for-text
text: Copyright 2024

Use the assert to command generate an assertion. This will take a screenshot and use it to identify some criteria that ensures the task was complete.

assert No error message is displayed

This will "assert" that there's no error message, just like a user would see. The generated command will look like this:

- command: assert
  expect: There is no erorr message

When TestDriver runs this test, it will look at the screen and verify that the value of expect is true. If it is not true, the test will fail and exit immediately.

Many asserts can slow down a test. Use async: true to speed things up.

- command: assert
  expect: There is no erorr message
  async: true

The TestDriver action is open source. You can find the source below.

Ephemeral Virtual Machine Runners

TestDriver tests are executed on private virtual machines managed by Amazon EC2. These VMs are ephemeral meaning they only exist for the lifetime of the test execution. After the test completes, the VM is destroyed and the hard disk is wiped.

Secrets

Any secrets supplied within Prerun Scripts or prompts will be transmitted over SSL to our API. Secrets supplied to agent prompts will be persisted (see Agent), but prerun scripts are not persisted.

A common workflow is to use Prerun Scripts to access a private staging website via basic auth. This will allow you to securely log into staging without persisting sensitive data on our servers.

Production

Testing production resources does not require any private information from your team. Simply provide the tests to TestDriver and point toward your publicly available endpoints. TestDriver does not need access to any private information.

Staging

Depending on your implementation, TestDriver may need secure information to test staging environments. See Secretsabove for more information on securely implementing tests on staging.

Development

TestDriver can clone feature branches and build code on it's virtual machines using a similar workflow to GitHub runners.

In order for TestDriver to test development branches of private codebases, it's necessary to supply a GitHub Token within the GitHub action. This personal access token is used to clone the codebase on the VM.

env:
  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Example Usage

yamlCopy codecommand: wait-for-image
description: trash icon

This command generates a YAML file with the history of executed commands and tasks.

> /save

  saving...

  Current test script:

  version: 4.0.0
  steps:
    - prompt: navigate to fiber.google.com
      commands:
        - command: focus-application
          name: Google Chrome
        - command: hover-text
          text: Search Google or type a URL
          description: main google search
          action: click
        - command: type
          text: fiber.google.com
        - command: press-keys
          keys:
            - enter

Example Usage

To run a test you've previously created, use the /run command.

testdriverai
> /run helloworld.yml

TestDriver will run the test plan back performing each command.

This command will exit the program upon execution. Any failures will be output and the program will exit with code 1.

Dashcam and TestDriver share the same API and web application back end. This web application includes the following privacy and security features:

Runs the specified file.

The command will return exit code 0if the test is successful and 1if a failure.

If you're on mac, you might see an error related to "screen capture permissions" when getting started. This means that the application invoking the `testdriverai` command does not have permission to capture.

You must enable screen capture permissions for this application

This might be a little confusing at first. It's not testdriverai that needs the permissions, it's the terminal or IDE that's calling testdriverai that needs them. This is probably:

• Mac Terminal

• iTerm

• VS Code

• Or something similar

Enable Screen Capture Permissions for TestDriver

To enable screen capture permissions, do the following.

• Open System Settings (or System Preferences on older versions).

• Navigate to Privacy & Security > Screen Recording.

• Find and enable permissions for Terminal (or any terminal-based app like iTerm2, if you're using it).

• If the desired application is unavailable in the list, click on "+" symbol below, you will get a finder pop up, here find the app that you are currently using under Application section and click on open.

• You may need to restart the terminal for the changes to take effect.

This is the core testdriver command that will launch the testdriver agent.

testdriverai

Supply a file command to specify output location

testdriverai path/to/file.yml

This defaults to testdriver/testdriver.yml