Computer-Use Agent

TestDriver's Computer Use Agent builds and runs test by using a computer.

Making tests with TestDriver is super easy! Just tell TestDriver what to test on your local machine and TestDriver will look at the screen, generate some commands to control the computer, and execute them.

Interactive Commands

For this example, we'll be testing an example todo app. The live site is here, and the source for this quick start is here.

TestDriver is not limited to web apps! It has full control of the desktop so you can test desktop apps, browser extensions, or anything else!

Prerequisites

Report issues to the testdriverai/testdriverai repo

You'll need NodeJS version 20 to get started. Windows users will need a couple extra tools.

Windows (Choclatey)

Install tools with Chocolatey

Install Python & Visual Studio Build Tools

choco install python visualstudio2022-workload-vctools -y

Install NodeJS

You will also need NodeJS if you don't have it yet.

choco install nodejs-lts --version="20.17.0"

Set Execution Policy

Open a new terminal with admin privileges and execute the following command :

Set-ExecutionPolicy RemoteSigned -Scope CurrentUser

This gives TestDriver the right to execute it's scripts and is only valid for the current user.

Windows (WinGet)

Install tools with WinGet

You can install winget if you haven't already by downloading the App installer from Microsoft store.

Once its downloaded you need to add it to the system PATH environment variable. It's installed here:

C:\Users\<your-username>\AppData\Local\Microsoft\WindowsApps 

Don't forget to replace it with your actual username.

Install Python

winget install python

Install Visual Studio Build Tools

winget install --id=Microsoft.VisualStudio.2022.Community --override "--add Microsoft.VisualStudio.Workload.VCTools" --accept-package-agreements --accept-source-agreements

Install NodeJS

winget install --id=OpenJS.NodeJS.LTS --version="20.17.0"

Set Execution Policy

Open a new terminal with admin privileges and execute the following command :

Set-ExecutionPolicy RemoteSigned -Scope CurrentUser

This gives TestDriver the right to execute it's scripts and is only valid for the current user.

Mac

Install NodeJS

Install NVM using brew:

brew install nvm

Install NodeJS 20

nvm install 20.17.0

Setup

Install TestDriver via NPM

Install testdriverai via NPM. This will make testdriverai available as a global command.

npm install testdriverai -g

Set up the project

Run testdriverai init in a new folder.

testdriverai init

This will walk you through setting up a local project.

Set up your test environment

Before we get started, let's set up your machine to collaborate with TestDriver.

Display

TestDriver isn't like any framework you've used before. TestDriver makes decisions based on what it can see on your display!

TestDriver only knows about what it can see on your primary display!

For now, set up your environment with a browser window and your terminal side by side like so:

Your desktop should look like this

When you enter commands into TestDriver, the current terminal window will minimize and the focus-window command will bring Chrome or other applications to the foreground.

Application State

The application you want to test should be visible before you run the testdriverai command.

For our example, make a new incognito window in Chrome and load our test webpage:

https://testdriverai.github.io/example-react-todo/

Teach TestDriver

It's time to test! The following command will launch TestDriver in interactive mode. TestDriver will generate test commands and save them totestdriver/test.yml.

testdriverai testdriver/test.yml

Instruct TestDriver

Now, just tell TestDriver what you want it to do. For our example, try click new task.

> click new task

TestDriver Generates a Test

TestDriver will look at your screen and generate a test script. TestDriver can see the screen, control the mouse, keyboard, and more!

> click new task

    thinking...

    To accomplish the goal of clicking "New Task" on the
    webpage, we need to focus on the browser window and then
    click the "New Task" button.

    Here is the YAML code to achieve this:

    commands:
      - command: focus-application
        name: Google Chrome
      - command: hover-text
        text: New Task
        description: button to create a new task
        action: click

command='focus-application' name='Google Chrome'
command='hover-text' text='New Task' description='button to create a new task' action='click'

    thinking...

    Based on the provided coordinates and text, the element
    that matches "New Task" with the description "button to
    create a new task" is at coordinates (864, 328).

    Here is the YAML code to click on that element:

    commands:
      - command: click
        x: 864
        y: 328
        click: single
        button: left

command='click' x=864 y=328 click='single' button='left'

TestDriver executes the test script

TestDriver will execute the commands found in the yml codeblocks of the response.

See the yml TestDriver generated? That's our own schema. You can learn more about it in the Test Steps.

Take your hands off the mouse and keyboard while TestDriver executes! TestDriver is not a fan of backseat drivers.

Keep going!

Feel free to ask TestDriver to perform some more tasks. Every time you prompt TestDriver it will look at your screen and generate more test step to complete your goal.

> enter a task title
> enter a task summary
> click create task
> delete the task

If something didn't work, you can use /undo to remove all of the test steps added since the last prompt.

Test the test locally

Now it's time to make sure the test plan works before we deploy it. Use testdriver run to run the test file you just created with /save .

testdriverai run testdriver/test.yml

Make sure to reset the test state!