Elements - TestDriver

Overview

TestDriver’s element finding system uses AI to locate elements on screen using natural language descriptions. The find() method returns an Element object with coordinates, metadata, and interaction methods. find() returns a ChainableElementPromise — a Promise<Element> that exposes chainable interaction methods so you can find and act in a single expression.

// Chain find + action in one line
await testdriver.find('submit button').click();

// Or capture the element for multiple interactions
const element = await testdriver.find('submit button');
await element.click();

Finding Elements

find()

Locate a single element on screen using a natural language description.

const element = await testdriver.find(description, options?)

description

string

required

Natural language description of the element to find. Be specific — include visual details, location context, or nearby text.

options

FindOptions

Optional configuration for the find operation.

Show properties

timeout

number

default:10000

Maximum time in milliseconds to search for the element. The SDK polls every 5 seconds within this window.

confidence

number

Minimum AI confidence threshold (0–1). Matches below this score are rejected.

type

string

Find strategy hint: "text", "image", "ui", or "any".

zoom

number

Zoom level for the find operation. Useful for small or dense UIs.

cacheKey

string

Cache key for this find. When provided, enables coordinate caching tied to this key.

AIConfig

Per-command AI configuration override.

Show properties

temperature

number

Controls randomness. 0 = deterministic.

top

object

Show properties

number

Nucleus sampling threshold (0–1).

number

Top-K sampling. 1 = greedy.

cache

object

Per-command cache threshold overrides.

Show properties

thresholds

object

Show properties

screen

number

Pixel diff threshold for screen comparison (0–1). 0.05 = 5% diff allowed.

element

number

OpenCV template match threshold (0–1). 0.8 = 80% correlation required.

Returns: ChainableElementPromise — a Promise<Element> with chainable methods.

// Find a button
const submitButton = await testdriver.find('the submit button');

// Find with options
const element = await testdriver.find('email input field', {
  timeout: 20000,
  confidence: 0.9
});

// Chain directly
await testdriver.find('red button in the top right corner').click();

Be specific in your descriptions. Include visual details, location context, or nearby text to improve accuracy. For example, "blue submit button below the email field" is better than "button".

findAll()

Locate all matching elements on screen.

const elements = await testdriver.findAll(description, options?)

description

string

required

Natural language description of the elements to find.

options

FindOptions

Same options as find().

Returns: Promise<Element[]> — Array of Element instances.

const buttons = await testdriver.findAll('navigation menu items');
console.log(`Found ${buttons.length} menu items`);

for (const button of buttons) {
  console.log(`Item at (${button.x}, ${button.y}): ${button.text}`);
}

Retry Behavior

The find() method uses a timeout-based retry loop:

Captures a screenshot and sends it to the AI with the description
If the element is not found, waits 5 seconds and retries
Repeats until the timeout expires (default: 10,000ms)
On final failure, throws an ElementNotFoundError with debug data

This means a timeout of 10000 allows approximately 2 attempts (initial + 1 retry).

ChainableElementPromise

When you call testdriver.find(), it returns a ChainableElementPromise — a special Promise that lets you chain interaction methods directly without awaiting first.

// These are equivalent:
await testdriver.find('button').click();

const el = await testdriver.find('button');
await el.click();

Available chainable methods

Method	Description
`.click()`	Click the element
`.hover()`	Hover over the element
`.doubleClick()`	Double-click the element
`.rightClick()`	Right-click the element
`.mouseDown()`	Press mouse button on element
`.mouseUp()`	Release mouse button on element
`.getCoordinates()`	Get `{ x, y }` coordinates
`.getResponse()`	Get full API response

Available chainable getters

Getter	Type	Description
`.found`	`boolean`	Whether element was located
`.x`	`number`	Top-left X coordinate
`.y`	`number`	Top-left Y coordinate
`.centerX`	`number`	Center X coordinate
`.centerY`	`number`	Center Y coordinate

Element Class

The Element class represents a located UI element. It provides methods for interaction and properties for element information.

class Element {
  description: string;
  coordinates: ElementCoordinates | null;

  // Getters
  x: number;
  y: number;
  centerX: number;
  centerY: number;
  width: number | null;
  height: number | null;
  boundingBox: ElementBoundingBox | null;
  screenshot: string | null;
  confidence: number | null;
  reasoning: string | null;
  text: string | null;
  label: string | null;

  // Methods
  found(): boolean;
  find(newDescription?: string, options?: FindOptions): Promise<Element>;
  click(action?: ClickAction): Promise<void>;
  hover(): Promise<void>;
  doubleClick(): Promise<void>;
  rightClick(): Promise<void>;
  mouseDown(): Promise<void>;
  mouseUp(): Promise<void>;
  getCoordinates(): Promise<ElementCoordinates>;
  getResponse(): Promise<object>;
  saveDebugScreenshot(filepath?: string): Promise<string>;
  getDebugInfo(): object;
  destroy(): void;
  toJSON(): object;
}

Methods

found()

Check if the element was successfully located.

element.found() // boolean

Returns: boolean — true if element coordinates were found.

const element = await testdriver.find('login button');
if (element.found()) {
  await element.click();
} else {
  console.log('Element not found');
}

find()

Re-locate the element, optionally with a new description.

await element.find(newDescription?, options?)

newDescription

string

New description to search for. If omitted, re-uses the original description.

options

FindOptions

Same options as testdriver.find().

Returns: Promise<Element> — This element instance (mutated in place).

const element = await testdriver.find('submit button');

// Page updates...
await element.find(); // Re-locate with same description

// Or update the description
await element.find('blue submit button');

click()

Click on the element.

await element.click(action?)

action

ClickAction

default:"click"

Type of click action.

const button = await testdriver.find('submit button');
await button.click();                  // Regular click
await button.click('double-click');    // Double-click
await button.click('right-click');     // Right-click

The element must be found before clicking. The find() method automatically locates the element.

hover()

Hover over the element without clicking.

await element.hover() // Promise<void>

const tooltip = await testdriver.find('info icon');
await tooltip.hover();
await testdriver.wait(1000); // Wait for tooltip to appear

doubleClick()

Double-click on the element.

await element.doubleClick() // Promise<void>

const file = await testdriver.find('README.txt file icon');
await file.doubleClick();

rightClick()

Right-click on the element to open a context menu.

await element.rightClick() // Promise<void>

const folder = await testdriver.find('Documents folder');
await folder.rightClick();

mouseDown() / mouseUp()

Press or release mouse button on the element. Used for drag operations.

await element.mouseDown() // Promise<void>
await element.mouseUp()   // Promise<void>

// Drag and drop
const item = await testdriver.find('draggable item');
await item.mouseDown();

const target = await testdriver.find('drop zone');
await target.hover();
await target.mouseUp();

getCoordinates()

Get the element’s coordinates. If the element hasn’t been found yet, this triggers a lazy find.

const coords = await element.getCoordinates()

Returns: Promise<{ x: number, y: number }> — Top-left coordinate.

getResponse()

Get the full API response from the find operation. If the element hasn’t been found yet, this triggers a lazy find.

const response = await element.getResponse()

Returns: Promise<object> — Raw API response including all metadata.

saveDebugScreenshot()

Save the screenshot used during element finding to a file.

const path = await element.saveDebugScreenshot(filepath?)

filepath

string

Output file path. Default: auto-generated in temp directory.

Returns: Promise<string> — Path to saved screenshot.

getDebugInfo()

Get a summary object useful for debugging.

const info = element.getDebugInfo()

Returns: { description, coordinates, confidence, reasoning, boundingBox }

destroy()

Clear internal references to free memory. Call when done with the element.

element.destroy()

Properties

Coordinates

Property	Type	Description
`coordinates`	`{ x, y } \| null`	Coordinate object after `find()`
`x`	`number`	Top-left X coordinate (from `boundingBox.x`)
`y`	`number`	Top-left Y coordinate (from `boundingBox.y`)
`centerX`	`number`	Center X coordinate
`centerY`	`number`	Center Y coordinate

const button = await testdriver.find('submit button');
console.log(`Position: (${button.x}, ${button.y})`);
console.log(`Center: (${button.centerX}, ${button.centerY})`);

Dimensions

Property	Type	Description
`width`	`number \| null`	Width in pixels (from boundingBox)
`height`	`number \| null`	Height in pixels (from boundingBox)
`boundingBox`	`ElementBoundingBox \| null`	`{ x, y, width, height }`

const dialog = await testdriver.find('dialog box');
if (dialog.boundingBox) {
  const { x, y, width, height } = dialog.boundingBox;
  console.log(`Dialog: ${width}x${height} at (${x}, ${y})`);
}

AI Metadata

Property	Type	Description
`confidence`	`number \| null`	AI confidence score (0–1)
`reasoning`	`string \| null`	AI’s explanation of what it found
`text`	`string \| null`	Text content extracted by AI
`label`	`string \| null`	Accessible label detected by AI

const el = await testdriver.find('submit button');
console.log(`Confidence: ${(el.confidence * 100).toFixed(1)}%`);
console.log(`Reasoning: ${el.reasoning}`);
console.log(`Text: ${el.text}`);

Confidence scores below 0.8 may indicate the element description was ambiguous or the wrong element was found.

Debug Data

Property	Type	Description
`screenshot`	`string \| null`	Base64-encoded PNG of screen during find

Screenshots can be large. They’re automatically excluded from error messages to prevent memory issues.

Property Availability

Property	When Available
`x`, `y`, `centerX`, `centerY`	Always after successful `find()`
`coordinates`	Always after successful `find()`
`width`, `height`, `boundingBox`	When AI detects element bounds
`text`	When AI extracts text content
`label`	When element has accessible label
`confidence`, `reasoning`	Always after AI element finding
`screenshot`	Only in DEBUG mode or on errors

Types

interface ElementCoordinates {
  x: number;
  y: number;
}

interface ElementBoundingBox {
  x: number;       // Top-left X
  y: number;       // Top-left Y
  width: number;   // Width in pixels
  height: number;  // Height in pixels
}

type ClickAction =
  | "click"
  | "double-click"
  | "right-click"
  | "hover"
  | "mouseDown"
  | "mouseUp";

interface FindOptions {
  timeout?: number;          // Default: 10000
  confidence?: number;       // Min confidence threshold
  type?: "text" | "image" | "ui" | "any";
  zoom?: number;
  cacheKey?: string;
  ai?: AIConfig;
  cache?: {
    thresholds?: {
      screen?: number;       // Default: 0.05
      element?: number;      // Default: 0.8
    };
  };
}

interface AIConfig {
  temperature?: number;
  top?: {
    p?: number;              // Nucleus sampling
    k?: number;              // Top-K sampling
  };
}

JSON Serialization

Elements implement toJSON() for safe serialization. Circular references and large binary data (screenshots) are automatically excluded.

const element = await testdriver.find('login button');
console.log(JSON.stringify(element, null, 2));

Serialized output:

{
  "description": "login button",
  "coordinates": { "x": 100, "y": 200, "centerX": 150, "centerY": 225 },
  "found": true,
  "threshold": 0.01,
  "x": 100,
  "y": 200,
  "cache": {
    "hit": true,
    "strategy": "pixel-diff",
    "createdAt": "2025-12-09T10:30:00.000Z",
    "diffPercent": 0.0023,
    "imageUrl": "https://cache.testdriver.ai/..."
  },
  "similarity": 0.98,
  "confidence": 0.95,
  "selector": "button#login",
  "aiResponse": "Found the blue login button in the center of the form..."
}

Property	Type	Description
`description`	`string`	Element search description
`coordinates`	`object`	`{ x, y, centerX, centerY }`
`found`	`boolean`	Whether element was located
`threshold`	`number`	Cache threshold used
`cache.hit`	`boolean`	Whether cache was used
`cache.strategy`	`string`	Cache strategy (e.g., `"pixel-diff"`)
`cache.createdAt`	`string`	ISO timestamp of cache entry
`cache.diffPercent`	`number`	Pixel difference from cached image
`cache.imageUrl`	`string`	URL to cached screenshot
`similarity`	`number`	Similarity score (0–1)
`confidence`	`number`	AI confidence score (0–1)
`selector`	`string`	CSS/XPath selector if available
`aiResponse`	`string`	AI’s explanation of what it found

Examples

Chained Find + Action

// Most concise — chain directly
await testdriver.find('submit button').click();
await testdriver.find('settings icon').rightClick();
await testdriver.find('draggable item').mouseDown();

Working with Forms

const nameField = await testdriver.find('name input field');
await nameField.click();
await testdriver.type('John Doe');

const emailField = await testdriver.find('email input field');
await emailField.click();
await testdriver.type('john@example.com');

await testdriver.find('submit button').click();

Conditional Interactions

const closeButton = await testdriver.find('close popup button');

if (closeButton.found()) {
  await closeButton.click();
} else {
  console.log('No popup to close');
}

Re-locating After UI Changes

const notification = await testdriver.find('success notification');
await testdriver.scroll('down', { amount: 300 });

// Re-locate — the element may have moved
await notification.find();
if (notification.found()) {
  await notification.click();
}

Debugging Element Detection

const el = await testdriver.find('submit button');
const debug = el.getDebugInfo();
console.log('Debug:', JSON.stringify(debug, null, 2));

// Save the screenshot used during detection
await el.saveDebugScreenshot('./debug-screenshot.png');

Best Practices

Be specific with descriptions

Include visual details, position context, and nearby text:

// Too vague
await testdriver.find('button');

// Specific
await testdriver.find('blue submit button below the email field');

Check if element was found

Always verify elements were located before conditional logic:

const element = await testdriver.find('submit button');
if (!element.found()) {
  throw new Error('Submit button not found');
}
await element.click();

Use findAll() for multiple elements

When you need all matching elements, not just the first:

const items = await testdriver.findAll('list items in the sidebar');
console.log(`Found ${items.length} items`);

Reuse element references

If you need to interact with the same element multiple times:

const input = await testdriver.find('search input');
await input.click();
await testdriver.type('first search');
await testdriver.pressKeys(['enter']);

await input.click();
await testdriver.pressKeys(['ctrl', 'a']);
await testdriver.type('second search');

Use timeout for slow-loading elements

Increase timeout for elements that take time to appear:

// Wait up to 30 seconds for a loading indicator to appear
const element = await testdriver.find('success message', {
  timeout: 30000
});

Overview

Guides

AI

Queries

Mouse

Keyboard

System

Utilities

SDK

Infrastructure

​Overview

​Finding Elements

​find()

​findAll()

​Retry Behavior

​ChainableElementPromise

​Available chainable methods

​Available chainable getters

​Element Class

​Methods

​found()

​find()

​click()

​hover()

​doubleClick()

​rightClick()

​mouseDown() / mouseUp()

​getCoordinates()

​getResponse()

​saveDebugScreenshot()

​getDebugInfo()

​destroy()

​Properties

​Coordinates

​Dimensions

​AI Metadata

​Debug Data

​Property Availability

​Types

​JSON Serialization

​Examples

​Chained Find + Action

​Working with Forms

​Conditional Interactions

​Re-locating After UI Changes

​Debugging Element Detection

​Best Practices

Overview

Finding Elements

find()

findAll()

Retry Behavior

ChainableElementPromise

Available chainable methods

Available chainable getters

Element Class

Methods

found()

find()

click()

hover()

doubleClick()

rightClick()

mouseDown() / mouseUp()

getCoordinates()

getResponse()

saveDebugScreenshot()

getDebugInfo()

destroy()

Properties

Coordinates

Dimensions

AI Metadata

Debug Data

Property Availability

Types

JSON Serialization

Examples

Chained Find + Action

Working with Forms

Conditional Interactions

Re-locating After UI Changes

Debugging Element Detection

Best Practices