find() - TestDriver

Overview

Locate UI elements on screen using AI-powered natural language descriptions. Returns an Element object that can be interacted with.

Syntax

const element = await testdriver.find(description)
const element = await testdriver.find(description, options)

Parameters

description

string

required

Natural language description of the element to find

options

object | number

Optional configuration for finding and caching

Show properties

cacheKey

string

Custom cache key for storing element location. Use this to prevent cache pollution when using dynamic variables in prompts, or to share cache across tests.

cacheThreshold

number

Similarity threshold (0-1) for cache matching. Lower values require more similarity. Set to -1 to disable cache.

timeout

number

Maximum time in milliseconds to poll for the element. Retries every 5 seconds until found or timeout expires.

confidence

number

Minimum confidence threshold (0-1). If the AI’s confidence score for the found element is below this value, the find will be treated as a failure (element.found() returns false). Useful for ensuring high-quality matches in critical test steps.

type

string

Element type hint that wraps the description for better matching. Accepted values:

"text" — Wraps the prompt as The text "..."
"image" — Wraps the prompt as The image "..."
"ui" — Wraps the prompt as The UI element "..."
"any" — No wrapping, uses the description as-is (default behavior)

zoom

boolean

default:false

Enable two-phase zoom mode for better precision in crowded UIs with many similar elements.

object

AI sampling configuration for this find call (overrides global ai config from constructor).

Show properties

temperature

number

Controls randomness. 0 = deterministic. Default: 0 for find verification.

top

object

Sampling parameters

Show properties

number

Top-P (nucleus sampling). Range: 0-1.

number

Top-K sampling. 1 = most deterministic.

Returns

Promise<Element> - Element instance that has been automatically located

Examples

Basic Element Finding

// Find by role
const button = await testdriver.find('submit button');
const input = await testdriver.find('email input field');

// Find by text content
const link = await testdriver.find('Contact Us link');
const heading = await testdriver.find('Welcome heading');

// Find by visual appearance
const icon = await testdriver.find('red warning icon');
const image = await testdriver.find('company logo image');

Finding with Context

// Provide location context
const field = await testdriver.find('username input in the login form');
const button = await testdriver.find('delete button in the top right corner');

// Describe nearby elements
const input = await testdriver.find('input field below the email label');
const checkbox = await testdriver.find('checkbox next to "Remember me"');

// Describe visual position
const menu = await testdriver.find('hamburger menu icon in the top left');

Interacting with Found Elements

// Find and click
const submitBtn = await testdriver.find('submit button');
await submitBtn.click();

// Find and verify
const message = await testdriver.find('success message');
if (message.found()) {
  console.log('Success message appeared');
}

// Find and extract info
const price = await testdriver.find('product price');
console.log('Price location:', price.coordinates);
console.log('Price text:', price.text);

Element Object

The returned Element object provides:

Methods

found() - Check if element was located
click(action) - Click the element
hover() - Hover over the element
doubleClick() - Double-click the element
rightClick() - Right-click the element
find(newDescription) - Re-locate with optional new description

Properties

coordinates - Element position {x, y, centerX, centerY}
x, y - Top-left coordinates
centerX, centerY - Center coordinates
text - Text content (if available)
screenshot - Base64 screenshot (if available)
confidence - AI confidence score
width, height - Element dimensions
boundingBox - Complete bounding box

See Elements Reference for complete details.

JSON Serialization

Elements can be safely serialized using JSON.stringify() for logging and debugging. Circular references are automatically removed:

const element = await testdriver.find('login button');

// Safe to stringify - no circular reference errors
console.log(JSON.stringify(element, null, 2));

// Output includes useful debugging info:
// {
//   "description": "login button",
//   "coordinates": { "x": 100, "y": 200, "centerX": 150, "centerY": 225 },
//   "found": true,
//   "threshold": 0.01,
//   "x": 100,
//   "y": 200,
//   "cache": {
//     "hit": true,
//     "strategy": "pixel-diff",
//     "createdAt": "2025-12-09T10:30:00Z",
//     "diffPercent": 0.0023,
//     "imageUrl": "https://..."
//   },
//   "similarity": 0.98,
//   "confidence": 0.95,
//   "selector": "button#login",
//   "aiResponse": "Found the blue login button..."
// }

This is useful for:

Debugging element detection issues
Logging test execution details
Sharing element information across processes
Analyzing cache performance

Best Practices

Be specific in descriptionsMore specific descriptions improve accuracy:

// ✅ Good
await testdriver.find('blue submit button below the email field');

// ❌ Too vague
await testdriver.find('button');

Always check if foundVerify elements were located before interacting:

const element = await testdriver.find('login button');
if (!element.found()) {
  throw new Error('Login button not found');
}
await element.click();

Include visual or positional context

// Include color
await testdriver.find('red error icon');

// Include position
await testdriver.find('search button in the top navigation bar');

// Include nearby text
await testdriver.find('checkbox next to "I agree to terms"');

Confidence Threshold

Require a minimum AI confidence score for element matches. If the confidence is below the threshold, find() treats the result as not found:

// Require at least 90% confidence
const element = await testdriver.find('submit button', { confidence: 0.9 });

if (!element.found()) {
  // AI found something but wasn't confident enough
  throw new Error('Could not confidently locate submit button');
}

await element.click();

This is useful for:

Critical test steps where an incorrect click could cause cascading failures
Distinguishing between similar elements (e.g., multiple buttons)
Failing fast when the UI has changed unexpectedly

// Combine with timeout for robust polling with confidence gate
const element = await testdriver.find('success notification', {
  confidence: 0.85,
  timeout: 15000,
});

The confidence value is a float between 0 and 1 (e.g., 0.9 = 90%). The AI returns its confidence with each find result, which you can also read from element.confidence after a successful find.

Element Type

Use the type option to hint what kind of element you’re looking for. This wraps your description into a more specific prompt for the AI, improving match accuracy — especially when users provide short or ambiguous descriptions.

// Find text on the page
const label = await testdriver.find('Sign In', { type: 'text' });
// AI prompt becomes: The text "Sign In"

// Find an image
const logo = await testdriver.find('company logo', { type: 'image' });
// AI prompt becomes: The image "company logo"

// Find a UI element (button, input, checkbox, etc.)
const btn = await testdriver.find('Submit', { type: 'ui' });
// AI prompt becomes: The UI element "Submit"

// No wrapping — same as omitting the option
const el = await testdriver.find('the blue submit button', { type: 'any' });

Type	Prompt sent to AI
`"text"`	`The text "..."`
`"image"`	`The image "..."`
`"ui"`	`The UI element "..."`
`"any"`	Original description (no wrapping)

This is particularly useful for short descriptions like "Submit" or "Login" where the AI may not know whether to look for a button, a link, or visible text. Specifying type removes the ambiguity.

Polling for Dynamic Elements

For elements that may not be immediately visible, use the timeout option to automatically poll:

// Poll for element (retries every 5 seconds until found or timeout)
const element = await testdriver.find('login button', { timeout: 30000 });
await element.click();

The timeout option:

Retries finding the element every 5 seconds
Stops when the element is found or the timeout expires
Logs progress during polling
Returns the element (check element.found() if not throwing on failure)

Zoom Mode for Crowded UIs

When dealing with many similar icons or elements clustered together (like browser toolbars), enable zoom mode for better precision:

// Enable zoom for better precision in crowded UIs
const extensionsBtn = await testdriver.find('extensions puzzle icon in Chrome toolbar', { zoom: true });
await extensionsBtn.click();

How Zoom Mode Works

Phase 1: AI identifies the approximate location of the element
Phase 2: A 30% crop of the screen is created around that location
Phase 3: AI performs precise location on the zoomed/cropped image
Result: Coordinates are converted back to absolute screen position

This two-phase approach gives the AI a higher-resolution view of the target area, improving accuracy when multiple similar elements are close together.

Use zoom: true when:

Clicking small icons in toolbars
Selecting from a grid of similar items
Targeting elements in dense UI areas
The default locate is clicking the wrong similar element
You get an AI verification rejection like “The crosshair is located in the empty space of the browser’s tab bar/title bar area” — this means the initial locate was imprecise and zoom will help the AI pinpoint the correct element

// Without zoom - may click wrong icon in toolbar
const icon = await testdriver.find('settings icon');

// With zoom - better precision for crowded areas
const icon = await testdriver.find('settings icon', { zoom: true });

Cache Options

Control caching behavior to optimize performance, especially when using dynamic variables in prompts.

Custom Cache Key

Use cacheKey to prevent cache pollution when prompts contain variables:

// ❌ Without cacheKey - creates new cache entry for each email value
const email = 'user@example.com';
await testdriver.find(`input for ${email}`); // Cache miss every time

// ✅ With cacheKey - reuses cache regardless of variable
const email = 'user@example.com';
await testdriver.find(`input for ${email}`, {
  cacheKey: 'email-input'
});

// Also useful for dynamic IDs, names, or other changing data
const orderId = generateOrderId();
await testdriver.find(`order ${orderId} status`, {
  cacheKey: 'order-status'  // Same cache for all orders
});

Cache Threshold

Control how similar a cached result must be to reuse it:

// Default: 95% similarity required
await testdriver.find('submit button');

// Strict threshold - 99% similarity required
await testdriver.find('submit button', {
  cacheThreshold: 0.01
});

// Disable cache entirely for this call
await testdriver.find('submit button', {
  cacheThreshold: -1
});

// Combine cacheKey with threshold
await testdriver.find('submit button', {
  cacheKey: 'submit-btn',
  cacheThreshold: 0.01
});

By default, TestDriver auto-generates a cache key from the SHA-256 hash of your test file. When you modify your test file, the hash changes automatically, invalidating stale cache entries.

Manual Polling (Alternative)

If you need custom polling logic:

async function waitForElement(testdriver, description, timeout = 30000) {
  const startTime = Date.now();
  
  while (Date.now() - startTime < timeout) {
    const element = await testdriver.find(description);
    if (element.found()) return element;
    await new Promise(r => setTimeout(r, 1000));
  }
  
  throw new Error(`Element "${description}" not found after ${timeout}ms`);
}

// Usage
const button = await waitForElement(testdriver, 'submit button', 10000);
await button.click();

Use Cases

Form Fields

const emailField = await testdriver.find('email input field');
await emailField.click();
await testdriver.type('user@example.com');

const passwordField = await testdriver.find('password input');
await passwordField.click();
await testdriver.type('MyP@ssw0rd');

Buttons and Links

const submitBtn = await testdriver.find('submit button');
await submitBtn.click();

const cancelLink = await testdriver.find('cancel link');
await cancelLink.click();

const menuIcon = await testdriver.find('hamburger menu icon');
await menuIcon.click();

Dynamic Content

// Wait for loading to complete
let content;
for (let i = 0; i < 30; i++) {
  content = await testdriver.find('results table');
  if (content.found()) break;
  await new Promise(r => setTimeout(r, 1000));
}

// Interact with loaded content
const firstRow = await testdriver.find('first row in the results table');
await firstRow.click();

Complex UI Elements

// Modals and dialogs
const modal = await testdriver.find('confirmation dialog');
if (modal.found()) {
  const confirmBtn = await testdriver.find('confirm button in the dialog');
  await confirmBtn.click();
}

// Dropdown menus
const dropdown = await testdriver.find('country dropdown');
await dropdown.click();

const option = await testdriver.find('United States option');
await option.click();

Complete Example

import { beforeAll, afterAll, describe, it, expect } from 'vitest';
import TestDriver from 'testdriverai';

describe('Element Finding', () => {
  let testdriver;

  beforeAll(async () => {
    client = new TestDriver(process.env.TD_API_KEY);
    await testdriver.auth();
    await testdriver.connect();
  });

  afterAll(async () => {
    await testdriver.disconnect();
  });

  it('should find and interact with elements', async () => {
    await testdriver.focusApplication('Google Chrome');
    
    // Find login form elements
    const usernameField = await testdriver.find('username input field');
    expect(usernameField.found()).toBe(true);
    
    await usernameField.click();
    await testdriver.type('testuser');
    
    // Find with context
    const passwordField = await testdriver.find('password input below username');
    await passwordField.click();
    await testdriver.type('password123');
    
    // Find button
    const submitBtn = await testdriver.find('green submit button');
    expect(submitBtn.found()).toBe(true);
    
    console.log('Button location:', submitBtn.centerX, submitBtn.centerY);
    
    await submitBtn.click();
    
    // Wait for success message
    let successMsg;
    for (let i = 0; i < 10; i++) {
      successMsg = await testdriver.find('success notification');
      if (successMsg.found()) break;
      await new Promise(r => setTimeout(r, 1000));
    }
    
    expect(successMsg.found()).toBe(true);
  });
});

click() - Click on found elements
hover() - Hover over elements
assert() - Verify element states
Elements Reference - Complete Element API

findAll()

Locate all elements matching a description, rather than just one.

Syntax

const elements = await testdriver.findAll(description, options)

Parameters

description

string

required

Natural language description of elements to find

options

object | number

Optional cache options (same as find())

Show properties

cacheKey

string

Cache key for storing element location

cacheThreshold

number

Similarity threshold (0-1) for cache matching. Set to -1 to disable cache.

Returns

Promise<Element[]> - Array of Element instances

Examples

Basic Usage

// Find all matching elements
const buttons = await testdriver.findAll('button');
console.log(`Found ${buttons.length} buttons`);

// Interact with specific element
if (buttons.length > 0) {
  await buttons[0].click(); // Click first button
}

// Iterate over all
for (const button of buttons) {
  console.log(`Button at (${button.x}, ${button.y})`);
}

Finding Multiple Items

// Find all list items
const items = await testdriver.findAll('list item');

// Find specific item by index
const thirdItem = items[2];
await thirdItem.click();

// Check all items
for (let i = 0; i < items.length; i++) {
  console.log(`Item ${i + 1}: ${items[i].text || 'No text'}`);
}

With Caching

// Cache element locations for faster subsequent runs
const menuItems = await testdriver.findAll('menu item', {
  cacheKey: 'main-menu-items'
});

// First run: ~2-3 seconds (AI call)
// Subsequent runs: ~100ms (cache hit)

Empty Results

// Returns empty array if nothing found (doesn't throw error)
const errors = await testdriver.findAll('error message');

if (errors.length === 0) {
  console.log('No errors found - test passed!');
} else {
  console.log(`Found ${errors.length} errors`);
}

Differences from find()

Feature	find()	findAll()
Return type	Single `Element`	Array of `Element[]`
If nothing found	Throws `ElementNotFoundError`	Returns empty array `[]`
Chainable	✅ Yes: `await find('button').click()`	❌ No (returns array)
Use case	One specific element	Multiple similar elements
Cache support	✅ Yes	✅ Yes

Use Cases

Table Rows

// Find all rows in a table
const rows = await testdriver.findAll('table row');

// Click every row
for (const row of rows) {
  await row.click();
  await new Promise(r => setTimeout(r, 500)); // Wait between clicks
}

// Or click specific row
await rows[2].click(); // Click third row

Checkboxes/Radio Buttons

// Find all checkboxes
const checkboxes = await testdriver.findAll('checkbox');

// Check all boxes
for (const checkbox of checkboxes) {
  await checkbox.click();
}

// Or select first unchecked
const unchecked = checkboxes[0];
await unchecked.click();

Navigation Links

// Find all navigation links
const navLinks = await testdriver.findAll('navigation link');

// Validate all are present
expect(navLinks.length).toBeGreaterThan(0);

// Click specific link by text
const homeLink = navLinks.find(link => 
  link.text?.toLowerCase().includes('home')
);

if (homeLink) {
  await homeLink.click();
}

Conditional Interactions

// Check if any error messages exist
const errors = await testdriver.findAll('error message');

if (errors.length > 0) {
  console.log(`Found ${errors.length} validation errors`);
  
  // Log each error location
  errors.forEach((error, i) => {
    console.log(`Error ${i + 1} at (${error.x}, ${error.y})`);
  });
} else {
  console.log('Form validation passed!');
}

Complete Example

import { test, expect } from 'vitest';
import { chrome } from 'testdriverai/presets';

test('select multiple items from list', async (context) => {
  const { testdriver } = await chrome(context, {
    url: 'https://example.com/products'
  });
  
  // Find all product cards
  const products = await testdriver.findAll('product card');
  
  expect(products.length).toBeGreaterThan(0);
  console.log(`Found ${products.length} products`);
  
  // Click first 3 products
  const productsToSelect = Math.min(3, products.length);
  
  for (let i = 0; i < productsToSelect; i++) {
    await products[i].click();
    console.log(`Selected product ${i + 1}`);
    await new Promise(r => setTimeout(r, 500)); // Brief pause
  }
  
  // Verify selections
  const selectedBadges = await testdriver.findAll('selected badge');
  expect(selectedBadges.length).toBe(productsToSelect);
});

Best Practices

Handle empty arrays gracefully

// ✅ Good - check length first
const items = await testdriver.findAll('list item');
if (items.length > 0) {
  await items[0].click();
}

// ❌ Bad - may throw error
const items = await testdriver.findAll('list item');
await items[0].click(); // Error if array is empty!

Use find() for single elements

// ✅ Use find() when you need exactly one
const submitBtn = await testdriver.find('submit button');
await submitBtn.click();

// ❌ Unnecessary - findAll() returns array
const buttons = await testdriver.findAll('submit button');
await buttons[0].click(); // Extra array handling

Cache for performance

// First run - slow (AI call)
const items = await testdriver.findAll('menu item', {
  cacheKey: 'menu-items'
});

// Subsequent runs - fast (cache hit)
// ~10-20x faster than without cache

Overview

Creating Tests

Running Tests

Scaling

Actions

SDK

​Overview

​Syntax

​Parameters

​Returns

​Examples

​Basic Element Finding

​Finding with Context

​Interacting with Found Elements

​Element Object

​Methods

​Properties

​JSON Serialization

​Best Practices

​Confidence Threshold

​Element Type

​Polling for Dynamic Elements

​Zoom Mode for Crowded UIs

​How Zoom Mode Works

​Cache Options

​Custom Cache Key

​Cache Threshold

​Manual Polling (Alternative)

​Use Cases

​Complete Example

​Related Methods

​findAll()

​Syntax

​Parameters

​Returns

​Examples

​Basic Usage

​Finding Multiple Items

​With Caching

​Empty Results

​Differences from find()

​Use Cases

​Complete Example

​Best Practices

Overview

Syntax

Parameters

Returns

Examples

Basic Element Finding

Finding with Context

Interacting with Found Elements

Element Object

Methods

Properties

JSON Serialization

Best Practices

Confidence Threshold

Element Type

Polling for Dynamic Elements

Zoom Mode for Crowded UIs

How Zoom Mode Works

Cache Options

Custom Cache Key

Cache Threshold

Manual Polling (Alternative)

Use Cases

Complete Example

Related Methods

findAll()

Syntax

Parameters

Returns

Examples

Basic Usage

Finding Multiple Items

With Caching

Empty Results

Differences from find()

Use Cases

Complete Example

Best Practices