Element Finding - TestDriver

Overview

TestDriver’s element finding system uses AI to locate elements on screen using natural language descriptions. The find() method returns an Element object that you can interact with.

Finding Elements

find()

Locate an element on screen using a natural language description.

const element = await testdriver.find(description)

Parameters:

description (string) - Natural language description of the element to find

Returns: Promise<Element> - Element instance that has been located Example:

// Find a button
const submitButton = await testdriver.find('the submit button');

// Find an input field with context
const emailField = await testdriver.find('email input field in the login form');

// Find an element by visual characteristics
const redButton = await testdriver.find('red button in the top right corner');

Be specific in your descriptions. Include visual details, location context, or nearby text to improve accuracy.

Element Class

The Element class represents a located (or to-be-located) UI element. It provides methods for interaction and properties for element information.

Methods

found()

Check if the element was successfully located.

element.found()

Returns: boolean - True if element coordinates were found Example:

const element = await testdriver.find('login button');
if (element.found()) {
  await element.click();
} else {
  console.log('Element not found');
}

find()

Re-locate the element, optionally with a new description.

await element.find(newDescription)

Parameters:

newDescription (string, optional) - New description to search for

Returns: Promise<Element> - This element instance Example:

// Re-locate if the UI changed
const element = await testdriver.find('submit button');
// ... page updates ...
await element.find(); // Re-locate with same description

// Or update the description
await element.find('blue submit button'); // Now looking for blue button

click()

Click on the element.

await element.click(action)

Parameters:

action (string, optional) - Type of click: 'click' (default), 'double-click', 'right-click', 'hover', 'mouseDown', 'mouseUp'

Returns: Promise<void> Example:

const button = await testdriver.find('submit button');
await button.click(); // Regular click

const file = await testdriver.find('document.txt');
await file.click('double-click'); // Double-click

const menu = await testdriver.find('settings icon');
await menu.click('right-click'); // Right-click

The element must be found before clicking. The find() method automatically locates the element.

hover()

Hover over the element without clicking.

await element.hover()

Returns: Promise<void> Example:

const tooltip = await testdriver.find('info icon');
await tooltip.hover();
// Wait to see tooltip
await new Promise(resolve => setTimeout(resolve, 1000));

doubleClick()

Double-click on the element.

await element.doubleClick()

Returns: Promise<void> Example:

const file = await testdriver.find('README.txt file icon');
await file.doubleClick();

rightClick()

Right-click on the element to open context menu.

await element.rightClick()

Returns: Promise<void> Example:

const folder = await testdriver.find('Documents folder');
await folder.rightClick();

mouseDown() / mouseUp()

Press or release mouse button on the element (for drag operations).

await element.mouseDown()
await element.mouseUp()

Returns: Promise<void> Example:

// Drag and drop
const item = await testdriver.find('draggable item');
await item.mouseDown();

// Move to drop target (using coordinates or another element)
const target = await testdriver.find('drop zone');
await target.hover();
await target.mouseUp();

Properties

Element properties provide additional information about located elements. Properties are available after a successful find() call.

coordinates

Get the element’s coordinates object containing all position information.

const coords = element.getCoordinates()
// or access directly
element.coordinates

Returns: Object | null - Coordinate object with { x, y, centerX, centerY } Example:

const button = await testdriver.find('submit button');
const coords = button.coordinates;

if (coords) {
  console.log(`Top-left: (${coords.x}, ${coords.y})`);
  console.log(`Center: (${coords.centerX}, ${coords.centerY})`);
}

x, y, centerX, centerY

Direct access to coordinate values. Always available after successful find().

element.x        // Top-left X coordinate (number)
element.y        // Top-left Y coordinate (number)
element.centerX  // Center X coordinate (number)
element.centerY  // Center Y coordinate (number)

Example:

const button = await testdriver.find('submit button');
console.log(`Button at: (${button.x}, ${button.y})`);
console.log(`Button center: (${button.centerX}, ${button.centerY})`);

// Use for custom mouse operations
await testdriver.click(button.centerX, button.centerY);

width, height

Element dimensions in pixels. Available when AI detects element bounds.

element.width   // Width in pixels (number | null)
element.height  // Height in pixels (number | null)

Example:

const button = await testdriver.find('submit button');

if (button.width && button.height) {
  console.log(`Button size: ${button.width}x${button.height}px`);
  
  // Check if button is large enough
  if (button.width < 50) {
    console.warn('Button might be too small');
  }
}

boundingBox

Complete bounding box information including position and dimensions.

element.boundingBox

Returns: Object | null - Bounding box with all dimension data

{
  x: number,        // Top-left X
  y: number,        // Top-left Y
  width: number,    // Width in pixels
  height: number    // Height in pixels
}

Example:

const element = await testdriver.find('dialog box');

if (element.boundingBox) {
  const { x, y, width, height } = element.boundingBox;
  console.log(`Dialog: ${width}x${height} at (${x}, ${y})`);
  
  // Calculate if element is in viewport
  const rightEdge = x + width;
  const bottomEdge = y + height;
  console.log(`Element extends to (${rightEdge}, ${bottomEdge})`);
}

screenshot

Base64-encoded PNG screenshot of the screen when element was found. Only available in DEBUG mode or when an error occurs.

element.screenshot

Returns: string | null - Base64-encoded PNG image Example:

const element = await testdriver.find('error message');

if (element.screenshot) {
  // Save screenshot to file
  const fs = require('fs');
  const base64Data = element.screenshot.replace(/^data:image\/\w+;base64,/, '');
  fs.writeFileSync('element-screenshot.png', Buffer.from(base64Data, 'base64'));
  console.log('Screenshot saved');
}

Screenshots can be large. They’re automatically excluded from error messages to prevent memory issues.

text

Text content extracted from the element by AI (if available).

element.text

Returns: string | null - Element’s text content Example:

const message = await testdriver.find('notification message');

if (message.text) {
  console.log('Message says:', message.text);
  
  // Use text content in assertions
  if (message.text.includes('success')) {
    console.log('Success message detected');
  }
}

// Another example - extracting button label
const button = await testdriver.find('blue button');
console.log('Button text:', button.text); // "Submit"

label

Accessible label or name of the element (if available). Useful for verifying accessibility.

element.label

Returns: string | null - Accessible label Example:

const input = await testdriver.find('first input field');

if (input.label) {
  console.log('Input label:', input.label); // "Email Address"
}

confidence

AI confidence score for the element match (0-1, where 1 is perfect confidence).

element.confidence

Returns: number | null - Confidence score between 0 and 1 Example:

const element = await testdriver.find('submit button');

if (element.confidence !== null) {
  const percentage = (element.confidence * 100).toFixed(1);
  console.log(`Match confidence: ${percentage}%`);
  
  if (element.confidence < 0.8) {
    console.warn('⚠️ Low confidence match - element might not be correct');
  } else if (element.confidence > 0.95) {
    console.log('✅ High confidence match');
  }
}

Confidence scores below 0.8 may indicate the element description was ambiguous or the wrong element was found.

Property Availability

Property	When Available
`x`, `y`, `centerX`, `centerY`	✅ Always after successful `find()`
`coordinates`	✅ Always after successful `find()`
`width`, `height`	⚠️ When AI detects element bounds
`boundingBox`	⚠️ When AI detects element bounds
`text`	⚠️ When AI extracts text content
`label`	⚠️ When element has accessible label
`confidence`	✅ Always after AI element finding
`screenshot`	⚠️ Only in DEBUG mode or on errors

Properties marked with ⚠️ may be null depending on what the AI could detect from the screenshot.

JSON Serialization

Element objects can be safely serialized using JSON.stringify() for logging, debugging, and data storage. Circular references are automatically removed:

const element = await testdriver.find('login button');

// Safe to stringify - no circular reference errors!
console.log(JSON.stringify(element, null, 2));

Serialized output includes:

{
  "description": "login button",
  "coordinates": { "x": 100, "y": 200, "centerX": 150, "centerY": 225 },
  "found": true,
  "threshold": 0.01,
  "x": 100,
  "y": 200,
  "cache": {
    "hit": true,
    "strategy": "pixel-diff",
    "createdAt": "2025-12-09T10:30:00.000Z",
    "diffPercent": 0.0023,
    "imageUrl": "https://cache.testdriver.ai/..."
  },
  "similarity": 0.98,
  "confidence": 0.95,
  "selector": "button#login",
  "aiResponse": "Found the blue login button in the center of the form..."
}

Serialized properties:

Property	Type	Description
`description`	string	Element search description
`coordinates`	object	Full coordinate object `{x, y, centerX, centerY}`
`found`	boolean	Whether element was located
`threshold`	number	Cache threshold used for this find
`x`, `y`	number	Top-left coordinates
`cache.hit`	boolean	Whether cache was used
`cache.strategy`	string	Cache strategy (e.g., “pixel-diff”)
`cache.createdAt`	string	ISO timestamp when cache was created
`cache.diffPercent`	number	Pixel difference from cached image
`cache.imageUrl`	string	URL to cached screenshot
`similarity`	number	Similarity score (0-1)
`confidence`	number	AI confidence score (0-1)
`selector`	string	CSS/XPath selector if available
`aiResponse`	string	AI’s explanation of what it found

Use cases:

// Debugging element detection
const element = await testdriver.find('submit button');
if (!element.found()) {
  console.error('Element not found:', JSON.stringify(element, null, 2));
}

// Logging cache performance
const data = JSON.parse(JSON.stringify(element));
if (data.cache.hit) {
  console.log(`Cache hit! Diff: ${(data.cache.diffPercent * 100).toFixed(2)}%`);
}

// Sharing element data across processes
const elementData = JSON.stringify(element);
// Send to another process, log to file, etc.

Use JSON serialization when you need to log element data or when debugging why an element wasn’t found. The serialized output excludes large binary data (screenshots) and circular references.

Examples

Basic Element Interaction

// Find and click
const submitButton = await testdriver.find('submit button');
await submitButton.click();

// Find, verify, then interact
const emailInput = await testdriver.find('email input field');
if (emailInput.found()) {
  await emailInput.click();
  await testdriver.type('user@example.com');
}

Working with Forms

// Fill out a multi-field form
const nameField = await testdriver.find('name input field');
await nameField.click();
await testdriver.type('John Doe');

const emailField = await testdriver.find('email input field');
await emailField.click();
await testdriver.type('john@example.com');

const submitButton = await testdriver.find('submit button');
await submitButton.click();

Conditional Interactions

// Check if element exists before interacting
const closeButton = await testdriver.find('close popup button');

if (closeButton.found()) {
  await closeButton.click();
  console.log('Popup closed');
} else {
  console.log('No popup to close');
}

Re-locating Dynamic Elements

// Element that moves or changes
const notification = await testdriver.find('success notification');

// Do something that might cause it to move
await testdriver.scroll('down', 300);

// Re-locate the element
await notification.find();

if (notification.found()) {
  await notification.click();
}

Best Practices

Be specific with descriptions

Include visual details, position context, and nearby text:

// ❌ Too vague
await testdriver.find('button');

// ✅ Specific
await testdriver.find('blue submit button below the email field');

Check if element was found

Always verify elements were located before interacting:

const element = await testdriver.find('submit button');
if (!element.found()) {
  throw new Error('Submit button not found');
}
await element.click();

Reuse element references when possible

If you need to interact with the same element multiple times, reuse the reference:

const input = await testdriver.find('search input');
await input.click();
await testdriver.type('first search');
await testdriver.pressKeys(['enter']);

// Re-use the same element reference
await input.click();
await testdriver.pressKeys(['ctrl', 'a']); // Select all
await testdriver.type('second search');

​Overview

​Finding Elements

​find()

​Element Class

​Methods

​found()

​find()

​click()

​hover()

​doubleClick()

​rightClick()

​mouseDown() / mouseUp()

​Properties

​coordinates

​x, y, centerX, centerY

​width, height

​boundingBox

​screenshot

​text

​label

​confidence

​Property Availability

​JSON Serialization

​Examples

​Basic Element Interaction

​Working with Forms

​Conditional Interactions

​Re-locating Dynamic Elements

​Best Practices

Overview

Finding Elements

find()

Element Class

Methods

found()

find()

click()

hover()

doubleClick()

rightClick()

mouseDown() / mouseUp()

Properties

coordinates

x, y, centerX, centerY

width, height

boundingBox

screenshot

text

label

confidence

Property Availability

JSON Serialization

Examples

Basic Element Interaction

Working with Forms

Conditional Interactions

Re-locating Dynamic Elements

Best Practices