Skip to main content

​
Overview

TestDriver’s element finding system uses AI to locate elements on screen using natural language descriptions. The find() method returns an Element object that you can interact with.

​
Finding Elements

​
find()

Locate an element on screen using a natural language description. 
const element = await testdriver.find(description)
Parameters:
  • description (string) - Natural language description of the element to find
Returns: Promise<Element> - Element instance that has been located Example: 
// Find a button
const submitButton = await testdriver.find('the submit button');

// Find an input field with context
const emailField = await testdriver.find('email input field in the login form');

// Find an element by visual characteristics
const redButton = await testdriver.find('red button in the top right corner');
Be specific in your descriptions. Include visual details, location context, or nearby text to improve accuracy.

​
Element Class

The Element class represents a located (or to-be-located) UI element. It provides methods for interaction and properties for element information.

​
Methods

​
found()

Check if the element was successfully located. 
element.found()
Returns: boolean - True if element coordinates were found Example: 
const element = await testdriver.find('login button');
if (element.found()) {
  await element.click();
} else {
  console.log('Element not found');
}

​
find()

Re-locate the element, optionally with a new description. 
await element.find(newDescription)
Parameters:
  • newDescription (string, optional) - New description to search for
Returns: Promise<Element> - This element instance Example: 
// Re-locate if the UI changed
const element = await testdriver.find('submit button');
// ... page updates ...
await element.find(); // Re-locate with same description

// Or update the description
await element.find('blue submit button'); // Now looking for blue button

​
click()

Click on the element. 
await element.click(action)
Parameters:
  • action (string, optional) - Type of click: 'click' (default), 'double-click', 'right-click', 'hover', 'mouseDown', 'mouseUp'
Returns: Promise<void> Example: 
const button = await testdriver.find('submit button');
await button.click(); // Regular click

const file = await testdriver.find('document.txt');
await file.click('double-click'); // Double-click

const menu = await testdriver.find('settings icon');
await menu.click('right-click'); // Right-click
The element must be found before clicking. The find() method automatically locates the element.

​
hover()

Hover over the element without clicking. 
await element.hover()
Returns: Promise<void> Example: 
const tooltip = await testdriver.find('info icon');
await tooltip.hover();
// Wait to see tooltip
await new Promise(resolve => setTimeout(resolve, 1000));

​
doubleClick()

Double-click on the element. 
await element.doubleClick()
Returns: Promise<void> Example: 
const file = await testdriver.find('README.txt file icon');
await file.doubleClick();

​
rightClick()

Right-click on the element to open context menu. 
await element.rightClick()
Returns: Promise<void> Example: 
const folder = await testdriver.find('Documents folder');
await folder.rightClick();

​
mouseDown() / mouseUp()

Press or release mouse button on the element (for drag operations). 
await element.mouseDown()
await element.mouseUp()
Returns: Promise<void> Example: 
// Drag and drop
const item = await testdriver.find('draggable item');
await item.mouseDown();

// Move to drop target (using coordinates or another element)
const target = await testdriver.find('drop zone');
await target.hover();
await target.mouseUp();

​
Properties

Element properties provide additional information about located elements. Properties are available after a successful find() call.

​
coordinates

Get the element’s coordinates object containing all position information. 
const coords = element.getCoordinates()
// or access directly
element.coordinates
Returns: Object | null - Coordinate object with { x, y, centerX, centerY } Example: 
const button = await testdriver.find('submit button');
const coords = button.coordinates;

if (coords) {
  console.log(`Top-left: (${coords.x}, ${coords.y})`);
  console.log(`Center: (${coords.centerX}, ${coords.centerY})`);
}

​
x, y, centerX, centerY

Direct access to coordinate values. Always available after successful find(). 
element.x        // Top-left X coordinate (number)
element.y        // Top-left Y coordinate (number)
element.centerX  // Center X coordinate (number)
element.centerY  // Center Y coordinate (number)
Example: 
const button = await testdriver.find('submit button');
console.log(`Button at: (${button.x}, ${button.y})`);
console.log(`Button center: (${button.centerX}, ${button.centerY})`);

// Use for custom mouse operations
await testdriver.click(button.centerX, button.centerY);

​
width, height

Element dimensions in pixels. Available when AI detects element bounds. 
element.width   // Width in pixels (number | null)
element.height  // Height in pixels (number | null)
Example: 
const button = await testdriver.find('submit button');

if (button.width && button.height) {
  console.log(`Button size: ${button.width}x${button.height}px`);
  
  // Check if button is large enough
  if (button.width < 50) {
    console.warn('Button might be too small');
  }
}

​
boundingBox

Complete bounding box information including position and dimensions. 
element.boundingBox
Returns: Object | null - Bounding box with all dimension data 
{
  x: number,        // Top-left X
  y: number,        // Top-left Y
  width: number,    // Width in pixels
  height: number    // Height in pixels
}
Example: 
const element = await testdriver.find('dialog box');

if (element.boundingBox) {
  const { x, y, width, height } = element.boundingBox;
  console.log(`Dialog: ${width}x${height} at (${x}, ${y})`);
  
  // Calculate if element is in viewport
  const rightEdge = x + width;
  const bottomEdge = y + height;
  console.log(`Element extends to (${rightEdge}, ${bottomEdge})`);
}

​
screenshot

Base64-encoded PNG screenshot of the screen when element was found. Only available in DEBUG mode or when an error occurs. 
element.screenshot
Returns: string | null - Base64-encoded PNG image Example: 
const element = await testdriver.find('error message');

if (element.screenshot) {
  // Save screenshot to file
  const fs = require('fs');
  const base64Data = element.screenshot.replace(/^data:image\/\w+;base64,/, '');
  fs.writeFileSync('element-screenshot.png', Buffer.from(base64Data, 'base64'));
  console.log('Screenshot saved');
}
Screenshots can be large. They’re automatically excluded from error messages to prevent memory issues.

​
text

Text content extracted from the element by AI (if available). 
element.text
Returns: string | null - Element’s text content Example: 
const message = await testdriver.find('notification message');

if (message.text) {
  console.log('Message says:', message.text);
  
  // Use text content in assertions
  if (message.text.includes('success')) {
    console.log('Success message detected');
  }
}

// Another example - extracting button label
const button = await testdriver.find('blue button');
console.log('Button text:', button.text); // "Submit"

​
label

Accessible label or name of the element (if available). Useful for verifying accessibility. 
element.label
Returns: string | null - Accessible label Example: 
const input = await testdriver.find('first input field');

if (input.label) {
  console.log('Input label:', input.label); // "Email Address"
}

​
confidence

AI confidence score for the element match (0-1, where 1 is perfect confidence). 
element.confidence
Returns: number | null - Confidence score between 0 and 1 Example: 
const element = await testdriver.find('submit button');

if (element.confidence !== null) {
  const percentage = (element.confidence * 100).toFixed(1);
  console.log(`Match confidence: ${percentage}%`);
  
  if (element.confidence < 0.8) {
    console.warn('⚠️ Low confidence match - element might not be correct');
  } else if (element.confidence > 0.95) {
    console.log('✅ High confidence match');
  }
}
Confidence scores below 0.8 may indicate the element description was ambiguous or the wrong element was found.

​
Property Availability

PropertyWhen Available
x, y, centerX, centerY✅ Always after successful find()
coordinates✅ Always after successful find()
width, height⚠️ When AI detects element bounds
boundingBox⚠️ When AI detects element bounds
text⚠️ When AI extracts text content
label⚠️ When element has accessible label
confidence✅ Always after AI element finding
screenshot⚠️ Only in DEBUG mode or on errors
Properties marked with ⚠️ may be null depending on what the AI could detect from the screenshot.

​
Polling for Elements

Use polling to wait for elements that may not be immediately visible: 
// Poll until element appears
let loginButton;
const maxAttempts = 30;

for (let i = 0; i < maxAttempts; i++) {
  loginButton = await testdriver.find('login button');
  if (loginButton.found()) break;
  await new Promise(resolve => setTimeout(resolve, 1000));
}

if (loginButton.found()) {
  await loginButton.click();
} else {
  throw new Error('Login button never appeared');
}
Helper function for polling: 
async function waitForElement(testdriver, description, timeout = 30000) {
  const startTime = Date.now();
  
  while (Date.now() - startTime < timeout) {
    const element = await testdriver.find(description);
    if (element.found()) return element;
    await new Promise(resolve => setTimeout(resolve, 1000));
  }
  
  throw new Error(`Element "${description}" not found after ${timeout}ms`);
}

// Usage
const button = await waitForElement(testdriver, 'submit button', 10000);
await button.click();

​
Examples

​
Basic Element Interaction

// Find and click
const submitButton = await testdriver.find('submit button');
await submitButton.click();

// Find, verify, then interact
const emailInput = await testdriver.find('email input field');
if (emailInput.found()) {
  await emailInput.click();
  await testdriver.type('[email protected]');
}

​
Working with Forms

// Fill out a multi-field form
const nameField = await testdriver.find('name input field');
await nameField.click();
await testdriver.type('John Doe');

const emailField = await testdriver.find('email input field');
await emailField.click();
await testdriver.type('[email protected]');

const submitButton = await testdriver.find('submit button');
await submitButton.click();

​
Conditional Interactions

// Check if element exists before interacting
const closeButton = await testdriver.find('close popup button');

if (closeButton.found()) {
  await closeButton.click();
  console.log('Popup closed');
} else {
  console.log('No popup to close');
}

​
Re-locating Dynamic Elements

// Element that moves or changes
const notification = await testdriver.find('success notification');

// Do something that might cause it to move
await testdriver.scroll('down', 300);

// Re-locate the element
await notification.find();

if (notification.found()) {
  await notification.click();
}

​
Best Practices

Include visual details, position context, and nearby text:
// ❌ Too vague
await testdriver.find('button');

// ✅ Specific
await testdriver.find('blue submit button below the email field');
Always verify elements were located before interacting:
const element = await testdriver.find('submit button');
if (!element.found()) {
  throw new Error('Submit button not found');
}
await element.click();
Don’t assume elements are immediately visible:
let element;
for (let i = 0; i < 30; i++) {
  element = await testdriver.find('loading complete message');
  if (element.found()) break;
  await new Promise(resolve => setTimeout(resolve, 1000));
}
If you need to interact with the same element multiple times, reuse the reference:
const input = await testdriver.find('search input');
await input.click();
await testdriver.type('first search');
await testdriver.pressKeys(['enter']);

// Re-use the same element reference
await input.click();
await testdriver.pressKeys(['ctrl', 'a']); // Select all
await testdriver.type('second search');