Self-Hosting TestDriver

​

Why self host?

• Enhanced security: Get complete control over ingress and egress rules.
• Complete customization: Modify the TestDriver Golden Image (our pre-configured AMI) to include custom dependencies, software, and configurations at launch time.
• Powerful Infrastructure: Run tests on bare metal infrastructure that support emulators and simulators.

​

Overview

• Spawn TestDriver instances on-demand in your AWS account
• Run tests on your own AWS infrastructure with custom configurations
• Integrate seamlessly with GitHub Actions CI/CD workflows
• Automatically clean up resources after test completion

1. CloudFormation Infrastructure: Deploy our setup/aws/cloudformation.yaml template to create the foundational AWS resources (VPC, security groups, IAM roles, and a launch template for instance creation).
2. On-Demand Instance Spawning: Use setup/aws/spawn-runner.sh with your Launch Template ID to programmatically spawn TestDriver instances whenever you need to run tests.
3. GitHub Actions Integration: Use .github/workflows/self-hosted.yml as a template for running tests in CI. This workflow demonstrates the complete lifecycle: spawning an instance, running tests, and shutting down the instance to minimize costs.

​

Prerequisites

• AWS account with permissions to run CloudFormation.
• AWS CLI installed locally.
  Be sure to run aws configure with your credentials
• Access to the TestDriver AMI (Golden Image)
  Contact us with your preferred AWS Region for access.
• A GitHub repository for committing your tests & workflow.

​

Step 1: Set Up AWS Infrastructure

​

Deploy CloudFormation Stack

• Dedicated VPC with public subnet
• Security group with proper port access
• IAM roles and instance profiles
• EC2 launch template for programmatic instance creation

# Deploy the CloudFormation stack
aws cloudformation deploy \
  --template-file setup/aws/cloudformation.yaml \
  --stack-name my-testdriver-infrastructure \
  --parameter-overrides \
    ProjectTag=testdriver \
    AllowedIngressCidr=0.0.0.0/0 \
    InstanceType=c5.xlarge \
    CreateKeyPair=true \
  --capabilities CAPABILITY_IAM

​

Get Launch Template ID

aws cloudformation describe-stacks \
  --stack-name my-testdriver-infrastructure \
  --query 'Stacks[0].Outputs[?OutputKey==`LaunchTemplateId`].OutputValue' \
  --output text

​

Step 2: Spawn a New TestDriver Runner

​

Using spawn-runner.sh

• Launches a new EC2 instance using your launch template from Step 1
• Waits for the instance to become ready
• Completes the TestDriver handshake
• Returns instance details (IP, instance ID) for CLI usage

# Launch an instance using environment variables
export AWS_REGION=us-east-2
export AMI_ID=ami-••••••••••  # Your TestDriver AMI (contact us to get one)
export AWS_LAUNCH_TEMPLATE_ID=lt-••••••••••  # From CloudFormation output from step 1
export RESOLUTION=1440x900 # Change screen resolution if desired (default is 1440x900)

/bin/bash ./setup/aws/spawn-runner.sh

PUBLIC_IP=1.2.3.4
INSTANCE_ID=i-1234567890abcdef0
AWS_REGION=us-east-2

​

Changing Resolution in Lifecycle Files

version: 6.0.0
steps:
  - prompt: set screen resolution
    commands:
      - command: exec
        lang: pwsh
        code: |
          C:\testdriver\SetResolution.ps1 -Width 1920 -Height 1080

​

CLI Usage

# Basic test execution
npx testdriverai@latest run test.yaml --ip=1.2.3.4

npx testdriverai@latest run testdriver/your-test.yaml \
  --ip="$PUBLIC_IP" \

​

Terminating Instances

# Terminate the instance
aws ec2 terminate-instances --instance-ids $INSTANCE_ID --region $AWS_REGION

​

Step 3: GitHub Actions Integration

​

Example Workflow

name: TestDriver Self-Hosted

on:
  workflow_dispatch:
  push:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Setup AWS Instance
        id: aws-setup
        run: |
          OUTPUT=$(./setup/aws/spawn-runner.sh | tee /dev/stderr)
          PUBLIC_IP=$(echo "$OUTPUT" | grep "PUBLIC_IP=" | cut -d'=' -f2)
          INSTANCE_ID=$(echo "$OUTPUT" | grep "INSTANCE_ID=" | cut -d'=' -f2)
          echo "public-ip=$PUBLIC_IP" >> $GITHUB_OUTPUT
          echo "instance-id=$INSTANCE_ID" >> $GITHUB_OUTPUT
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_REGION: ${{ secrets.AWS_REGION }}
          AWS_LAUNCH_TEMPLATE_ID: ${{ secrets.AWS_LAUNCH_TEMPLATE_ID }}
          AMI_ID: ${{ secrets.AMI_ID }}

      - name: Run TestDriver
        run: |
          npx testdriverai run your-test.yaml \
            --ip="${{ steps.aws-setup.outputs.public-ip }}"
        env:
          TD_API_KEY: ${{ secrets.TD_API_KEY }}

      - name: Shutdown AWS Instance
        if: always()
        run: |
          aws ec2 terminate-instances \
            --region ${{ secrets.AWS_REGION }}
            --instance-ids ${{ steps.aws-setup.outputs.instance-id }}
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
		  AWS_REGION: ${{ secrets.AWS_REGION }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

​

Required Secrets

​

AMI Customization

​

Using the Base AMI

• Windows Server with desktop environment
• VNC + web server for remote desktop access through the browser

• Python (with pip)
• Node.js (with npm)
• Git

• TestDriver agent and dependencies
• Optimized settings for test execution
• Pre-configured networking for TestDriver CLI communication

​

Modifying the AMI

1. Launch an instance from our base AMI
2. Make your changes (install software, configure settings)
3. Create a new AMI from your modified instance
4. Update your workflow to use the new AMI ID

​

Amazon Image Builder

# Example Image Builder pipeline
Components:
  - Name: testdriver-base
    Version: 1.0.0
    Platform: Windows
    Type: BUILD
    Data: |
      name: TestDriver Custom Setup
      description: Custom TestDriver AMI with additional software
      schemaVersion: 1.0
      phases:
        - name: build
          steps:
            - name: InstallSoftware
              action: ExecutePowerShell
              inputs:
                commands:
                  - "# Your custom installation commands here"

​

Security Considerations

​

Network Security

1. Restrict CIDR blocks: Only allow access from your known IP ranges
2. Use VPC endpoints: For private communication with AWS services
3. Enable VPC Flow Logs: For network monitoring and debugging

​

AWS Authentication

permissions:
  id-token: write
  contents: read

steps:
  - name: Configure AWS credentials
    uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsRole
      aws-region: us-east-2

​

Instance Security

• Terminate instances immediately after use
• Monitor costs with AWS billing alerts
• Use least-privilege IAM roles for instance profiles
• Enable CloudTrail for audit logging

​

Troubleshooting

​

Common Issues

• Security group rules: The CloudFormation template configures all necessary ports (RDP 3389, VNC 5900, and TestDriver communication ports). Verify your security group hasn’t been modified.
• Instance status checks: Ensure the instance has passed both system and instance status checks in the AWS console.
• AMI compatibility: Verify the AMI is compatible with your selected instance type (some instance types don’t support certain AMIs).

• Verify network connectivity from runner to instance
• Check VPC routing and internet gateway configuration
• Confirm instance is in correct subnet

• Validate AWS credentials and permissions
• Check AWS service quotas and limits
• Verify region consistency across all resources

​

Getting Help

• Contact your account manager for AMI access issues
• Use support channels for infrastructure questions
• Check the TestDriver documentation for CLI usage