Physical & Vision Agents - Robotic Arm, AI Vision & I/O
Automate touchscreen gestures, electrical fault injection, and AI-powered HMI display validation inside a single TestBot sequence.
OVERVIEW
Software protocol agents validate what the ECU communicates. Physical agents validate what the device physically does - how its display renders under real CAN signals, how its touchscreen responds to real finger pressure, how it behaves when its sensor resistance changes. These are the tests that prevent field failures that no protocol check can catch.
TestBot's physical and vision agents give you a 4-DoF SCARA robotic arm for touchscreen interaction and button actuation, AI Vision agents for instrument cluster and HMI display validation using NVIDIA Xavier AGX GPU inference, an I/O Module for digital/analog/PWM signal injection and fault simulation, a Decade Resistance Box for programmable resistance-based sensor fault injection, and a Panel Key Controller for reliable keypad actuation. All six agents integrate directly into your TestBot sequence - no separate HIL platform required.
Our Test Jigs & Hardware Fixtures
Purpose-built hardware jigs designed to automate physical interaction, signal injection, and visual inspection across embedded and industrial test setups.
SCARA Robot Test Jig
Automated mechanical interaction for physical UI testing, button presses, actuate and touch input validation on real hardware.
Decade Resistance Box Jig
Programmable resistance substitution jig for simulating sensor inputs, fault conditions, and threshold boundary testing on analog interfaces.
IO Module Test Jig
Configurable digital and analog Input & Output interface board for GPIO, signal injection, and hardware-in-the-loop test scenarios.
AI Vision Camera Jig
Vision-based inspection jig using AI-powered image analysis for display validation, LED state detection, and visual output verification.
MECHANICAL
Mechanical Agents
Panel Key Controller for button actuation and 3D Robotic Arm for touchscreen interaction and gesture automation.
ELECTRONICS
Electronics Agents
I/O Module for GPIO/PWM/analog control and Decade Resistance Box for programmable resistance fault injection.
AI VISION
AI Vision Agents
Camera-based and NVIDIA Xavier GPU-accelerated AI inference for instrument cluster and HMI display validation.
Physical and Vision Testing Capabilities
Agents that go beyond protocol validation to test what the device physically does.
Instrument Cluster & HMI Validation
The AI Vision Agent reads what the display actually shows - speed values, warning icons, widget states - using a trained AI model running on the test system. Pass/fail is based on what the human eye would see, not what the ECU sent. Camera module for standard displays; NVIDIA Xavier AGX for complex automotive clusters.
Touchscreen Gesture Automation
The Robotic Arm Agent programs precise tap, swipe, hold, and drag gestures on real touchscreens - with configurable speed, force, and end effector settings. Validate infotainment navigation, touchscreen calibration, and multi-touch interaction without a human operator.
Electrical Fault Injection
The I/O Module Agent drives digital outputs, reads digital inputs, generates PWM, and samples analog voltages. Simulate switch closures, relay states, voltage thresholds, and PWM duty cycles to inject the conditions that trigger fault handling in your firmware.
Resistance-Based Sensor Fault Simulation
The DRB Agent sets programmable resistance from 0.1Ω to 99kΩ. Simulate NTC/PTC sensor failures, open/short circuit conditions, and resistance tolerance boundaries to validate your device's fault detection and reporting behaviour.
Physical Button & Keypad Testing
The PKC Agent actuates physical buttons and keypad keys via the Panel Key Controller - repeatable, timestamped actuation without human hands. Validate debounce timing, long-press detection, and key combination handling in firmware.
Combined HIL Sequences
Chain AI Vision, Robotic Arm, I/O Module, DRB, and CAN agents in one HIL sequence. Inject a fault via I/O Module → validate the cluster warning light via AI Vision → confirm the DTC over CAN UDS - all in one automated run with ISO 26262-aligned evidence.
Typical Physical and Vision Test Scenarios with TestBot
| Test Scenario | Agents Used | Validated Outcome |
|---|---|---|
| Instrument Cluster Warning Light | AI Vision Agent + CAN / CAN FD Agent | Warning icon rendered within 200ms of CAN trigger, correct colour and position confirmed |
| Touchscreen Navigation Validation | Robotic Arm Agent + AI Vision Agent | All menu taps register correctly, screen transitions complete within 500ms, no ghost touches |
| Sensor Open-Circuit Fault | DRB Agent + UDS Client Agent | DTC stored within 100ms of resistance threshold, freeze frame captured correctly |
| Button Long-Press Detection | PKC Agent | Long-press triggers correct firmware action at 1.5s ±50ms, no false positives at 1.2s |
| I/O Module Relay Fault Injection | I/O Module Agent + CAN / CAN FD Agent | Relay-open condition detected, fault logged, device enters safe state within spec |
| Full HMI Regression (Nightly) | AI Vision + Robotic Arm + CAN + UDS | All cluster states, touch interactions, and diagnostic responses validated - HTML report with visual evidence generated |
Frequently Asked Questions - Physical & Vision Agents
The AI Vision Camera Module is a standalone calibrated camera for validating simpler LCD and LED displays - status indicators, segment displays, and basic HMI screens. The AI Vision Full Unit includes a NVIDIA Xavier AGX with onboard GPU inference for complex automotive instrument clusters, dynamic animations, and multi-widget validation at automotive grade. Both use trained AI models - no pixel-coordinate scripting.
RELATED PAGES
Ready to Add Physical Testing to Your Automation Suite?
Talk to our team about hardware configuration and FastStart onboarding.