4,000 CAN Test Cases Per Night: How an Automotive ECU Team Automated Their Regression Cycle

Before automation, the team was running 50 test cases per day across 2 hardware units. Three validation engineers. Four hours of bench work per day, per engineer. Release cycle: six weeks. Defect escape rate to integration: too high to publish.

Twelve months after deploying TestBot, the same team runs 4,000 test cases per night across 10 hardware units, with no engineer involvement during execution. Release cycle: three and a half weeks. Defects found at integration: a fraction of the previous rate.

Here is how that transition happened - the architecture, the tooling decisions, and the specific things that made the difference.

The Starting Point: What They Were Testing Manually

The device is a powertrain ECU - engine management, with CAN as the primary communication layer and UDS for diagnostics. The ECU has 47 CAN messages, 312 signals, 68 UDS DIDs, and a flash programming sequence that takes 8 minutes end-to-end.

Manual test coverage before automation:

• ~20% of CAN signals validated per release (the ones known to have caused problems before)
• ~15 UDS services tested manually (session management, a handful of DIDs, basic flash)
• Zero fault injection testing - too slow to set up manually
• Zero timing validation - engineers could not run bus load profiles manually
• Zero nightly regression - tests only ran before planned release milestones

The result: defects that had been present in the firmware for weeks were found at integration testing, at a point where root cause analysis required reconstructing builds from two weeks prior.

The Architecture: 10 Units, One Controller, One Pipeline

The team deployed 10 physical ECU units in a rack - each connected to its own CAN Adapter, its own UDS session, and its own TestBot station. A single Jenkins pipeline, triggered on every firmware build merge to the main branch, flashes all 10 units in parallel and then launches 10 simultaneous TestBot test suite executions.

The test suite is divided by ECU variant: the 10 units cover 4 hardware variants and 2 software configurations, with unit assignments managed by a JSON configuration file that the pipeline reads. Adding a new variant means updating the JSON - not changing the test suite.

Each TestBot station runs the same master test suite - approximately 400 test cases - against its assigned ECU. 10 units × 400 test cases = 4,000 test case executions per pipeline run. A complete run takes 3.5 hours, finishing before the engineering team arrives in the morning.

What the Test Suite Covers

CAN signal validation (180 test cases): All 312 signals in the DBC, grouped by message. Each test case sends a stimulus (via I/O Module Agent for sensor inputs, or direct CAN frames for network inputs) and asserts signal values, timing, and encoding. Edge cases - minimum, maximum, invalid range - are dataset rows, not separate test cases.

UDS diagnostic coverage (85 test cases): All 68 DIDs with read validation, the 12 writable DIDs with write-readback-reset cycles, SecurityAccess sequence (including wrong-key and lockout tests), DTC injection via I/O Module and read-back via 0x19, and the complete flash programming cycle (0x34–0x37 with checksum verification and post-flash boot confirmation).

Fault injection (60 test cases): The I/O Module Agent and DRB Agent inject electrical faults - open circuit sensors, out-of-range voltages, short circuits - and the test suite validates that the correct DTCs are stored, the correct MIL behaviour occurs, and the ECU recovers correctly when the fault is cleared.

Timing and bus load (45 test cases): Message periodicity validation across a simulated 70% bus load window (CAN Send agent generates background traffic). Startup burst measurement. Missing message detection after software state transitions.

Regression baseline comparison (30 test cases): Signal values, DID values, and timing parameters are compared against a stored baseline from the last known-good build. Any deviation outside the tolerance band fails the test and triggers a regression alert.

The Three Decisions That Made It Work

Decision 1: Dataset-driven test cases. Instead of writing one test case per signal value, the team defined datasets for each test case covering nominal, boundary, and fault values. 312 signals in 47 messages became 180 test cases - each running across 3–5 dataset rows. Maintenance burden: change the DBC, regenerate the dataset from a script, done.

Decision 2: Parallel hardware, single suite. The instinct was to have separate test suites per variant. The right choice was one suite with a variant configuration file. One suite to maintain, one suite to review, one suite that every engineer understands. Variant differences are expressed in the configuration, not in the test logic.

Decision 3: Hard failure gates, soft comparison alerts. The pipeline blocks the next stage on any hard failure (missing DTC, wrong flash result, failed UDS service). It sends an alert but does not block on soft comparison differences (signal value within 5% of baseline). This prevents the team from ignoring alerts because there are too many.

The Numbers That Justified the Investment

The team's validation engineers now spend their time on test design, failure analysis, and certification evidence - not on running test cases. The 35% reduction in release cycle time came from two places: fewer defects reaching integration (less rework time) and parallel hardware execution (less wall-clock test time).

The ROI calculation was straightforward: three validation engineers at bench for four hours per day, six days per week, 48 weeks per year. That is 3,456 engineer-hours per year of test execution time. At the team's loaded cost rate, the annual savings from automation exceeded the entire TestBot investment in the first three months.

The 4,000 test cases per night number is not special - it is what happens when you deploy 10 units and run 400 test cases per unit. The point is that the number is arbitrary. The architecture scales: add more units, run more test cases, increase coverage. The engineering team stays the same size.