Project Overview
During firmware development for one of our projects, we used mutation testing to check whether our automated tests could catch errors, running the firmware in Renode before the physical board was available. Renode lets developers run compiled firmware against a simulated microcontroller on a computer.
Our automated tests checked application behavior, including motion detection. We had deliberately introduced faults into working code and confirmed that the tests detected them, which gave us confidence in their ability to catch those errors.
We then tried the same approach with a required hardware initialization step.
Challenge
The target microcontroller’s manual specified that the GPIO peripheral clock needed to be enabled for the peripheral to operate as intended. GPIO pins connect the microcontroller to external signals, such as sensor inputs.
We wanted to find out whether our tests would detect a missing clock-enable instruction.
This mattered because a passing emulation test does not necessarily confirm that every hardware requirement has been met. If a required dependency is not enforced in the simulation, the tests could pass while the firmware would still fail on the actual board.
Approach
Removing the Clock-Enable Instruction
We deliberately removed the GPIO clock-enable instruction and reran the automated tests, expecting the relevant checks to fail.
Instead, all seven tests in that run passed. The simulated motion-detection behavior continued working, and the test output gave no warning about the missing instruction.
Figure: All seven tests passed with the GPIO clock-enable instruction removed
Understanding the Testing Gap
The result indicated that our Renode setup did not enforce this clock-enable dependency. The tested GPIO behavior continued to work even though the instruction required by the target chip was missing.
The tests had detected previously introduced faults, so those earlier results remained useful. However, this experiment showed that they did not cover this particular hardware requirement.
The physical board was not available at the time. The expected failure on the target chip was therefore based on its manual, rather than a test on actual hardware.
Improving the Verification Process
We continued using emulation and created a separate list of checks to complete once the board arrived. The hardware bring-up checklist includes confirming that the required GPIO peripheral clocks are enabled on the real board.
Requirements that could not be confirmed through our emulation tests were recorded for physical hardware verification. This helped us distinguish between what we had already tested and what still needed to be checked on the board.
Outcome
We identified a gap in our emulation coverage: all seven tests passed despite a required GPIO clock-enable instruction being missing.
The finding changed how we tracked verification work. We created a separate list of checks requiring physical hardware, keeping those requirements visible alongside the automated test results.
If you’re developing an embedded device and need support with firmware testing, we’d be happy to help. Feel free to Contact Us for embedded firmware development, automated testing, and hardware integration support.
