Embedded software testing presents particular challenges that don't arise when testing software destined to run on a desktop PC. The challenges vary depending on the type of testing being carried out, but generally relate to the physical environment of the embedded system in one way or another.
This article discusses the challenges, explains why it is important to find bugs as early as possible, and goes on to explain how Embunit addresses the challenges of embedded software testing.
Unit testing tests a module of code in isolation from the rest of the system. The module typically comprises a class, a single function, or perhaps a group of related functions. Tests are written that exercise the module according to its specification. In order to build the unit test, other parts of the system have to be simulated. This is normally done by writing stub functions or using a mocking framework. Building and running the unit test should be automated to make it easier to perform regression testing.
At the unit testing level the first challenge of embedded software testing concerns the physical I/O. All embedded systems interact with their environment in some way via their I/O, and this can be difficult to simulate using the methods described above. Fortunately many IDEs provide a simulated hardware environment that allows you to run your code on the PC, and some can be scripted to provide the necessary I/O transitions. Nevertheless, in certain cases there is no proper substitute for running on real hardware.
The second issue that affects many embedded systems is one of resources. Unit testing and mocking frameworks designed for developing PC applications have access to virtually unlimited memory. By contrast, embedded systems generally have limited amounts of memory in order to minimise size and power consumption. This may be accompanied by restrictions on the language facilities that can be used. For example, some C++ compilers provide the facility to disable exception handling.
The final challenge relates to automating the unit tests. It is generally quite straightforward to build unit tests automatically, because most IDEs provide a command line interface for building executables. However, it is not always so easy to automate the process of loading and running a test, or to retrieve the results. Not all IDEs have support for scripting, and even those that do are not always well documented.
See below to find out how Embunit addresses the challenges of embedded software testing.
Integration testing follows unit testing. Two or more modules are brought together and their combined functionality is tested. At this level, the challenges of embedded software testing are much the same as with unit testing. As more modules are combined together there is less need to stub out software functionality, but the hardware issues remain.
For small embedded systems this type of integration testing may lead directly to system testing. Larger systems however are composed of a number of smaller sub-systems. In these cases integration testing continues with two or more sub-systems, with the rest of the system being simulated in some way. This is typically achieved by injecting pre-defined test data into the sub-system, often under the control of a desktop PC.
In some respects embedded software testing at the system level should be simpler than (for example) unit testing, because you have the complete system to play with. However, that brings its own challenges if the system is particularly large, is distributed across a large geographic area, or is designed to operate in a harsh environment.
Such systems are often subject to a factory acceptance test (FAT) prior to delivery, where the relevant conditions are simulated as far as possible. Following installation the system is commissioned, and the commissioning phase usually concludes with a site acceptance test (SAT). Once commissioned, restrictions may be placed on the amount of time the system can be taken out of service while upgrades are installed and tested.
The importance of finding bugs early in the testing process
As each stage progresses the cost of fixing any bugs that are discovered increases substantially. Therefore it is important to find bugs as early in the process as possible, when they are relatively cheap to fix. Effective unit testing can help you achieve this.
Embunit addresses the challenges of embedded software testing at the unit testing and integration testing levels.
Unit tests created using Embunit employ a minimal set of language features, and the reporting of test results, and other test execution information is completely configurable for each hardware environment. This makes it possible to create tests that run on the smallest embedded systems, built using the most restricted compilers.
Embunit makes the test case and test step values available to stub code, and also has a user-defined 'context' variable that can simplify the job of writing and maintaining stub code.
With Embunit, the process of creating unit tests is separate from the process of loading and running them. You have to do the scripting yourself, but you are not locked into a particular sub-set of tools. If you decide to change processor on an existing product, or develop a new product using a different tool set, you don't need to worry about whether or not Embunit supports it.
And if your IDE does not support scripting you can still use Embunit to create your unit tests and run them manually. Results can be reported in many different ways, depending on what I/O is available on the hardware.
Download Embunit today and evaluate it for 30 days using the built in evaluation key. The first time you run Embunit just click the button that says "I want to evaluate the software" in the license management dialog box.
Need longer to evaluate?
No problem, contact us and we will send you a new evaluation key.
Attention all students!
See the FAQ page to find out how to get Embunit for free.