TDD for embedded development

I recently attended an interesting talk on test driven development (TDD)
for embedded development given by Zulke UK for the  BCS SPA specialist group.

Robotshop Rover

The talk was the result of some experiments using an Arduino board, a bluetooth interface and an android tablet. The embedded platform was a Robotshop Rover with a number of sensors and controllers. The sensors were used to guide the robot along a track indicated by a solid black line; the motors were used to control the direction of the robot and also its speed.

Although the standard arduino development environment isn't a full IDE and is limited in its functionality, does come with some good code
examples. An alternative environment is the Eclipse CDT with the AVR plugin added to handle the download of the arduino image to the target platform. To support the development using TDD, CPPUTEST was used as the test framework. CPPUTEST is recommended as a framework suitable for embedded development (see James Greening book on TDD for Embedded Development) and appeared to the presenters to be more effective than CPPUNIT. It was noted by members of the audience that there are few tools which have good integration with the continuous integration platforms such as Jenkins.

An overview of TDD was given, and its application for an embedded target environment

1/ TDD needs to be able to test on both the development and target environments. This requires that two projects are created, one with arduino as the target environment and an other targeted at an x86 environment

2/ The cycle of test->code->refactor needs to be followed with the tests being chosen from a backlog. Code shouldn't be written unless it is to satisfy an existing test.

3/ The cycle should be choose test->write test->run-test->fail! If it doesn't fail (and it could be due to compiler or link failure), then it normally indicates that the test has been incorrectly written.

4/ Limited design is required although normal good software engineering practice should still be adopted (no monolithic functions etc).

5/ Mock interfaces should be used to unit test sensor interfaces. This allows the logic to be tested and debugged first before loading on to the target.

6/ If a state machine is required, some design is essential before any test cases can be identified and written.

The exercise in TDD with the arduino target resulted in no logic errors once the code was exercised on the target. However, the behaviour of the robot, in particular the speed of the motors required some further development and enhancement to the codebase. Given the fact that arduino boards are typically focused on the school market, I was surprised that C++ was the chosen development language. However, the C++ used on the arduino is a cut down version which removes many of the complexities of the full C++ language.

Whilst the associated Android development (on a Motorola XOOM tablet) appeared to be successful in terms of developing a simple user interface to send commends via Bluetooth to control the robot (e.g. stop, start,...), the development of the application revealed some shortcomings. Although the Android development kit works very well with Eclipse (the code is a mixture of Java and xml) and allows on target debugging, the use of TDD is less appropriate given the extensive use of callbacks (e.g.onclick, ....). The testing of GUI applications cannot be adequately tested within a development environment; fortunately the android debugger is excellent at pinpointing issues (typically null pointer exceptions). Android emulators can help to a limited extent but are not a sufficient replacement to an actual device. Android development for tablets is just evolving as the platform moves from a phone, with relatively simple applications, to potentially far more complex applications. The launch of Android 4.0 development kit (aka Ice-Cream sandwich) will clearly accelerate the development of more complex applications which will necessitate the employment of sound software engineering principles in delivering quality products.

In summary, the session demonstrated successfully that TDD could be applied in an embedded environment and that through the use of appropriate open source tools, software development for the expanding android market can follow tried and tested techniques.

Automated Testing Experience

Keith Braithwaite presented his recent experience of automated testing, particularly using Test Driven Development, at SPA 2008 Conference.

The audience, a mixture of youth and experience, had already some good experience of automated testing tools such as Xunit, Selenium, Cruise Control, jMock, dbUnit and Build-o-Matic (funny how all of these tools are open source).

Keith outlined some of the problems that he saw with manual testing such as:

• Takes a long time...


• Not consistently applied


• Not particularly effective


• Requires scheduling of people

Keith advocated automated testing as a better way with a clear view that 'Checked Examples' need to be performed automatically. Before describing his approach he went back to the basis of software engineering with regards how the system requirements were specified -natural language which is often vague and ambiguous. He advocated that unless the customer talked the same language as the engineers, then there would always be problems in building systems. One of the major problems in systems are specifying the boundaries or constraints of the system to be developed; writing these rules precisely was often very difficult with multiple exceptions to the norm. However, the customer could often give examples of 'what the system should do' - these weren't the rules but were sufficient for the reference results to be manually calculated by hand. There was no guarantee that the reference results were correct but by ensuring that there were multiple contributors to the production of this data, the probability of erroneous data was reduced.

With reference data available, a test framework was developed using the FIT library which was then used to automatically compare the results from the system with the reference results. At the end there were several hundred scenarios (or examples) used, with the test data being available before the features were developed, thereby facilitating the incremental development. Keith stated that the approach using reference data and automatic comparison of the system against the reference data detected problems early and also revealed issues with the design (if it couldn't be instrumented, then the design was questionable).

Keith stated that the delivered system was defect-free with no failures being reported.

Is Defect-Free possible?

• Developers make mistakes and errors, ...so


• The system contains defects,... so


• The user experiences failures

So, if the user doesn't experience failures, does this mean the system is without defects? Probably not, but this is hard to verify. I would agree with this by saying that it is very difficult to claim that any development is defect free, only that the defects haven't revealed themselves during the testing and normal (and presumably intended) operation of the system. It also depends on the impact that the failure has, as clearly some systems have a more severe impact than others. By adopting Keith's approach of developing reference data with the customer, the potential for misunderstanding the system before delivery is reduced.

Keith summarised by saying that the tests weren't really tests at all, merely gauges of how the system is working for the users. The reference data were 'checked examples' which were a very powerful way of ensuring the customer and developers worked together.