Why automated testing is useful
When applications are developed code has to be tested. One can test the application manually by executing some operations and verifying their correctness. However, a more constructive procedure would be to develop test code that exercises (parts of) the application.
Automating the process of testing has a number of advantages which will be discussed in the next paragraphs.
Reproducibility
Obviously, by automating tests, test can be executed over and over again without effort. This means that effort into thinking out possible tricky parts of an application and creating a test procedure for that is not lost and can be used over and over again.
Keep test code that is developed anyway
Often, new code is already tested in early phases of development by test code before that code is turned into official production code. It is a waste to not keep this test code. By having a framework for automated testing we have a comfortable place where we can keep this test code without having it to be overwritten by the final production code.
Refactoring
By using an incremental development approach we heavily rely on refactoring the code design as we go. By changing code that has already been tested we introduce the possibility of new problems. Having a test suite in place will turn out very valuable when we refactor. Moreover, we should develop appropriate tests before doing any refactoring when initial tests are not yet available or do not exercise the functionality to a satisfying degree.
Tests serve as an example
By creating tests for a certain part of code we automatically document a large part of how the code is intended to be used and what its purpose is. Also, by creating tests we get a chance to play around with the code, getting to know it better, without having to use it directly in production code. This is especially useful when it is not clear how the code should exactly be used and what its supposed behavior is.
Reproducing bugs
Automated tests are also a valuable tool in reproducing bugs. Whenever a bug has been detected we can try to find out the minimal procedure for reproducing the bug by implementing the proper test that exposes the bug. This test code is useful both in removing the bug and preventing the bug from surfacing again.
Tests result in code with better abstractions
In order to be able to properly test a certain chunk of code, we should be able to isolate that code from the application it is used in, and exercise it individually. This means that the code should not be too entangled with the rest of the application. Code that has this property and can be easily tested is better code. Therefore, by developing test code we also induce favorable properties of the code that is being developed. These properties include cleaner interfaces, higher reusability and less complexity.
Code is tested more thoroughly and is more complete
Often a certain code component is developed for generic purposed but is exercised by its host application is a limited, specific way. Automated tests provide a means to exercise other functionality that is being developed when it is developed without needing the host application to test those other features. This also means that easy to develop features can be added right away without the need for the host application to use them.
How automated testing is used at Charta Software
The most basic test of the code base is performed on a dedicated machine by automatically checking out the latest version of the code base and try to build every project. If building a project fails with compilation errors developers are automatically notified of these. This automatic build is commonly referred to as the nightly build. The automatic build will be performed every night and the goal is to keep the code base without problems. Also warnings and hints will be logged in order to let the projects compile as clean as possible.
Next to checking the project for compilation problems, the code base of the Charta Platform also contains a testing framework based on the TUnitTest class. By creating derived classes and their instances one can add extra tests to the test base. Every test can be classified with a runtime indication. Tests that are classified as short will run always at application start up. Tests that are classified as long will only be executed during the automated build of the code base.
How to develop tests
Now we have made clear the importance and usefulness of automated tests we need a few guide lines on how to develop tests.
Test on the same level as level the code
When developing a new class or function one should test this functionality on the level of that class or function. This means that one can assume that the functionality the class or function depends on is already tested thoroughly by other tests.
Develop test code at the same time with the code to be tested
The perfect time to create test code is when you develop the code itself. By iteratively adding functionality and tests that exercise that functionality code and test code should grow easily. Problems with your code (or its interface) are caught early on preventing much of the need of debugging later on.
Provide good code coverage
Naturally, when developing tests it is best to try to achieve tests that exercise all parts of the code. If a certain else is never executed by test code it is not tested. This should be avoided. Furthermore, it is not necessary to test all possible inputs to a function when it is obvious that groups of the same input can be represented by a single test case. Be sure to check around boundary conditions to prevent most errors involved in limiting the test cases.
What to test?
In principle every piece of code should be tested through an automated test. However, often this ideal cannot be achieved that easily. Good candidates for automated tests are:
- Application critical code
- Scientific algorithms
- Third party code and DLLs