Complete Automation of Unit Testing

Program testing is expensive and labor-intensive, often consuming more than half of the total development costs, and yet it is frequently not done well and the results are not always satisfactory. However, testing is the primary method to ensure that programs comply with requirements. The aim of this work is to completely automate unit testing of object-oriented programs. We investigate the use of an evolutionary approach, such as genetic algorithms, for test data generation and the use of program specifications, written in JML, for test result determination. A prototype tool will be developed to show that a complete automation is not only feasible but also effective for unit testing Java programs. Automated testing techniques such as ours can complement manual testing by testing significant portion of object-oriented programs, as methods in object-oriented programs tend to be small; manual testing can focus more interesting problems, e.g., inter-class testing.

The ultimate goal of this work is to completely automate unit testing of object-oriented program. A programmer should be able to perform unit testing by a single click of button or a single command execution.

JML is used both as a tool for describing test oracles and as a basis for generating test data. Our approach uses specifications as test oracles, and JML is used to write such specifications. Each class to be tested is assumed to be annotated with JML assertions, such as pre and postconditions and class invariants that describe the behavior of the class. The JML's runtime assertion checker is used to detect assertions violations at run-time and to interpret them as either test success or failure. For this, a JUnit test class called a test oracle class is generated automatically.

In addition to the engineering challenge of extending, adapting, and integrating different components, such as JML, JUnit, and genetic algorithms, the key research component of our work is to advance the genetic algorithm techniques to generate test data for object-oriented programs. In particular, our study focuses on answering the following questions.

1. How to encode objects and values as chromosomes and genes? An efficient way need be defined to represent test data as chromosomes. A chromosome is a sequence of genes---e.g., a receiver and arguments for a method call. In object-oriented programs, a gene may be an object or a primitive value. For an object gene, its state may be described as a sequence of statements that need be executed to bring the object to the state of interest. However, the fact that object's state is encapsulated creates a fundamental difficulty. It is not trivial and often impossible to automatically create an object of the desired state.
2. What genetic operations? We need to define genetic operations and an evolutionary process to create a better set of chromosomes. The well-known genetic operations such as cross-over and mutation may be employed, but care must be taken to ensure that the test data matches the signature of the method being tested. It is not clear, however, whether manipulating objects as bit patterns would be effective.
3. How to determine fitness of test data? We need to define the fitness of individual test data and the test set as a whole. Our plan is to study both the specification-based coverage and the program-based coverage. In particular, condition coverage on the postcondition seems promising, though we are still evaluating its effectiveness. In terms of tool support, we need to generate coverage information from specifications or source code, by extending the JML's runtime assertion checker or instrumenting source code.