Skip to end of metadata
Go to start of metadata

This page provides some hints for using Groovy to assist generate test data in particular all combinations and all pair combinations.

Frequently you may have to test combinations, e.g. a method has several enumerations for its arguments or a web page has several dropdowns with multiple values, or you have assorted hardware combinations (as in the example below). You could manually work out the combinations, or you can let Groovy help you. We are going to look at two ways groovy can help you:

  • By generating your test cases in Groovy
  • By reading and invoking XML data produced by a specialist tool called Whitch

The most (effective) way to test large numbers of combinations is called "all pairs".

Combinations algorithms in Groovy

Here is a script which calculates all combinations for a particular example:

Running this script yields the following results:

Note: you would normally invoke your test method rather than just printing out the test case as we have done here.

You could then use this information as the input for a data-driven test.

It turns out though, that running all of these combinations is often overkill. If for instance, some bug occurs when memory is low on Windows, then both of the following test cases will illustrate the bug:

A technique known as all pairs or orthogonal array testing suggests using just a subset of the input data combinations with high likelihood of finding all bugs and greatly reduced test execution time.

To calculate all pairs for the above example, you could use the following script:

This code is not optimised. It builds all combinations and then removes unneeded pairs. We could greatly reduce the amount of code by restructuring our all combinations example and then calling that - but we wanted to make each example standalone. Here is the result of running this script:

We saved half of the combinations. This might not seem like much but when the number of input items is large or the number of alternatives for each input data is large, the saving can be substantial.

If this is still too many combinations, we have one more additional algorithm that is sometimes useful. We call it minimal pairs. Rather than making sure each possible pair combination is covered, minimal pairs only adds a new test data item into the results if it introduces a new pair combination not previously seen. This doesn't guarantee all pairs are covered but tends to produce a minimal selection of interesting pairs across the possible combinations. Here is the script:

Here is the result of running the script:

Combinations using Whitch

The IBM alphaworks site hosts the Intelligent Test Case Handler (WHITCH) project. From their website: This technology is an Eclipse plug-in for generation and manipulation of test input data or configurations. It can be used to minimize the amount of testing while maintaining complete coverage of interacting variables. Intelligent Test Case Handler enables the user to generate small test suites with strong coverage properties, choose regression suites, and perform other useful operations for the creation of systematic software test plans.

You should follow the instructions on the WHITCH site for installing the plug-in into your Eclipse directory (we used Eclipse 3.2). The user guide is then part of Eclipse help and details the features and instructions for using the tool. We will simply highlight how you might decide to use it.

First create a new Whitch file. File -> New -> Other... -> Whitch file. You must select the project and provide a file name. We used 'groovy.whitch'.

Now add your types similar to the earlier example. You should end up with something like:

Now add some attributes corresponding to the types you have just added. The result will be something like:

Now select Whitch -> Build. Type in a test suite name. We used 'AllPairs' and select
an interaction level of 2 (for pairs) as follows:

Now click 'Build Test Suite' to obtain your results:

Now save your Whitch file. The result will be that the test cases are now stored into an XML file.

Note: We have not shown any advanced Whitch features, e.g. it lets you add test cases in the Include tab which must always be added into the test suite and test cases which are not possible into the Exclude tab. It also lets you try to reduce your test case size, give weightings, choose different algorithms for test case generation and more. See the Whitch user guide for more details.

Running this script yields the following results:

Note: the value "DC.DC" indicates a don't care value and can be replaced with any value for that field.

  • No labels