Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 5.3

This page provides

Excerpt

some hints for using Groovy to assist generate test data in particular all combinations and all pair combinations

.

Frequently you may have to test combinations, e.g. a method has several enumerations for its arguments or a web page has several dropdowns with multiple values, or you have assorted hardware combinations (as in the example below). You could manually work out the combinations, or you can let Groovy help you. We are going to look at two ways groovy can help you:

  • By generating your test cases in Groovy
  • By reading and invoking XML data produced by a specialist tool called Whitch

The most (effective) way to test large numbers of combinations is called "all pairs".

Combinations algorithms in Groovy

Here is a script which calculates all combinations for a particular example:

Code Block
results = new HashSet()
def buildCombinations(Map partialCombinations, inputsLeft) {
    def first = inputsLeft.entrySet().toList().get(0)
    def partialResults = [ ]
    first.value.each{
        def next = [(first.key):it]
        next.putAll(partialCombinations)
        partialResults << next
    }
    if (inputsLeft.size() == 1) {
        results.addAll(partialResults)
    } else {
        partialResults.each{
            rest = inputsLeft.clone()
            rest.remove(first.key)
            buildCombinations(it, rest)
        }
    }
}

def configurations = [memory:['256M', '512M', '1G', '2G'],
                      disk:['5G', '10G'],
                      os:['MacOS', 'Windows', 'Linux']]

buildCombinations([:], configurations)
println results.size() + " combinations:"
results.each{ println it }

Running this script yields the following results:

Code Block
24 combinations:
["memory":"512M", "os":"MacOS", "disk":"5G"]
["memory":"2G", "os":"Linux", "disk":"5G"]
["memory":"1G", "os":"Linux", "disk":"10G"]
["memory":"512M", "os":"Linux", "disk":"5G"]
["memory":"512M", "os":"Windows", "disk":"10G"]
["memory":"2G", "os":"MacOS", "disk":"10G"]
["memory":"1G", "os":"Windows", "disk":"5G"]
["memory":"256M", "os":"MacOS", "disk":"5G"]
["memory":"1G", "os":"Linux", "disk":"5G"]
["memory":"2G", "os":"Windows", "disk":"10G"]
["memory":"512M", "os":"MacOS", "disk":"10G"]
["memory":"256M", "os":"Windows", "disk":"5G"]
["memory":"256M", "os":"Windows", "disk":"10G"]
["memory":"1G", "os":"MacOS", "disk":"10G"]
["memory":"1G", "os":"Windows", "disk":"10G"]
["memory":"512M", "os":"Windows", "disk":"5G"]
["memory":"256M", "os":"Linux", "disk":"10G"]
["memory":"2G", "os":"Windows", "disk":"5G"]
["memory":"2G", "os":"MacOS", "disk":"5G"]
["memory":"2G", "os":"Linux", "disk":"10G"]
["memory":"256M", "os":"Linux", "disk":"5G"]
["memory":"1G", "os":"MacOS", "disk":"5G"]
["memory":"256M", "os":"MacOS", "disk":"10G"]
["memory":"512M", "os":"Linux", "disk":"10G"]

Note: you would normally invoke your test method rather than just printing out the test case as we have done here.

You could then use this information as the input for a data-driven test.

It turns out though, that running all of these combinations is often overkill. If for instance, some bug occurs when memory is low on Windows, then both of the following test cases will illustrate the bug:

Code Block
["memory":"256M", "os":"Windows", "disk":"5G"]
["memory":"256M", "os":"Windows", "disk":"10G"]

A technique known as all pairs or orthogonal array testing suggests using just a subset of the input data combinations with high likelihood of finding all bugs and greatly reduced test execution time.

To calculate all pairs for the above example, you could use the following script:

Code Block
initialResults = new HashSet()
results = new HashSet()

def buildPairs(Map partialCombinations, inputsLeft) {
    def first = getFirstEntry(inputsLeft)
    def partialResults = [ ]
    first.value.each{
        def next = [(first.key):it]
        def nextEntry = getFirstEntry(next)
        next.putAll(partialCombinations)
        partialResults << next
    }
    if (inputsLeft.size() == 1) {
        initialResults.addAll(partialResults)
    } else {
        partialResults.each{
            rest = inputsLeft.clone()
            rest.remove(first.key)
            buildPairs(it, rest)
        }
    }
}

def adjustPairs() {
    results = initialResults.clone()
    initialResults.each {
        def rest = results.clone()
        rest.remove(it)
        if (allPairsCovered(it, rest)) {
            results.remove(it)
        }
    }
}

def getFirstEntry(Map map) {
    return map.entrySet().toList().get(0)
}

def getAllPairsFromMap(map) {
    if (map.size() <= 1) return null
    def allPairs = new HashSet()
    def first = getFirstEntry(map)
    def rest = map.clone()
    rest.remove(first.key)
    rest.each{
        def nextPair = new HashSet()
        nextPair << first
        nextPair << it
        allPairs << nextPair
    }
    def restPairs = getAllPairsFromMap(rest)
    if (restPairs != null) {
        allPairs.addAll(restPairs)
    }
    return allPairs
}

boolean allPairsCovered(candidate, remaining) {
    def totalCount = 0
    def pairCombos = getAllPairsFromMap(candidate)
    pairCombos.each { candidatePair ->
        def pairFound = false
        def pairs = candidatePair.toList()
        for (it in remaining) {
            def entries = it.entrySet()
            if (!pairFound && entries.contains(pairs[0]) && entries.contains(pairs[1])) {
                pairFound = true
                totalCount++
            }
        }
    }
    return (totalCount == pairCombos.size())
}

def updateUsedPairs(map) {
    getAllPairsFromMap(map).each{ usedPairs << it }
}

def configurations = [memory:['256M', '512M', '1G', '2G'],
                      disk:['5G', '10G'],
                      os:['MacOS', 'Windows', 'Linux']]

buildPairs([:], configurations)
adjustPairs()
println results.size() + " pairs:"
results.each{ println it }

This code is not optimised. It builds all combinations and then removes unneeded pairs. We could greatly reduce the amount of code by restructuring our all combinations example and then calling that - but we wanted to make each example standalone. Here is the result of running this script:

Code Block
12 pairs:
["memory":"1G", "os":"Linux", "disk":"5G"]
["memory":"512M", "os":"MacOS", "disk":"10G"]
["memory":"256M", "os":"Windows", "disk":"10G"]
["memory":"1G", "os":"Windows", "disk":"10G"]
["memory":"512M", "os":"Windows", "disk":"5G"]
["memory":"2G", "os":"MacOS", "disk":"5G"]
["memory":"2G", "os":"Windows", "disk":"5G"]
["memory":"2G", "os":"Linux", "disk":"10G"]
["memory":"1G", "os":"MacOS", "disk":"5G"]
["memory":"256M", "os":"Linux", "disk":"5G"]
["memory":"512M", "os":"Linux", "disk":"10G"]
["memory":"256M", "os":"MacOS", "disk":"10G"]

We saved half of the combinations. This might not seem like much but when the number of input items is large or the number of alternatives for each input data is large, the saving can be substantial.

If this is still too many combinations, we have one more additional algorithm that is sometimes useful. We call it minimal pairs. Rather than making sure each possible pair combination is covered, minimal pairs only adds a new test data item into the results if it introduces a new pair combination not previously seen. This doesn't guarantee all pairs are covered but tends to produce a minimal selection of interesting pairs across the possible combinations. Here is the script:

Code Block
results = new HashSet()
usedPairs = new HashSet()

def buildUniquePairs(Map partialCombinations, inputsLeft) {
    def first = getFirstEntry(inputsLeft)
    def partialResults = [ ]
    first.value.each{
        def next = [(first.key):it]
        def nextEntry = getFirstEntry(next)
        next.putAll(partialCombinations)
        partialResults << next
    }
    if (inputsLeft.size() == 1) {
        partialResults.each {
            if (!containsUsedPairs(it)) {
                updateUsedPairs(it)
                results << it
            }
        }
    } else {
        partialResults.each{
            rest = inputsLeft.clone()
            rest.remove(first.key)
            buildUniquePairs(it, rest)
        }
    }
}

def getFirstEntry(map) {
    return map.entrySet().toList().get(0)
}

def getAllPairsFromMap(map) {
    if (map.size() <= 1) return null
    def allPairs = new HashSet()
    def first = getFirstEntry(map)
    def rest = map.clone()
    rest.remove(first.key)
    rest.each{
        def nextPair = new HashSet()
        nextPair << first
        nextPair << it
        allPairs << nextPair
    }
    def restPairs = getAllPairsFromMap(rest)
    if (restPairs != null) {
        allPairs.addAll(restPairs)
    }
    return allPairs
}

boolean containsUsedPairs(map) {
    if (map.size() <= 1) return false
    def unpaired = true
    getAllPairsFromMap(map).each {
        if (unpaired) {
            if (usedPairs.contains(it)) unpaired = false
        }
    }
    return !unpaired
}

def updateUsedPairs(map) {
    getAllPairsFromMap(map).each{ usedPairs << it }
}

def configurations = [memory:['256M', '512M', '1G', '2G'],
                      disk:['5G', '10G'],
                      os:['MacOS', 'Windows', 'Linux']]

buildUniquePairs([:], configurations)
println results.size() + " pairs:"
results.each{ println it }

Here is the result of running the script:

Code Block
6 pairs:
["memory":"512M", "os":"Windows", "disk":"5G"]
["memory":"256M", "os":"MacOS", "disk":"5G"]
["memory":"2G", "os":"Linux", "disk":"10G"]
["memory":"1G", "os":"Linux", "disk":"5G"]
["memory":"512M", "os":"MacOS", "disk":"10G"]
["memory":"256M", "os":"Windows", "disk":"10G"]

Combinations using Whitch

The IBM alphaworks site hosts the Intelligent Test Case Handler (WHITCH) project. From their website: This technology is an Eclipse plug-in for generation and manipulation of test input data or configurations. It can be used to minimize the amount of testing while maintaining complete coverage of interacting variables. Intelligent Test Case Handler enables the user to generate small test suites with strong coverage properties, choose regression suites, and perform other useful operations for the creation of systematic software test plans.

You should follow the instructions on the WHITCH site for installing the plug-in into your Eclipse directory (we used Eclipse 3.2). The user guide is then part of Eclipse help and details the features and instructions for using the tool. We will simply highlight how you might decide to use it.

First create a new Whitch file. File -> New -> Other... -> Whitch file. You must select the project and provide a file name. We used 'groovy.whitch'.

Now add your types similar to the earlier example. You should end up with something like:

Now add some attributes corresponding to the types you have just added. The result will be something like:

Now select Whitch -> Build. Type in a test suite name. We used 'AllPairs' and select
an interaction level of 2 (for pairs) as follows:

Now click 'Build Test Suite' to obtain your results:

Now save your Whitch file. The result will be that the test cases are now stored into an XML file.

Note: We have not shown any advanced Whitch features, e.g. it lets you add test cases in the Include tab which must always be added into the test suite and test cases which are not possible into the Exclude tab. It also lets you try to reduce your test case size, give weightings, choose different algorithms for test case generation and more. See the Whitch user guide for more details.

Code Block
def testsuite = new XmlSlurper().parse(new File('groovy.whitch'))
def attributes = testsuite.Model.Profile.Attribute
def testcases = testsuite.TestCases.TestCase
println testcases.size() + ' pairs:'
testcases.each{ testcase ->
    def map = [:]
    (0..2).each{
        def key = attributes[it].'@name'.toString()
        def value = testcase.Value[it].'@val'.toString()
        map.put(key, value)
    }
    println map
}

Running this script yields the following results:

Code Block
14 pairs:
["memory":"256M", "os":"Windows", "disk":"5G"]
["memory":"512M", "os":"MacOS", "disk":"5G"]
["memory":"1G", "os":"Linux", "disk":"5G"]
["memory":"2G", "os":"DC.DC", "disk":"5G"]
["memory":"1G", "os":"MacOS", "disk":"10G"]
["memory":"2G", "os":"Windows", "disk":"10G"]
["memory":"256M", "os":"DC.DC", "disk":"10G"]
["memory":"512M", "os":"Linux", "disk":"10G"]
["memory":"2G", "os":"Linux", "disk":"DC.DC"]
["memory":"512M", "os":"Windows", "disk":"DC.DC"]
["memory":"256M", "os":"MacOS", "disk":"DC.DC"]
["memory":"256M", "os":"Linux", "disk":"DC.DC"]
["memory":"2G", "os":"MacOS", "disk":"DC.DC"]
["memory":"1G", "os":"Windows", "disk":"DC.DC"]

Note: the value "DC.DC" indicates a don't care value and can be replaced with any value for that field.