The TCK is the standard test suite for verifying that a Groovy implementation complies with the Groovy Language Specification. It is analogous to the JCK (Java Compatibility Kit) test suite used to verify compliance with the Java Language Specification.
The TCK may also be used as a non-normative specification for the Groovy language, or as a part of a functional test suite for a Groovy implementation.
On disk, the TCK consists of an Ant script (build.xml), source code for the controlling logic (src directory), source code for the TCK tests (test directory), and the various libraries required to support the running of the TCK (lib directory). To run the tests, the jar file containing the Groovy implementation under test must be placed into the testlib directory, or by default it will use Reference Implementation (RI) libraries for testing.
To run the tests, the Ant script is started (the RI will invoke Ant via Maven). The Ant script:
Step #2, in which TCK test source is converted into raw test source, is the source of much unintended confusion. For clarity, it is important to keep in mind the distinction between controlling unit tests and raw unit tests.
and if you are running outside of the RI maven build, the Ant script:
Each Groovy source file in the test directory contains a GroovyTestCase class. Markers (containing the characters '@fail') may be inserted into this file to indicate lines that would cause the test case to fail if included.
There are also '@pass' markers to indicate lines that would not cause the test case to fail if included.
Step #2 in the ant script converts each Groovy test case class into a raw test case class. The raw test case class is a Java JUnit TestCase subclass. Each test method in the raw test case class runs the entire Groovy test case, but with a different one of the '@fail' or '@pass' lines included. The raw test class also includes a version of the Groovy test case with none of the '@fail' or '@pass' lines included.
To recap, if a Groovy test case (from the test directory) has five '@fail' lines and two '@pass' lines, the resulting raw test case (in the gentest directory) will have eight test methods:
When reading the TCK results, keep in mind that each individual test consists of an entire run of the Groovy test case. Generally, a 'pass' indicates that every test in the Groovy test case passed; if a single test fails, the entire test is marked as failed. For tests including an '@fail' linek, a pass indicates that there was at least one failure, though it may not have been the failure that the author intended.
What does this mean? If the TCK results show 100% tests pass, then the Groovy implementation under test has passed the TCK. If less than 100% pass, you should be careful when interpreting the result: a 75% pass rate does not necessarily indicate that the Groovy implementation under test is 75% complete.
For tests within tests, the TCK now reports the errors back up to the top level, be it a maven build or a test report, so no test failures should be missed, please let us know if this isn't the case...
There are many aspects of a Groovy implementation that are not tested by the TCK, such as stability under load. A successful TCK run does not imply that the Groovy implementation under test is suitable for any particular use.