The core module contains the generic execution engine of DuckHawk. It identifies the following concepts:
- ways to run tests, which might vary according to the need
- objects that executes the test itself
- objects that do listen to the test results
The following class diagram provides a visual overview.
The actual testing code is performed by TestExecutor, which contains a way to perform a single timed run (run) and a way to check the results out of the timing control (check). It's an interface, so one is free to implement it as it sees fit. Compared to JUnit, a TestExecutor represents a single method in a test class.
The TestExecutorFactory provides way to create test executors and metadata. TestMetadata contains the minimum identification informations for the running test, that is, the product, the version of the product, and the test id.
On the other side of the fence we have TestListeners, objects interested in the outcome of the tests. The callbacks provide the necessary notification about the test run start, the test run end, and about the output of each single test call (performance and stress runners will invoke the TestExecutor.run(...) method many times). The module contains two general use listeners, one, the PrintStreamListener, simply dumps the test events out to the provided PrintStream, whilst PerformanceSummarizer accumulate timings of each single call in order to provide statistics such as average and median time.
The TestRunner hierarchy represents the link between the world of test execution and test listening. Each test runner accepts a number of listeners that might be interested in the test results and runs the test generated by the test factory, in different ways according to its nature:
- ConformanceTestRunner will create one executor and run it just once, and verify no exception has been thrown during the
runorcheckphases (a test is supposed to fail when an exception is thrown, just like in JUnit, where each failing assertXXX(...) call will actually throw a non checked Exception) - PerformanceTestRunner will create one executor and run it once, in order to warm it up, and then
repetitiontimes in order to get a good sample of the actual performance. The optionaltimeandrandomparameters allow to distribute the calls overtimesecond, turning this runner in a building block for a workload test. - StressTestRunner will create
numThreadsseparate threads, each with its own executor, and have each one run in parallel. The user can also provide a ramp up time, in this case the threads won't be started in parallel, but created one after the other in order to launch the last thread when the ramp up time expired.
To allow for some extensibility and generality, each TestExecutor and TestListener calls do get a TestProperties parameter, which is really just a Map<String, Object. The properties act like a communication bus between the tests and the listeners. Basically any event is setup in a way that a fresh property map gets created, passed to the executor that can fill it up with interesting information (like, for example, request and response dumps for web services calls), then it's passed onto each listener, in the order they were added, so one listener can compute some summary information, or transform some key, and have other listeners can use them as they see fit.
There are two of such buses:
- one for each single TestExecutor.run() call. The properties map is cleared before the test is run, and then filled in the test executor itself, and in each listener
- one for the start/end events. The same map cleared before start occurrs, and is used in both events.
A listener listening on both kind events can act as a stateful summarizer, such as PerformanceSummarizer, that does listen to each call event, and then builds statistics to be put in the property map when the end event occurs.
If you need more information, please refer to the class javadocs.
Labels