Introduction
The general architecture is made up of three independent blocks:
- a runner, used to execute and time the performance tests
- a storage engine, used to store and retrieve timings of the various performance runs
- an indepent reporting application, possibly web based, that will provide various kind of charts and tables to interpret data, compare it with reference runs as well as recent runs (moving window comparison).
The main objective of DuckHawk is to make continuous performance testing as natural as writing JUnit test, so the first implementation will be very coder oriented. Yet, it does not really need to be so limited. By creating the right extension points it should be possible to build on top of the library and address other usages, like one time interactive testing.
The runner
JUnit is a natural candidate for the first block, developers are used to write tests already, build systems and continuous build engines already automate unit test runs.
All that is needed is to bend the unit testing backend so that running the tests results in a performance run as well.
Yet, binding to classic JUnit alone seems like a limiting choice. Some projects might want to use JUnit4 or TestNG, define their tests in a scripting language or in XML.
All that the performance evaluator needs is something similar to Runnable to execute over and over, how this has been extracted out of a codebase is not the business of the actual performance evaluation harness.
Storing the performance results is not the business of the performance evaluation subsystem: this one will limit itself to raise events with the test results, and a variety of listeners will handle them, each in its specific way. Some listeners will perform data storage, some others might just dump them on the standard output, and others might fire notifications about some notable results.
The storage engine
Storage requirements as well might vary wildly.
Depending on the user, some might prefer a quick and easy embedded database, whilst other might want to store the results in an Oracle enterprise system.
Some other may not want a long term storage engine at all, but may want to generate a summary PDF reports at the end of the run.
A storage subsystem will implement a set of DAO interfaces, and the default implementation will leverage Hibernate running on top of an embedded database.
The reporting application
The data is stored using the storage engine may be visualized in a variety of ways. One or more interactive applications can be developed on top of the DAO interfaces to provide interactive display of the results, drill down into the actual numbers and the like. Again, we'll roll out a simple one to get started, and see if anyone comes up with nicer alternative.
The modules
Here is a list of the modules, each one gets a page providing an overview of it: