DuckHawk is here to help developers handle one crucial aspect in the long time evolution of a java library/application: performance.
Nowadays we have free tools to handle features and bugs, packaging, documentation and the like, all integrated in the continuous build process of an application. The topic of performance is not as well covered. This is understandable, whatever the application does must first be correct and accessible.
There are some performance related tools around, but none seems to be fitting the bill.
Tools like JUnitPerf can be used to harness the familiar JUnit approach to set hard performance requirements, such as "operation x must not take more than y seconds" or "response time must not exceed z when the application is under load by n concurrent users". Yet this approach fails to satisfy the needs of most open source projects, as well as some commercial ones:
- hard requirements are test host dependent (change the hardware where the test is run, you'll have to adapt all of the limits used as performance checks)
- an open source project has no hard performance requirements to meet
- hard requirements satisfaction does not tell anything about the actual performance evolution, developers won't get notified that performance dropped 2 times if the hard requirement is still satisfied
On the other side of the spectrum we have utilities like JMeter or The Grinder, allowing a user to setup a multithreaded load test, run the tests and graph the results. Unfortunately there are limitations there too:
- JMeter cannot be used as a library, so programmatic control is weak, setup is tedious
- The Grinder is programmatic, but forces the user to write the tests in Jython (which for someone may be nice, but I personally don't like being forced to learn a new programming language in order to write performance tests) and seems quite hard to get into (spent a day on it, could not even figure out how to use it...)
- none of them addresses the continuous performance checking issue
The Eclipse project created a specific performance harness based on relative comparisons between some reference builds and the current one. Data is stored and can be used to make up graphs like these ones:
This allows for relative comparison, and for evolution tracking over time. When a modification disrupts performance it can be noticed quickly, limiting the number of actual version control commits to be checked for, and counter measure can be taken to bring back performance on the previous level. Granted, care must be taken anyways, the machine running performance tests needs to be always the same, and no other activity must run while the performance assessment is underway. Yet, automated, relative and continuous performance checks allow performance to be kept under control just as easily as hard bugs.
This is all nice and dandy, but unfortunately the Eclipse approach is... Eclipse specific. Not much is known about it, no one else uses it (I cannot even say if performance tests are available or not).
Open source projects caring about performance do need an easy to use set of tools to:
- setup performance tests
- automate perfomance runs executions
- record results of performance runs
- analize them in a friendly environment
More importantly, it has to be open source, free for everyone to use an improve, and easily adaptable to other relative performance checking needs. That's the reason DuckHawk is being developed.