I have just finished the first draft of the benchmarking suite based on the banking benchmark that was given to us. This present a framework to separate the engine execution from the main benchmark framework. I currently have just one benchmark, banking, for drools - but it should be enough for people to understand the framework. I have not yet made an extensible build system, so its hard coded to demonstrate the banking example.
So how does it work?
There is a class benchmark.ExecuteBenchmarks this reads in a file called main.conf. This property file specifies the benchmark to run, both it's configuration file and the class:
benchmark-conf=banking.conf
benchmark-class=benchmark.banking.BankingBenchmarkRunner
allowedEngines=drools
Currently its hard-coded to "main.conf", but this really should be provided by a main(String[] args) argument, to allow ant to dictate which benchmarks will be executed.
So this instantiations the BankingBenchmarkRunner class whish is responsible for the data loading for each iteration. Note this class is engine independant, it achieves this independance by using a thing api wrapper over the engine session with the interface SessionRuleRunner which just has the following methods (think slimmed down jsr94) insert, setGlobal, run, getSingleStats.
So then it opens up the banking.conf, as specified by the main.conf. This specifies the benchmark name, some variables the benhmark needs, i.e. "approxCashFlows" and provides one or more engien configurarions that it will execute against (although this is filtered by teh allowedEngine property in main.conf).
benchmark-name=Banking
approxCashFlows=5000
drools=banking-drools.conf
So lets look at banking-drools.conf:
benchmark-name=Banking
SetRunner = benchmark.drools.DroolsSetRuleRunner
iterations = 5
rule-files = /benchmark/banking/drools/banking.drl
forceRuleBaseReload = false
stdout=output/benchmark/banking/drools/banking-drools.stdout
statsout=output/benchmark/banking/drools/banking-drools.stats
appout=output/benchmark/banking/drools/banking-drools.app
Again it sets the benchmark name, this should be the same as the .conf above. It species the drools specific concrete implementation of SetRuleRunner as DroolsSetRuleRunner - this class (interface) is responsible for building the rulebase and returning sessions of the SessionRuleRunner interface. iterations is the number of iterations it will repeat the benchmark fore, rule-files is a comma delimteted list of rule files to add to the rulebase, forceRuleBaseReload when true will reload the rulebase from the rule files on each iteration. stdout is the output file from printouts of the rule files, statsout is there the statistics output go and appout is any misc application output as generated by the BankingBenchmarkRunner class.
So the only drools specific parts to the benchmarking framework are DroolsSessionRuleRunner and DroolsSetRuleRunner, note these are not specific to the banking benchmark and should be re-usueable for all benchmark classes. So once you have implemented these two "drivers" it should just be a matter of adding the configuration files shown above to plug them into created benchmark file.
While executing it will gather rule load times (only once, unless forced to repeat), data load times and rulefiring times (I also need to get it to add the total rule firings, haven't don that yet). It prints these out for each iteration and then an over all average set of timings. Nor have a added memory usage, I've left this out as the "freememory" method is scientific and only an finger in the air, we can add it back in or an alternative way once we all agree on what the best way is.