This page lists a few tricks of the trade and things to consider when benchmarking JMS providers.
Non-durable testing
This is the easiest and most straightforward of test cases. Typically the main test is throughput on each node. Often the number of connections used can have an effect - as typically 1 connection maps to 1 socket.
There's often a tcpNoDelay flag in a JMS provider which indicates whether or not an entire packet/buffer is filled with messages before it is sent/flushed to the socket or not. For throughput tests its better to pack packets completely full with message data which tends to maximise throughput. However this flag affects latency, since if you send a small message it needs to wait around a while until another message comes along to fill the packet/buffer.
So its probably worth having 2 settings for most providers
- high throughput setting
- low latency setting
Then, for providers which support such configurations, its easy to see the effect of this
Durable testing
This is often the hardest thing to test as there is a fair variety of ways of implementing it. The main issue is - syncing to disk on every message leads to a very slow throughput. So there are many tricks that providers use to improve throughput.
The most important thing to ensure when testing this category of providers is to always test the same thing to avoid confusing users. There follows a list of things to consider
When does the message really get written to disk?
There are optimisations a JMS provider can make to decide when to write a message to disk. Disk syncs are very slow; so there are many tricks to boost perceived performance. One is to only asynchronously sync to disk. This gives great performance numbers, but does mean you can loose messages if there is a crash after a JMS send() has been called but before the message bits the disk.
Another performance optimisation is to pause each writing thread a little, then batch many threads writes together into one physical write. This is how high performance transaction logs work.
Does a send() RPC the broker synchronously or async?
When performing a JMS send() in PERSISTENT delivery mode, strictly speaking a JMS provider should RPC the message broker, waiting for a successful response that the message has been written to disk. So this should be slow 
. i.e. the send() should wait for a disk sync.
However often you can disable this so that the send is async, or the RPC returns as soon as the message is received. Care should be taken to try test the same thing. e.g. have a rigorous, cannot loose a message test and a, durable but lower the rigor test etc.
Testing transactions is more accurate
A JMS Session.commit() must complete when things are sync'd to disk and so is a good way to test durable messaging - whereas the PERSISTENT send() mentioned above is a little more vague.
Batching
In practice, to get high throughput, often end users use transactions and batch things. e.g. consume 100 messages and send 100 messages, all async, then do a commit() (which is always a sync RPC with the broker and so slow).
This can also be used to batch the acknowledgement of messages as well.
Message Broker configurations
Some JMS providers, like ActiveMQ allow the JMS broker to be embedded in the JMS client so that the RPC time when performing persistence is low. Often they are remote. Some QoS settings allow for clusters / DR ability so that messages are replicated across different brokers synchronously.