Esper performance

Added by Alexandre Vasseur, last edited by Alexandre Vasseur on Aug 11, 2007  (view change)

Performance kit

Esper includes a NIO based event server and client that can be used to perform performance testing and tuning.

The kit is available in the source repository under GPL license, in esper/trunk/examples/benchmark/ . Read http://xircles.codehaus.org/projects/esper/repo for source access.

A snapshot build of the kit is also available attached to this page.

To learn more on how to use the kit, refer to Esper reference manual at http://esper.codehaus.org/esper/documentation/documentation.html
You may want to read the documentation to fully understand the results presented here and make up your owns.

Want to help?

"Contribute or challenge us"

We are willing to publish results on different hardware/OS/JVM platforms. If you want to contribute your results please let us know on the mailing list
or by using the contact link on the top right corner of the http://esper.codehaus.org page.

Performance results

Configuration

Item Specification
Operating System for Esper event server and client Red Hat 64bit: Red Hat Enterprise Linux AS release 4 (Nahant Update 4) Linux version 2.6.9-42.ELsmp (bhcompile@ls20-bc1-13.build.redhat.com) (gcc version 3.4.6 20060404 (RedHat 3.4.6-2)) #1 SMP Wed Jul 12 23:32:02 EDT 2006 (x86_64)
Hardware for Esper event server 2 x Intel Xeon 5130 2GHz (4 cores total), 16GB RAM
Hardware for client (all clients on same box) 2 x Intel Xeon 5130 2GHz (4 cores total), 4GB RAM
Network 100 Mbit/s
JVM version (server and client) BEA JRockit(R) R27.3 linux 32bit
JVM paremeters (server) (basically 2GB heap, generational garbage collector) -Xms2g -Xmx2g -Xns128m -Xgc:gencon

Note that those parameters apply to JRockit. For Sun JVM you may want to use: -Xms2g -Xmx2g -XX:NewSize=128m -XX:MaxNewSize=128m or similar.

Summary

All statements are suffixed with " output last every 30 seconds"
There are 1000 symbols in the market (so for example 100 000 evt/s means 10 000 evt/s/symbol)
With 1000 symbols, each event is 28 bytes sized that is 224 bits (symbol of 4 characters, one double, one long, one int). On a 100 Mbit/s network this means the theoretical limit is around 460 000 evt/s (likely less due to TCP headers).

Run Number of statements Results Esper latency Average Esper latency Notes
simulation without remote clients, no statement 0 995 900 evt/s 99.78% < 5us 2.6us server simulates load, no remote client
no statement 0 408 678 evt/s 99.2% < 5us 1.5us CPU at 50%, throughput limited by 100Mbit/s network bandwith
select * from MarketData(symbol='$') ... 1000 407 216 evt/s 99.50% < 10us 2.9us CPU at 75%, throughput limited by 100Mbit/s network bandwith
select * from MarketData(symbol='$').win:length(1) ... 1000 355 110 evt/s 99.77% < 15us 5us CPU at 95% (this run was likely too short to get fully optimized
select * from Market(ticker='NONE') ... 1 407 070 evt/s 99.60% < 10us 2.6us CPU at 70%, throughput limited by 100Mbit/s network bandwith
select * from Market(ticker='NONE-$') ... 1000 409 163 evt/s 99.57% < 10us 2.5us CPU at 70%, throughput limited by 100Mbit/s network bandwith
select '$' as ticker, avg(price), count(*)
, sum(price) from Market(ticker='$').win:length(1000) ... 		1000 395 451 evt/s 99.38% < 10us 3.2us CPU at 80%
select '$' as ticker, avg(price), count(*)
, sum(price) from Market(ticker='$').win:time(2 seconds) ... 		1000 411 906 evt/s 99.64% < 10us 2.5us CPU at 65%, throughput limited by 100Mbit/s network bandwith
select '$' as ticker from Market(ticker='$').win:length(1000).stat:weighted_avg('price', 'volume') ... 1000 412 503 evt/s 99.62% < 10us 2.5us CPU at 65%, throughput limited by 100Mbit/s network bandwith. See equivalent run with smaller event below that reach more than 500 000 evt/s.
select '$' as ticker from Market(ticker='$').win:time(2 seconds).stat:weighted_avg('price', 'volume') ... 1000 todo evt/s todo < 10us todo

In order to go beyond 100Mbit limitation we reduced the event size by removing a long in the event that is used to track end to end latency, and we shrinked the volume atribute from int to byte.
With such a configurationn, each event is 17 bytes sized that is 136 bits, and on a 100 Mbit/s network this means the theoretical limit is around 770 000 evt/s (likely less due to TCP headers).
Under such a configuration we got the following:

Run Number of statements Results Esper latency Average Esper latency Notes
select '$' as ticker from Market(ticker='$').win:length(1000).stat:weighted_avg('price', 'volume') ... 1000 519 728 evt/s 99.66% < 10us 2.8us CPU at 85%, 70 Mbit/s

Adding an extra 50 000 event/s in the system overloaded the CPU and pushed performance back to around 500 000 evt/s, with an average latency at 5us.

What does this all means?

This basically means that with just a 2 CPU dual core 2GHz platform, we get Esper performance at more than 500 000 event/s on a VWAP over 1000 symbols and 1000 statements, with a latency average below 3 microseconds, and below 10 microseconds with more than 99% predictability.
Also, we get around 400 000 evt/s accross all runs with different statements and we easily overload the 100Mbit/s network, with CPU usage at about 70%. The internal Esper latency is at around 5 microseconds average and below 10 microsecond with more than 99% predictability (all but one where we need to get at 15 microseconds to reach 99% predictability).

If you have ideas for more complex statements, please let us know. We will work on CEP statements later on (those are all ESP statements).

Detailled results

Notes

Throughput and latency are choosen after about a 5 minutes run.

Throughput is expressed in evt/s, and usually indicated the number of client connected. The active and pending information are specific to the performance kit and can be ignored.

We indicate two kinds of results (unless otherwise specified):

  • the results with 1 client sending 100 000 evt/s early in the run, which gives an idea of under-optimized performance results. Indeed, modern JVM makes use of advanced optimization techniques and a warm up of more than 5 minutes is common to obtain best performance.
  • the results with 4 client sending 100 000 evt/s each, measured later in the run (although usually no longer than 10 min after startup)
    The JVM optimization effect also surfaces when looking at the latency statistics under heavier load (f.e. 4x 100 000 evt/s) which are measured later in the run with a more optimized code path. Latency with heavier load appears better than latency with lower load only due to the first measurement beeing made early in the run.
End to end latency statistics
End to end latency is measured by computing a delta between the time the client sends an event and the server ends handling it. Those results are impacted by clock synchronisation (ntpd accuracy). You may check how accurrate are your clocks with ntptrace OS level tools.
In those results we had about 6 microseconds delta so we can assume end to end latency (unit in millisecond) is correctly reported.

Simulation (-rate 14x100000), no statements

Throughput 995900 (active 0 pending 0)

---Stats - engine (unit: ns)
  Avg: 2656 #9936457
        0 <    5000:  99.78%  99.78% #9914178
     5000 <   10000:   0.20%  99.98% #20120
    10000 <   15000:   0.01%  99.98% #646
    15000 <   20000:   0.00%  99.99% #69
    20000 <   25000:   0.00%  99.99% #36
    25000 <   50000:   0.00%  99.99% #56
    50000 <  100000:   0.00%  99.99% #26
   100000 <  500000:   0.00%  99.99% #189
   500000 < 1000000:   0.00%  99.99% #2
  1000000 < 2500000:   0.00%  99.99% #68
  2500000 < 5000000:   0.01% 100.00% #748
  5000000 <    more:   0.00% 100.00% #319

No statements

1x 100 000 evt/s

Throughput 102027 (active 0 pending 0 cnx 1)
23 Mbit/s

---Stats - engine (unit: ns)
  Avg: 668 #1020230
        0 <    5000:  99.32%  99.32% #1013265
     5000 <   10000:   0.67%  99.98% #6797
    10000 <   15000:   0.01%  99.99% #112
    15000 <   20000:   0.00% 100.00% #10
    20000 <   25000:   0.00% 100.00% #2
    25000 <   50000:   0.00% 100.00% #28
    50000 <  100000:   0.00% 100.00% #1
   100000 <  500000:   0.00% 100.00% #15
   500000 < 1000000:   0.00% 100.00% #0
  1000000 < 2500000:   0.00% 100.00% #0
  2500000 < 5000000:   0.00% 100.00% #0
  5000000 <    more:   0.00% 100.00% #0
---Stats - endToEnd (unit: ms)
  Avg: -1 #1020230
        0 <       1:  62.37%  62.37% #636289
        1 <       5:  27.16%  89.53% #277091
        5 <      10:  10.43%  99.96% #106401
       10 <      50:   0.04% 100.00% #449
       50 <     100:   0.00% 100.00% #0
      100 <     250:   0.00% 100.00% #0
      250 <     500:   0.00% 100.00% #0
      500 <    1000:   0.00% 100.00% #0
     1000 <    more:   0.00% 100.00% #0

3x 100 000 evt/s

Throughput 307132 (active 0 pending 0 cnx 3)
70 Mbit/s

---Stats - engine (unit: ns)
  Avg: 1337 #3066309
        0 <    5000:  99.28%  99.28% #3044091
     5000 <   10000:   0.61%  99.88% #18564
    10000 <   15000:   0.11%  99.99% #3366
    15000 <   20000:   0.00%  99.99% #75
    20000 <   25000:   0.00%  99.99% #47
    25000 <   50000:   0.00% 100.00% #108
    50000 <  100000:   0.00% 100.00% #9
   100000 <  500000:   0.00% 100.00% #41
   500000 < 1000000:   0.00% 100.00% #0
  1000000 < 2500000:   0.00% 100.00% #3
  2500000 < 5000000:   0.00% 100.00% #5
  5000000 <    more:   0.00% 100.00% #0
---Stats - endToEnd (unit: ms)
  Avg: 1 #3066580
        0 <       1:  44.01%  44.01% #1349488
        1 <       5:  28.19%  72.20% #864541
        5 <      10:  21.61%  93.81% #662752
       10 <      50:   6.19% 100.00% #189799
       50 <     100:   0.00% 100.00% #0
      100 <     250:   0.00% 100.00% #0
      250 <     500:   0.00% 100.00% #0
      500 <    1000:   0.00% 100.00% #0
     1000 <    more:   0.00% 100.00% #0

4x 100 000 evt/s

Throughput 408678 (active 0 pending 0 cnx 4)
93 Mbit/s

---Stats - engine (unit: ns)
  Avg: 1508 #4081567
        0 <    5000:  99.20%  99.20% #4048966
     5000 <   10000:   0.67%  99.88% #27523
    10000 <   15000:   0.12%  99.99% #4759
    15000 <   20000:   0.00% 100.00% #117
    20000 <   25000:   0.00% 100.00% #54
    25000 <   50000:   0.00% 100.00% #35
    50000 <  100000:   0.00% 100.00% #17
   100000 <  500000:   0.00% 100.00% #76
   500000 < 1000000:   0.00% 100.00% #3
  1000000 < 2500000:   0.00% 100.00% #10
  2500000 < 5000000:   0.00% 100.00% #7
  5000000 <    more:   0.00% 100.00% #0
---Stats - endToEnd (unit: ms)
  Avg: 5 #4083498
        0 <       1:  20.73%  20.73% #846355
        1 <       5:  18.36%  39.09% #749686
        5 <      10:  25.64%  64.72% #1046876
       10 <      50:  35.28% 100.00% #1440581
       50 <     100:   0.00% 100.00% #0
      100 <     250:   0.00% 100.00% #0
      250 <     500:   0.00% 100.00% #0
      500 <    1000:   0.00% 100.00% #0
     1000 <    more:   0.00% 100.00% #0

5x 100 000 evt/s

Well, network bandwith is not enough.

Graph

select * from MarketData(symbol='$') ... (x1000) (Straight Through Processing)

1x 100 000 evt/s

Throughput 100848 (active 0 pending 0 cnx 1)

---Stats - engine (unit: ns)
  Avg: 1804 #1002338
        0 <    5000:  95.98%  95.98% #962055
     5000 <   10000:   3.78%  99.76% #37899
    10000 <   15000:   0.22%  99.98% #2178
    15000 <   20000:   0.00%  99.98% #25
    20000 <   25000:   0.00%  99.98% #23
    25000 <   50000:   0.01%  99.99% #96
    50000 <  100000:   0.00%  99.99% #7
   100000 <  500000:   0.00% 100.00% #50
   500000 < 1000000:   0.00% 100.00% #2
  1000000 < 2500000:   0.00% 100.00% #0
  2500000 < 5000000:   0.00% 100.00% #0
  5000000 <    more:   0.00% 100.00% #3
---Stats - endToEnd (unit: ms)
  Avg: 3 #1002507
        0 <       1:  29.58%  29.58% #296580
        1 <       5:  17.85%  47.44% #178966
        5 <      10:  23.09%  70.53% #231522
       10 <      50:  28.17%  98.70% #282362
       50 <     100:   0.07%  98.76% #678
      100 <     250:   1.24% 100.00% #12399
      250 <     500:   0.00% 100.00% #0
      500 <    1000:   0.00% 100.00% #0
     1000 <    more:   0.00% 100.00% #0

3x 100 000 evt/s

Throughput 304790 (active 0 pending 0 cnx 3)

---Stats - engine (unit: ns)
  Avg: 2692 #3021513
        0 <    5000:  85.76%  85.76% #2591254
     5000 <   10000:  13.70%  99.46% #414083
    10000 <   15000:   0.49%  99.96% #14943
    15000 <   20000:   0.03%  99.99% #816
    20000 <   25000:   0.00%  99.99% #143
    25000 <   50000:   0.00%  99.99% #89
    50000 <  100000:   0.00% 100.00% #48
   100000 <  500000:   0.00% 100.00% #115
   500000 < 1000000:   0.00% 100.00% #4
  1000000 < 2500000:   0.00% 100.00% #5
  2500000 < 5000000:   0.00% 100.00% #2
  5000000 <    more:   0.00% 100.00% #11
---Stats - endToEnd (unit: ms)
  Avg: 914570251413509 #3021833
        0 <       1:  21.88%  21.88% #661306
        1 <       5:   3.47%  25.36% #104950
        5 <      10:   4.89%  30.25% #147860
       10 <      50:  16.99%  47.25% #513559
       50 <     100:   2.40%  49.65% #72584
      100 <     250:   0.31%  49.95% #9286
      250 <     500:   0.00%  49.95% #0
      500 <    1000:   0.00%  49.95% #0
     1000 <    more:  50.05% 100.00% #1512288

4x 100 000 evt/s

Throughput 407216 (active 0 pending 0 cnx 4)

---Stats - engine (unit: ns)
  Avg: 2930 #4069337
        0 <    5000:  86.85%  86.85% #3534350
     5000 <   10000:  12.65%  99.50% #514830
    10000 <   15000:   0.46%  99.96% #18673
    15000 <   20000:   0.02%  99.99% #898
    20000 <   25000:   0.00%  99.99% #187
    25000 <   50000:   0.00%  99.99% #183
    50000 <  100000:   0.00% 100.00% #47
   100000 <  500000:   0.00% 100.00% #129
   500000 < 1000000:   0.00% 100.00% #4
  1000000 < 2500000:   0.00% 100.00% #9
  2500000 < 5000000:   0.00% 100.00% #5
  5000000 <    more:   0.00% 100.00% #22
---Stats - endToEnd (unit: ms)
  Avg: -1164451198077914110 #4069865
        0 <       1:  40.83%  40.83% #1661833
        1 <       5:   2.48%  43.32% #101089
        5 <      10:   3.29%  46.60% #133808
       10 <      50:  14.87%  61.47% #605055
       50 <     100:   1.05%  62.52% #42868
      100 <     250:   0.00%  62.52% #0
      250 <     500:   0.00%  62.52% #0
      500 <    1000:   0.00%  62.52% #0
     1000 <    more:  37.48% 100.00% #1525212

Graph

select * from MarketData(symbol='$').win:length(1) ... (x1000) (Straight Through Processing)

1x 100 000 evt/s

Throughput 97912 (active 0 pending 0 cnx 1)

---Stats - engine (unit: ns)
  Avg: 2096 #987795
        0 <    5000:  92.26%  92.26% #911315
     5000 <   10000:   7.21%  99.46% #71189
    10000 <   15000:   0.48%  99.95% #4787
    15000 <   20000:   0.02%  99.97% #165
    20000 <   25000:   0.02%  99.99% #199
    25000 <   50000:   0.01% 100.00% #104
    50000 <  100000:   0.00% 100.00% #6
   100000 <  500000:   0.00% 100.00% #27
   500000 < 1000000:   0.00% 100.00% #0
  1000000 < 2500000:   0.00% 100.00% #0
  2500000 < 5000000:   0.00% 100.00% #0
  5000000 <    more:   0.00% 100.00% #3
---Stats - endToEnd (unit: ms)
  Avg: 4 #987993
        0 <       1:  26.62%  26.62% #262982
        1 <       5:  15.04%  41.66% #148596
        5 <      10:  19.44%  61.10% #192083
       10 <      50:  37.55%  98.65% #371038
       50 <     100:   0.07%  98.72% #672
      100 <     250:   1.28% 100.00% #12622
      250 <     500:   0.00% 100.00% #0
      500 <    1000:   0.00% 100.00% #0
     1000 <    more:   0.00% 100.00% #0

4x 100 000 evt/s

Throughput 355110 (active 0 pending 0 cnx 4)

---Stats - engine (unit: ns)
  Avg: 5094 #3518671
        0 <    5000:  32.47%  32.47% #1142688
     5000 <   10000:  59.11%  91.58% #2079757
    10000 <   15000:   8.18%  99.77% #287959
    15000 <   20000:   0.19%  99.95% #6646
    20000 <   25000:   0.02%  99.97% #565
    25000 <   50000:   0.02%  99.99% #654
    50000 <  100000:   0.00%  99.99% #122
   100000 <  500000:   0.01% 100.00% #206
   500000 < 1000000:   0.00% 100.00% #7
  1000000 < 2500000:   0.00% 100.00% #14
  2500000 < 5000000:   0.00% 100.00% #9
  5000000 <    more:   0.00% 100.00% #44
---Stats - endToEnd (unit: ms)
  Avg: -911492425168174340 #3519041
        0 <       1:  28.72%  28.72% #1010610
        1 <       5:   0.03%  28.75% #987
        5 <      10:   0.06%  28.81% #2157
       10 <      50:  56.19%  85.00% #1977518
       50 <     100:   8.89%  93.89% #312703
      100 <     250:   6.08%  99.97% #213938
      250 <     500:   0.03% 100.00% #1128
      500 <    1000:   0.00% 100.00% #0
     1000 <    more:   0.00% 100.00% #0

Graph

select * from Market(ticker='NONE') (filter all, one statement)

1x 100 000 evt/s

Throughput 102220 (active 0 pending 0 cnx 1)

---Stats - engine (unit: ns)
  Avg: 1035 #1020000
        0 <    5000:  99.05%  99.05% #1010347
     5000 <   10000:   0.90%  99.96% #9229
    10000 <   15000:   0.03%  99.99% #324
    15000 <   20000:   0.00%  99.99% #20
    20000 <   25000:   0.00%  99.99% #26
    25000 <   50000:   0.00% 100.00% #32
    50000 <  100000:   0.00% 100.00% #3
   100000 <  500000:   0.00% 100.00% #19
   500000 < 1000000:   0.00% 100.00% #0
  1000000 < 2500000:   0.00% 100.00% #0
  2500000 < 5000000:   0.00% 100.00% #0
  5000000 <    more:   0.00% 100.00% #0
---Stats - endToEnd (unit: ms)
  Avg: -0 #1020000
        0 <       1:  56.00%  56.00% #571230
        1 <       5:  24.61%  80.61% #250991
        5 <      10:  16.85%  97.46% #171904
       10 <      50:   2.54% 100.00% #25875
       50 <     100:   0.00% 100.00% #0
      100 <     250:   0.00% 100.00% #0
      250 <     500:   0.00% 100.00% #0
      500 <    1000:   0.00% 100.00% #0
     1000 <    more:   0.00% 100.00% #0

4x 100 000 evt/s

Throughput 407070 (active 0 pending 0 cnx 4)

---Stats - engine (unit: ns)
  Avg: 2607 #4069069
        0 <    5000:  96.74%  96.74% #3936429
     5000 <   10000:   2.86%  99.60% #116203
    10000 <   15000:   0.38%  99.98% #15587
    15000 <   20000:   0.01%  99.99% #451
    20000 <   25000:   0.00%  99.99% #92
    25000 <   50000:   0.00%  99.99% #96
    50000 <  100000:   0.00% 100.00% #42
   100000 <  500000:   0.00% 100.00% #115
   500000 < 1000000:   0.00% 100.00% #5
  1000000 < 2500000:   0.00% 100.00% #24
  2500000 < 5000000:   0.00% 100.00% #18
  5000000 <    more:   0.00% 100.00% #7
---Stats - endToEnd (unit: ms)
  Avg: 5 #4069614
        0 <       1:  24.99%  24.99% #1016951
        1 <       5:  18.22%  43.21% #741398
        5 <      10:  24.01%  67.21% #976968
       10 <      50:  32.79% 100.00% #1334297
       50 <     100:   0.00% 100.00% #0
      100 <     250:   0.00% 100.00% #0
      250 <     500:   0.00% 100.00% #0
      500 <    1000:   0.00% 100.00% #0
     1000 <    more:   0.00% 100.00% #0

Graph

select * from Market(ticker='NONE-$') (filter all, x1000 statements)

h3 1x 100 000 evt/s

Throughput 102735 (active 0 pending 0 cnx 1)

---Stats - engine (unit: ns)
  Avg: 1026 #1031736
        0 <    5000:  99.06%  99.06% #1022084
     5000 <   10000:   0.89%  99.95% #9146
    10000 <   15000:   0.04%  99.99% #388
    15000 <   20000:   0.00%  99.99% #44
    20000 <   25000:   0.00% 100.00% #23
    25000 <   50000:   0.00% 100.00% #29
    50000 <  100000:   0.00% 100.00% #2
   100000 <  500000:   0.00% 100.00% #20
   500000 < 1000000:   0.00% 100.00% #0
  1000000 < 2500000:   0.00% 100.00% #0
  2500000 < 5000000:   0.00% 100.00% #0
  5000000 <    more:   0.00% 100.00% #0
---Stats - endToEnd (unit: ms)
  Avg: -0 #1031937
        0 <       1:  54.53%  54.53% #562669
        1 <       5:  24.47%  79.00% #252538
        5 <      10:  17.77%  96.77% #183350
       10 <      50:   3.23% 100.00% #33380
       50 <     100:   0.00% 100.00% #0
      100 <     250:   0.00% 100.00% #0
      250 <     500:   0.00% 100.00% #0
      500 <    1000:   0.00% 100.00% #0
     1000 <    more:   0.00% 100.00% #0

4x 100 000 evt/s

Throughput 409163 (active 0 pending 0 cnx 4)

---Stats - engine (unit: ns)
  Avg: 2563 #4087058
        0 <    5000:  96.90%  96.90% #3960406
     5000 <   10000:   2.67%  99.57% #109060
    10000 <   15000:   0.40%  99.97% #16398
    15000 <   20000:   0.01%  99.98% #531
    20000 <   25000:   0.00%  99.99% #148
    25000 <   50000:   0.01%  99.99% #282
    50000 <  100000:   0.00% 100.00% #76
   100000 <  500000:   0.00% 100.00% #102
   500000 < 1000000:   0.00% 100.00% #4
  1000000 < 2500000:   0.00% 100.00% #24
  2500000 < 5000000:   0.00% 100.00% #13
  5000000 <    more:   0.00% 100.00% #14
---Stats - endToEnd (unit: ms)
  Avg: -4678322695573553 #4087472
        0 <       1:  42.39%  42.39% #1732801
        1 <       5:  12.01%  54.41% #491050
        5 <      10:  17.36%  71.76% #709453
       10 <      50:  28.24% 100.00% #1154168
       50 <     100:   0.00% 100.00% #0
      100 <     250:   0.00% 100.00% #0
      250 <     500:   0.00% 100.00% #0
      500 <    1000:   0.00% 100.00% #0
     1000 <    more:   0.00% 100.00% #0

Graph

select '$' as ticker, avg(price), count(*) , sum(price) from Market(ticker='$').win:length(1000) ... (x1000)

1x 100 000 evt/s

Throughput 95663 (active 0 pending 0 cnx 1)

---Stats - engine (unit: ns)
  Avg: 2392 #934559
        0 <    5000:  81.46%  81.46% #761332
     5000 <   10000:  17.93%  99.40% #167613
    10000 <   15000:   0.57%  99.97% #5339
    15000 <   20000:   0.01%  99.98% #50
    20000 <   25000:   0.00%  99.98% #40
    25000 <   50000:   0.01%  99.99% #118
    50000 <  100000:   0.00%  99.99% #14
   100000 <  500000:   0.01% 100.00% #50
   500000 < 1000000:   0.00% 100.00% #0
  1000000 < 2500000:   0.00% 100.00% #0
  2500000 < 5000000:   0.00% 100.00% #0
  5000000 <    more:   0.00% 100.00% #3
---Stats - endToEnd (unit: ms)
  Avg: 5 #934559
        0 <       1:  26.20%  26.20% #244817
        1 <       5:  12.39%  38.59% #115806
        5 <      10:  16.30%  54.89% #152328
       10 <      50:  43.26%  98.14% #404247
       50 <     100:   0.00%  98.14% #0
      100 <     250:   1.86% 100.00% #17361
      250 <     500:   0.00% 100.00% #0
      500 <    1000:   0.00% 100.00% #0
     1000 <    more:   0.00% 100.00% #0

2x 100 000 evt/s

Throughput 167713 (active 0 pending 0 cnx 2)

---Stats - engine (unit: ns)
  Avg: 3652 #1659084
        0 <    5000:  70.09%  70.09% #1162851
     5000 <   10000:  28.91%  99.00% #479612
    10000 <   15000:   0.86%  99.86% #14319
    15000 <   20000:   0.11%  99.97% #1830
    20000 <   25000:   0.01%  99.98% #114
    25000 <   50000:   0.01%  99.99% #215
    50000 <  100000:   0.00% 100.00% #64
   100000 <  500000:   0.00% 100.00% #70
   500000 < 1000000:   0.00% 100.00% #0
  1000000 < 2500000:   0.00% 100.00% #0
  2500000 < 5000000:   0.00% 100.00% #0
  5000000 <    more:   0.00% 100.00% #9
---Stats - endToEnd (unit: ms)
  Avg: -4940257817557899 #1659382
        0 <       1:  59.40%  59.40% #985706
        1 <       5:   4.97%  64.37% #82519
        5 <      10:   6.53%  70.91% #108363
       10 <      50:  27.37%  98.28% #454252
       50 <     100:   0.00%  98.28% #0
      100 <     250:   0.13%  98.41% #2120
      250 <     500:   1.59% 100.00% #26422
      500 <    1000:   0.00% 100.00% #0
     1000 <    more:   0.00% 100.00% #0

3x 100 000 evt/s

Throughput 306512 (active 0 pending 0 cnx 3)
70 Mbit/s

---Stats - engine (unit: ns)
  Avg: 2077 #3058344
        0 <    5000:  97.72%  97.72% #2988684
     5000 <   10000:   1.88%  99.61% #57588
    10000 <   15000:   0.38%  99.99% #11704
    15000 <   20000:   0.01%  99.99% #197
    20000 <   25000:   0.00% 100.00% #36
    25000 <   50000:   0.00% 100.00% #38
    50000 <  100000:   0.00% 100.00% #21
   100000 <  500000:   0.00% 100.00% #68
   500000 < 1000000:   0.00% 100.00% #1
  1000000 < 2500000:   0.00% 100.00% #1
  2500000 < 5000000:   0.00% 100.00% #0
  5000000 <    more:   0.00% 100.00% #6
---Stats - endToEnd (unit: ms)
  Avg: -13246053553247574 #3058703
        0 <       1: 100.00% 100.00% #3058703
        1 <       5:   0.00% 100.00% #0
        5 <      10:   0.00% 100.00% #0
       10 <      50:   0.00% 100.00% #0
       50 <     100:   0.00% 100.00% #0
      100 <     250:   0.00% 100.00% #0
      250 <     500:   0.00% 100.00% #0
      500 <    1000:   0.00% 100.00% #0
     1000 <    more:   0.00% 100.00% #0

We see a slight increase in throughput - likely because ramp-up is done after 5 min.

4x 100 000 evt/s

Throughput 395451 (active 0 pending 0 cnx 4)
88 Mbit/s

---Stats - engine (unit: ns)
  Avg: 3256 #3947217
        0 <    5000:  76.43%  76.43% #3017039
     5000 <   10000:  22.95%  99.38% #905884
    10000 <   15000:   0.56%  99.94% #21967
    15000 <   20000:   0.04%  99.98% #1650
    20000 <   25000:   0.00%  99.99% #157
    25000 <   50000:   0.01%  99.99% #239
    50000 <  100000:   0.00%  99.99% #68
   100000 <  500000:   0.00% 100.00% #151
   500000 < 1000000:   0.00% 100.00% #8
  1000000 < 2500000:   0.00% 100.00% #23
  2500000 < 5000000:   0.00% 100.00% #9
  5000000 <    more:   0.00% 100.00% #22
---Stats - endToEnd (unit: ms)
  Avg: -10656750366414336 #3947732
        0 <       1:  75.76%  75.76% #2990770
        1 <       5:   0.07%  75.83% #2843
        5 <      10:   0.54%  76.37% #21226
       10 <      50:  20.03%  96.40% #790700
       50 <     100:   2.38%  98.78% #94058
      100 <     250:   1.22% 100.00% #48135
      250 <     500:   0.00% 100.00% #0
      500 <    1000:   0.00% 100.00% #0
     1000 <    more:   0.00% 100.00% #0

Graph

select '$' as ticker, avg(price), count(*) , sum(price) from Market(ticker='$').win:time(2 seconds) ... (x1000)

1x 100 000 evt/s

Throughput 98500 (active 0 pending 0 cnx 1)

---Stats - engine (unit: ns)
  Avg: 2114 #982745
        0 <    5000:  93.40%  93.40% #917900
     5000 <   10000:   6.21%  99.61% #61047
    10000 <   15000:   0.36%  99.98% #3558
    15000 <   20000:   0.00%  99.98% #24
    20000 <   25000:   0.00%  99.98% #24
    25000 <   50000:   0.01%  99.99% #106
    50000 <  100000:   0.00% 100.00% #44
   100000 <  500000:   0.00% 100.00% #37
   500000 < 1000000:   0.00% 100.00% #1
  1000000 < 2500000:   0.00% 100.00% #0
  2500000 < 5000000:   0.00% 100.00% #0
  5000000 <    more:   0.00% 100.00% #4
---Stats - endToEnd (unit: ms)
  Avg: 4 #982876
        0 <       1:  29.31%  29.31% #288078
        1 <       5:  15.32%  44.63% #150578
        5 <      10:  19.83%  64.46% #194873
       10 <      50:  33.69%  98.14% #331084
       50 <     100:   0.08%  98.22% #776
      100 <     250:   1.78% 100.00% #17487
      250 <     500:   0.00% 100.00% #0
      500 <    1000:   0.00% 100.00% #0
     1000 <    more:   0.00% 100.00% #0

3x 100 000 evt/s

Throughput 273035 (active 0 pending 0 cnx 3)

---Stats - engine (unit: ns)
  Avg: 4108 #2717446
        0 <    5000:  37.27%  37.27% #1012720
     5000 <   10000:  61.18%  98.45% #1662617
    10000 <   15000:   1.39%  99.85% #37905
    15000 <   20000:   0.13%  99.98% #3574
    20000 <   25000:   0.01%  99.98% #211
    25000 <   50000:   0.01%  99.99% #228
    50000 <  100000:   0.00%  99.99% #47
   100000 <  500000:   0.00% 100.00% #118
   500000 < 1000000:   0.00% 100.00% #7
  1000000 < 2500000:   0.00% 100.00% #0
  2500000 < 5000000:   0.00% 100.00% #1
  5000000 <    more:   0.00% 100.00% #18
---Stats - endToEnd (unit: ms)
  Avg: 3934131616041082 #2717886
        0 <       1:  17.53%  17.53% #476363
        1 <       5:   0.01%  17.54% #393
        5 <      10:   0.12%  17.66% #3271
       10 <      50:  60.65%  78.32% #1648489
       50 <     100:   0.03%  78.35% #879
      100 <     250:   4.02%  82.37% #109235
      250 <     500:   0.00%  82.37% #0
      500 <    1000:   0.00%  82.37% #0
     1000 <    more:  17.63% 100.00% #479256

4x 100 000 evt/s

Throughput 411906 (active 0 pending 0 cnx 4)
93 Mbit/s

---Stats - engine (unit: ns)
  Avg: 2447 #4140913
        0 <    5000:  97.70%  97.70% #4045816
     5000 <   10000:   1.93%  99.64% #80111
    10000 <   15000:   0.34%  99.98% #14162
    15000 <   20000:   0.01%  99.99% #433
    20000 <   25000:   0.00%  99.99% #85
    25000 <   50000:   0.00%  99.99% #95
    50000 <  100000:   0.00% 100.00% #41
   100000 <  500000:   0.00% 100.00% #116
   500000 < 1000000:   0.00% 100.00% #6
  1000000 < 2500000:   0.00% 100.00% #21
  2500000 < 5000000:   0.00% 100.00% #9
  5000000 <    more:   0.00% 100.00% #18
---Stats - endToEnd (unit: ms)
  Avg: -14056368354582506 #4141359
        0 <       1:  75.26%  75.26% #3116825
        1 <       5:   0.00%  75.26% #0
        5 <      10:   0.00%  75.26% #0
       10 <      50:   0.00%  75.26% #0
       50 <     100:   0.00%  75.26% #0
      100 <     250:   0.00%  75.26% #0
      250 <     500:   0.00%  75.26% #0
      500 <    1000:   0.00%  75.26% #0
     1000 <    more:  24.74% 100.00% #1024534

Graph

select '$' as ticker from Market(ticker='$').win:length(1000).stat:weighted_avg('price', 'volume') ... (x1000)

1x 100 000 evt/s

Throughput 87367 (active 0 pending 0 cnx 1)

---Stats - engine (unit: ns)
  Avg: 3818 #863279
        0 <    5000:   0.42%   0.42% #3665
     5000 <   10000:  98.18%  98.61% #847606
    10000 <   15000:   1.30%  99.91% #11235
    15000 <   20000:   0.03%  99.94% #219
    20000 <   25000:   0.01%  99.94% #66
    25000 <   50000:   0.04%  99.99% #388
    50000 <  100000:   0.00%  99.99% #25
   100000 <  500000:   0.01% 100.00% #63
   500000 < 1000000:   0.00% 100.00% #2
  1000000 < 2500000:   0.00% 100.00% #0
  2500000 < 5000000:   0.00% 100.00% #0
  5000000 <    more:   0.00% 100.00% #10
---Stats - endToEnd (unit: ms)
  Avg: 15 #863396
        0 <       1:   0.75%   0.75% #6434
        1 <       5:   0.99%   1.74% #8552
        5 <      10:   2.12%   3.85% #18269
       10 <      50:  91.27%  95.13% #788062
       50 <     100:   0.10%  95.22% #827
      100 <     250:   4.36%  99.58% #37634
      250 <     500:   0.42% 100.00% #3618
      500 <    1000:   0.00% 100.00% #0
     1000 <    more:   0.00% 100.00% #0

2x 100 000 evt/s

Throughput 161871 (active 0 pending 0 cnx 2)

---Stats - engine (unit: ns)
  Avg: 3563 #1737956
        0 <    5000:  52.95%  52.95% #920318
     5000 <   10000:  45.42%  98.38% #789438
    10000 <   15000:   1.47%  99.84% #25494
    15000 <   20000:   0.12%  99.96% #2057
    20000 <   25000:   0.01%  99.97% #126
    25000 <   50000:   0.02%  99.99% #267
    50000 <  100000:   0.01%  99.99% #154
   100000 <  500000:   0.00% 100.00% #86
   500000 < 1000000:   0.00% 100.00% #0
  1000000 < 2500000:   0.00% 100.00% #3
  2500000 < 5000000:   0.00% 100.00% #1
  5000000 <    more:   0.00% 100.00% #12
---Stats - endToEnd (unit: ms)
  Avg: -1140698800016657 #1738081
        0 <       1:  27.06%  27.06% #470291
        1 <       5:   0.00%  27.06% #0
        5 <      10:   0.00%  27.06% #0
       10 <      50:  42.98%  70.04% #747055
       50 <     100:   0.01%  70.05% #152
      100 <     250:   2.93%  72.98% #50874
      250 <     500:   0.00%  72.98% #0
      500 <    1000:   0.00%  72.98% #0
     1000 <    more:  27.02% 100.00% #469709

3x 100 000 evt/s

---Stats - engine (unit: ns)
  Avg: 3571 #2652591
        0 <    5000:  70.08%  70.08% #1858844
     5000 <   10000:  28.41%  98.49% #753682
    10000 <   15000:   1.42%  99.91% #37692
    15000 <   20000:   0.07%  99.98% #1972
    20000 <   25000:   0.00%  99.99% #92
    25000 <   50000:   0.01%  99.99% #140
    50000 <  100000:   0.00% 100.00% #44
   100000 <  500000:   0.00% 100.00% #97
   500000 < 1000000:   0.00% 100.00% #2
  1000000 < 2500000:   0.00% 100.00% #5
  2500000 < 5000000:   0.00% 100.00% #1
  5000000 <    more:   0.00% 100.00% #20
---Stats - endToEnd (unit: ms)
  Avg: -1110728043020364160 #2653120
        0 <       1:  53.82%  53.82% #1427864
        1 <       5:   0.00%  53.82% #0
        5 <      10:   0.00%  53.82% #0
       10 <      50:  25.28%  79.10% #670731
       50 <     100:   0.27%  79.37% #7265
      100 <     250:   2.46%  81.84% #65326
      250 <     500:   0.10%  81.93% #2647
      500 <    1000:   0.00%  81.93% #0
     1000 <    more:  18.07% 100.00% #479287

Throughput 269388.718750 (active 0 pending 0 cnx 3)

4x 100 000 evt/s

75 Mbit/s
Throughput 375351 (active 0 pending 0 cnx 4)

---Stats - engine (unit: ns)
  Avg: 3746 #3767382
        0 <    5000:  77.24%  77.24% #2910019
     5000 <   10000:  21.46%  98.71% #808608
    10000 <   15000:   1.25%  99.96% #47191
    15000 <   20000:   0.02%  99.98% #808
    20000 <   25000:   0.00%  99.98% #154
    25000 <   50000:   0.01%  99.99% #308
    50000 <  100000:   0.00%  99.99% #76
   100000 <  500000:   0.00% 100.00% #146
   500000 < 1000000:   0.00% 100.00% #9
  1000000 < 2500000:   0.00% 100.00% #15
  2500000 < 5000000:   0.00% 100.00% #8
  5000000 <    more:   0.00% 100.00% #40
---Stats - endToEnd (unit: ms)
  Avg: -889816895069356670 #3767890
        0 <       1:  26.63%  26.63% #1003280
        1 <       5:   0.00%  26.63% #0
        5 <      10:   0.00%  26.63% #0
       10 <      50:   6.89%  33.51% #259464
       50 <     100:   9.58%  43.10% #361055
      100 <     250:   3.61%  46.71% #136085
      250 <     500:   0.00%  46.71% #0
      500 <    1000:   0.00%  46.71% #0
     1000 <    more:  53.29% 100.00% #2008006

After some more time

Throughput 412503 (active 0 pending 0 cnx 4)
94 Mbit/s

---Stats - engine (unit: ns)
  Avg: 2528 #4101107
        0 <    5000:  97.01%  97.01% #3978672
     5000 <   10000:   2.60%  99.62% #106669
    10000 <   15000:   0.35%  99.97% #14337
    15000 <   20000:   0.02%  99.99% #971
    20000 <   25000:   0.00%  99.99% #177
    25000 <   50000:   0.00% 100.00% #89
    50000 <  100000:   0.00% 100.00% #41
   100000 <  500000:   0.00% 100.00% #120
   500000 < 1000000:   0.00% 100.00% #2
  1000000 < 2500000:   0.00% 100.00% #7
  2500000 < 5000000:   0.00% 100.00% #5
  5000000 <    more:   0.00% 100.00% #18
---Stats - endToEnd (unit: ms)
  Avg: -2704829444341073400 #4101609
        0 <       1:  75.01%  75.01% #3076609
        1 <       5:   0.00%  75.01% #0
        5 <      10:   0.00%  75.01% #0
       10 <      50:   0.00%  75.01% #0
       50 <     100:   0.00%  75.01% #0
      100 <     250:   0.00%  75.01% #0
      250 <     500:   0.00%  75.01% #0
      500 <    1000:   0.00%  75.01% #0
     1000 <    more:  24.99% 100.00% #1025000

5x 100 000 evt/s with smaller event size

End to end latency tracking removed, event size reduced from 224 bit to 136 bit.

Throughput 519728.093750 (active 0 pending 0 cnx 5)
70 Mbit/s, 85% CPU

---Stats - engine (unit: ns)
  Avg: 2837 #5168150
        0 <    5000:  97.40%  97.40% #5033545
     5000 <   10000:   2.27%  99.66% #117113
    10000 <   15000:   0.31%  99.98% #16229
    15000 <   20000:   0.01%  99.99% #602
    20000 <   25000:   0.00%  99.99% #141
    25000 <   50000:   0.00%  99.99% #160
    50000 <  100000:   0.00%  99.99% #61
   100000 <  500000:   0.00% 100.00% #174
   500000 < 1000000:   0.00% 100.00% #4
  1000000 < 2500000:   0.00% 100.00% #35
  2500000 < 5000000:   0.00% 100.00% #33
  5000000 <    more:   0.00% 100.00% #53

Graph

with normal size event:

with event size reduced:

select '$' as ticker from Market(ticker='$').win:time(2 seconds).stat:weighted_avg('price', 'volume') output last every 30 seconds ... (x1000)

TODO

Esper performance index

select * from MarketData(symbol='$') output last every 30 seconds

3x 100 000 evt/s

Version at %CPU Throughput Latency
1.10 55% 308014 Avg: 2610 #3042094
0 < 5000: 85.70% 85.70% #2606926
5000 < 10000: 13.59% 99.28% #413321
1.7 70% 294078 Avg: 3572 #2954984
0 < 5000: 47.74% 47.74% #1410817
5000 < 10000: 51.29% 99.04% #1515656

1x 100 000 evt/s after a 3x run (ie optimized)

Version at %CPU Throughput Latency
1.10 12% 102231 Avg: 859 #1019076
0 < 5000: 99.18% 99.18% #1010678
5000 < 10000: 0.79% 99.96% #8023

select '$' as ticker from Market(ticker='$').win:length(1000).stat:weighted_avg('price', 'volume') output last every 30 seconds ... (x1000)

Version at %CPU Throughput Latency Notes

TODO

Performance on a laptop

You should be carrefull when measuring Esper performance on a laptop. The following shows two such runs and compares them to real server runs.

Configuration

1x dual core Intel T7400 2.16GHz (centrino duo), 4GB RAM
Windows XP SP2
JVM using -Xmx1600m (windows 32bit does not want to allocate 2g)

Simulation, no statements (laptop)

In this configuration our 2x CPU dual core 2GHz server hardware could handle -rate 14x100000 with a throughput at about 1 000 000 evt/s with Esper latency at 99.78% < 5us.

By contrast the laptop give us

-rate Throughput CPU Esper latency
3x100000 191 900 evt/s 70% 99,71% < 5us
4x100000 216 600 evt/s 100% 99,69% < 5us
2x150000 192 000 evt/s 75% 99,70% < 5us

So throughput is 5x less on this laptop, although latency is comparable.

Simulation, select '$' as ticker from Market(ticker='$').win:length(1000).stat:weighted_avg('price', 'volume') ... (x1000) (laptop)

In this configuration our 2x CPU dual core 2GHz server hardware could handle more than 400 000 evt/s, with 4 clients sending apprx 100 000 evt/s each, with Esper latency at 98.71% < 10us.

By contrast the laptop does:

-rate Throughput CPU Esper latency
2x50000 50200 evt/s 54% 99,13% < 25us (41,48% < 10us)
2x100000 72 800 evt/s 100% 99,61% < 50us (61,43% < <10us)

So throughput is again about 5x less on this laptop, latency is also largely impacted. Also the results here compares a laptop with simulated mode (no remote client thus no unmarshalling of events etc) to a server/client run over TCP. The actual difference is thus likely bigger.

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.