Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 42 Next »

The need for performance metrics and comparison

Since we released AspectWerkz 1.0, and more generally for every release of any AOP / interceptor framework (AspectWerkz, AspectJ, JBoss AOP, Spring AOP, cglib etc), a question is always raised: "what is the performance cost of such an approach ?", "how much do I loose per method invocation when an advice / interceptor is applied ?".

This is indeed an issue that needs to be carefully addressed, and that in fact has affected the design of every mature enough framework.

We are probably all scared by the cost of the java.lang.reflect despite its relative power, and usually, even before starting to evaluate semantics robustness and ease of use in general - we start doing some hello world bench.

We have started AWbench for that purpose. Offering a single place to measure the relative performance of AOP / interceptor framework, and even measure it by your own.

More than providing performance comparison, AWbench is a good place to figure out the semantics differences and ease of use of each framework by using them for the same rather simple purpose. A "line of count" metrics will be provided in a next report.

Current performance results

This table provides the figures from a bench in "nanosecond per advised method invocation". A non advised method invocation is roughly about 5 ns/iteration on the bench hardware/software that was used.

The results were obtained with 2 millions iterations.

In this table, the two first lines in bold are the most important ones. In a real world application, it is likely that the before or around advice will really do something and for such needs to access runtime information like method parameter and target instance. It is likely as well that the join point is affected by more than one advice.

On the opposite it is very unlikely to have just a before advice that does nothing, but it gives us a good evaluation on the most minimal overhead we can expect.

Note: comparing such results when the difference is small (f.e. 15 ns vs 10 ns) might not be relevant. Before doing so you should run the bench several time and compute an average after removing the smallest and highest measurements.

AWBench (ns/invocation)

aspectwerkz

aspectwerkz_1_0

aspectj

jboss

spring

dynaop

cglib

ext:aopalliance

ext:spring

ext:aspectj

before, args() target()

10

576

10

215

591

2463

135

-

210

-

around x 2, args() target()

85

641

45

290

681

4647

150

461

461

-

before

10

476

15

130

526

320

75

-

40

10

before, static info access

30

480

20

130

520

305

75

-

40

-

before, rtti info access

50

506

50

135

526

2373

70

-

35

-

after returning

10

475

10

165

535

325

80

-

40

15

after throwing

3360

5468

2984

5187

-

6604

7991

-

-

3440

before + after

20

480

15

245

671

340

85

-

40

20

before, args() primitives

10

546

10

260

580

370

135

-

200

-

before, args() objects

5

540

15

185

561

350

105

-

195

-

around

60

470

10

125

460

310

75

70

70

85

around, static info access

90

480

25

125

481

330

75

70

70

-

around, rtti info access

85

506

50

140

495

2433

85

70

75

-


This table provides the figures from the same bench where for each category AspectWerkz 2.0.RC1 is the reference.
The first line illustrates that for the most simple before advice, AspectWerkz is 13 times faster than JBoss AOP 1.0.

AWBench (relative %)

aspectwerkz

aspectwerkz_1_0

aspectj

jboss

spring

dynaop

cglib

ext:aopalliance

ext:spring

ext:aspectj

before, args() target()

1 x

57.6 x

1 x

21.5 x

59.1 x

246.3 x

13.5 x

-

21 x

-

around x 2, args() target()

1 x

7.5 x

0.5 x

3.4 x

8 x

54.6 x

1.7 x

5.4 x

5.4 x

-

before

1 x

47.6 x

1.5 x

13 x

52.6 x

32 x

7.5 x

-

4 x

1 x

before, static info access

1 x

16 x

0.6 x

4.3 x

17.3 x

10.1 x

2.5 x

-

1.3 x

-

before, rtti info access

1 x

10.1 x

1 x

2.7 x

10.5 x

47.4 x

1.4 x

-

0.7 x

-

after returning

1 x

47.5 x

1 x

16.5 x

53.5 x

32.5 x

8 x

-

4 x

1.5 x

after throwing

1 x

1.6 x

0.8 x

1.5 x

-

1.9 x

2.3 x

-

-

1 x

before + after

1 x

24 x

0.7 x

12.2 x

33.5 x

17 x

4.2 x

-

2 x

1 x

before, args() primitives

1 x

54.6 x

1 x

26 x

58 x

37 x

13.5 x

-

20 x

-

before, args() objects

1 x

108 x

3 x

37 x

112.2 x

70 x

21 x

-

39 x

-

around

1 x

7.8 x

0.1 x

2 x

7.6 x

5.1 x

1.2 x

1.2 x

1.2 x

1.4 x

around, static info access

1 x

5.3 x

0.2 x

1.3 x

5.3 x

3.6 x

0.8 x

0.7 x

0.8 x

-

around, rtti info access

1 x

5.9 x

0.5 x

1.6 x

5.8 x

28.6 x

1 x

0.8 x

0.8 x

-

Bench were run on a Java HotSpot 1.4.2, Windows 2000 SP4, Pentium M 1.6 GHz, 1 Go RAM.

Notes:

  • Some figures are not available when the underlying framework does not allow the feature. For the ext: ones, that can be due to pending work (AOP alliance interfaces can emulate a before advice just as it is the case in JBoss AOP).
  • after throwing advice appears to be slow since it is using an instanceof test at each invocation (to catch only runtime exceptions)

AWbench internals

Summary

AWbench is a micro benchmark suite, which aims at staying simple. The test application is very simple, and AWbench is mainly the glue around the test application that applies one or more very simple advice / interceptor of the framework of your choice.

AWbench comes with an Ant script that allows you to run it on you own box, and provide some improvement if you know some for a particular framework.

What is the scope for the benchmark?

So far, AWbench includes method execution pointcuts, since call side pointcuts are not supported by proxy based framework (Spring AOP, cglib, dynaop etc).

The awbench.method.Execution class is the test application, and contains one method per construct to bench. An important fact is that bytecode based AOP may provide much better performance for before advice and after advice, as well as much better performance when it comes to accessing contextual information.
Indeed, proxy based frameworks are very likely to use reflection to give the user access to intercepted method parameters at runtime from within an advice, while bytecode based AOP may use more advanced constructs to provide access at the speed of a statically compiled access.

The current scope is thus:

For method execution pointcut

Construct

Contextual information access

Notes

 

 

 

before advice

none

 

before advice

static information (method signature etc)

 

before advice

contextual information accessed reflectively

Likely to use of casting and unboxing of primitives

before advice

contextual information accessed with explicit framework capabilities

Only supported by AspectJ and AspectWerkz 2.x

 

 

 

after advice

none

 

after returning advice

return value

 

after throwing advice

exception instance

 

 

 

 

before + after advice

none

 

 

 

 

around advice

optimized

AspectJ and AspetWerkz 2.x provides specific optimizations (thisJoinPointStaticPart vs thisJoinPoint)

around advice

non optimizezd

 

 

 

 

2 around advice

contextual information

 

By accessing contextual information we means:

  • accessing a method parameter using its real type (i.e. boxing and unboxing might be needed)
  • accessing a the advised instance using its real type (i.e. casting might be needed)

A pseudo code block is thus likely to be:

Which AOP and Proxy frameworks are benched?

The following are included in AWbench:

Bytecode based frameworks

Proxy based frameworks

Framework

URL

Spring AOP (1.1.1)

http://www.springframework.org/

cglib proxy (2.0.2)

http://cglib.sourceforge.net/

dynaop (1.0 beta)

https://dynaop.dev.java.net/

Moreover, AWbench includes AspectWerkz Extensible Aspect Container that allow to run any Aspect / Interceptor framework within the AspectWerkz 2.x runtime:

AspectWerkz Extensible Aspect Container running

Notes

AspectJ

 

AOP Alliance

http://aopalliance.sourceforge.net/

Spring AOP

 

AWbench is extensible. Refer to Contributing to add a framework.

What's next ?

Running awbench by your own

AWBench is released under LGPL.
There will never be a distribution of it, but source can be checked out:

Once checked out, you can run the bench using several different Ant target

Contributing

If you notice some optimizations for one of the implementation by respecting the requirements, we will add the fix in awbench and update the results accordingly.

  • No labels