Sunday 7 July 2013

How To Write Micro benchmark In Java

So many article has been written on how to write micro-bench mark in java, this blog is my attempt to explain the topic.

What is Micro benchmark
Micro benchmark try to measure actual speed of small piece of code, measuring performance is very complex because JIT does magic to make code fast and you have to measure the code after all the JIT optimization is done.

What factors should be considered for benchmark 
Some of the factors that you should consider for correct bench-marking are

  • Warm-up phase - This is to start measuring only after JIT has compiled hot code, JIT will compile functions only after 10K invocation.
  • Client vs Server - Running JVM in client or server mode will give different performance result
  • GC  - Garbage Collector makes measurement really difficult 
  • Re-compilation effect - Code executing first time will be slow
More details are available @ MicroBenchmarks Rules

How to write micro benchmark
This can be very difficult to write but not to worry, now there are some framework available to do that
 - Caliper  - It is Google solution for writing benchmark
 - JMH   - This one is from java guys, in this blog i will explain some of example using JMH

How to use JMH
JMH is maven based project but don't know the reason why it is not hosted on maven central.
You need TortoiseHg to download code, once you have the code then it is cake walk to setup/build the JMH project.
Refer to Setup page for detail instruction on setup.

Lets look at some examples
As every programming language starts with hello world, so does JHM .
JMH has used annotation based approach to specify which function you want to run as part of benchmark,  GenerateMicroBenchmark annotation is used for that.

-Empty function 
JVM is complex piece of software and it has some overhead, having empty function checks that overhead
Numbers that come outs with this test are 

Run result "wellHelloThere": 2893313.782 ▒(95%) 118441.448 ▒(99%) 161902.046 ops
/msec
Run statistics "wellHelloThere": min = 2388762.991, avg = 2893313.782, max = 310
8116.404, stdev = 253075.135
Run confidence intervals "wellHelloThere": 95% [2774872.334, 3011755.229], 99% [
2731411.736, 3055215.827]

This is interesting number, if you have some empty function then it can be called 2.8 million times per mili second, that is very fast.

JHM does all the magic and shows number that we are interested in. JMH is based on code generation, it generate lots of code to measure performance, all the generated code is available under "generated-sources"

Have look at generated file

That is lot of code!

-Benchmark modes
JHM has benchmark modes, very useful, it has most of the mode that you can think of for eg
 - Throughput
- AverageTime
 - SampleTime - Time distribution, percentile estimation
- SingleShotTime

I must say very well thought of by java team.

-States
This is very common use case , this test measure overhead of shared object vs per thread object. Result are interesting.
Run result "measureUnshared": 1264160.208 (95%) 20349.174 (99%)
Run result "measureShared": 965178.728 (95%) 20847.118 (99%)

Unshared test is around 30% fast as compared to shared & it is due to contention on memory .

-DeadCode
This benchmark measure effect of dead code, very common case.
Lets look at sample code
Results are interesting in this case, JVM is very smart enough to remove dead code from call and time take for function as good as there is no call to log function.

-Constant Folding
This is another interesting optimization that is done by compiler,Constant folding techniques will optimize call to functions that takes constant parameter,sometime this can result in tricky defects.
Way to avoid trap is read inputs via some state. Lets have look at sample code

measureWrong function is faster because JVM performs optimization based on constant folding rules, but be very cautious this an cause some defect in you code!

-Multiple Implementation
This is very interesting test, this test has multiple implementation of counter interface, code is same but just 2 implementation and result are random, you can't conclude why one implementation fast as compared to other.

JVM will mix profile of two test and due to which result are random. JMH has option to measure this type of test properly, there is annotation Fork that will execute benchmark in new JVM.

- Compile Control
JMH has support to measure effect of inline, no inline, exclude compile. These things are really useful to find performance number under different scenario. Sample benhmark for compile control

- Asymmetric
JMH has very good support for asymmetric execution for eg producer/consumer, in traditional test we have to write thread related code to make this happen, but JMH has declarative way of specifying such behavior, multiple benchmark methods can be grouped by using Group annotation.
Sample code for grouping.

Conclusion
JHM is very useful tool for micro bench marking, it is worth spending time to move some of the performance test of application to JHM.
Writing benchmark will give very good understanding of how JVM works.

23 comments:

  1. It’s very helpful for us, thank you so much for sharing such an amazing article. Visit Ogen Infosystem for top Website Designing and PPC Services in Delhi at an affordable price.
    Website Designing Company in India

    ReplyDelete
  2. Awesome post with lots of data and I have bookmarked this page for my reference. Share more ideas frequently.

    Java Training in Chennai

    Java Course in Chennai

    ReplyDelete
  3. Are you looking for the best Web design company in pune? If yes then connect with us to get an unique web design for your organization. Get all types of website design and development services by hiring CSS Founder.

    ReplyDelete
  4. We are a company CSS Founder that is known as the best website designing company in Lucknow. Connect with if you want re-design or re-develop your existing website.

    ReplyDelete
  5. Very nice information you have shared by posting this nice article and blog. Thank you for sharing this keep posting so that every one get knowledge by reading your nice article.
    Residential projects in sector 79 Noida

    ReplyDelete
  6. Amazing post, thank you for sharing this valuable and interesting post. Keep posting like this. Visit Al mumtaz tents in dubai and get the best tents for your party and events.
    Tent rental in Abu Dhabi

    ReplyDelete
  7. Do you know to buy smartphone insurance is always a good for you if you are a careless person towards your phone.

    ReplyDelete
  8. We are known as the best website design & development service provider all over the world. You can connect with us for more information. Get the best deal for your website.
    website design company in chennai

    ReplyDelete
  9. You are providing such a piece of nice information to us. Keep it up!
    Brackish Water Desalination 80000 GPD R.O. System Djibouti

    ReplyDelete
  10. you've got achieved a omnipresent process upon this article. Its wholly proper and intensely qualitative. you've got even managed to make it readable and clean to access. you have a few valid writing function. thank you consequently much. Windows 7 Ultimate Product Key

    ReplyDelete
  11. Thanks for the informative and helpful post, obviously in your blog everything is good. Website design company in Coimbatore

    ReplyDelete
  12. We understand your business goals and deliver them according to your business values. Thousands of people have trusted CSS Founder and in today's time have given their business a new identity with the help of a good Website click here for contact us Website design company tauranga

    ReplyDelete
  13. We are the best website design company that provides services in this digital world, if you want to increase your business through a website, and you need the best website design and development company then without delay contact us. Go to the given link Website design company in Groningen

    ReplyDelete
  14. IB Solar is a Tubular Battery provider in India. We provide the best quality solar batteries at the best price. Our company is one of the leading solar battery manufacturers in India Solar Energy Converter

    ReplyDelete

  15. Artattackk is a well-knowned Website Designing Company In Delhi having 11+years of experience.

    ReplyDelete
  16. Nice post. Thanks for sharing! I want people to know just how good this information is in your article. It’s interesting content and Great work.
    IT training and job placement

    ReplyDelete
  17. If you are looking for a journal where you can connect with your peers, IJRASET is the right place. The journal has a very good reputation for publishing high-quality research papers. The journal also publishes research papers from various disciplines including engineering, chemistry, biology, and arts. Best Journals to Publish Research Papers, Paid Journal

    ReplyDelete
  18. Best Job Consultants company In India Thanks for providing this nice information post, I wait for your next post. We are the best job consultant company in India to hire the right person for the right vacancy. Our team of experts has come up with the truth and the latest updated knowledge. Contact us or call us, we will help you in searching of your dream job. Our company name is nayratalents which is also known as the best Job Consultants company in India,

    ReplyDelete
  19. Yemen Visa Thank you for providing this wonderful information blog I look forward to your next post Hello everyone we are providing the best visa services if you are searching for the best visa service company then you must trust us once Have to do, to contact us you can search visits visa this name on Google, or you can contact us through the given resource Thank you

    ReplyDelete
  20. Whether you're a student looking for direction or an institution seeking support, our team is dedicated to providing the best consultation services. Get in touch with Gev Consultants today. study visa consultants

    ReplyDelete