Follow by Email

Thursday, 19 March 2020

Incremental build with maven

This is 2020 and if you are starting any new java based project then gradle should be first option but for some reason if you are still stuck with Maven then you might find this post useful.

Image result for maven cartoon

Maven java/scala compiler plugin has decent support for incremental compilation but it is not able to handle few edge case like

  • Trigger compilation when file is deleted from source folder.
  • Skip unit test when code is not changed.

Just to handle deleted file scenario most of the time we have to run "mvn clean install" and that means full code is complied and unit test are executed. 

Compilation of scala code is slow and if project contain slow running test like starting webserver , spark context, IO etc then this becomes more worse. In many case wait time could be minutes.
I am not accounting for wasted CPU cycles for running test even when code is not changed.

As an experiment i took some ideas from Gradle and wrote add-on maven plugin that handles above stated issues by

 1. Cleaning target location when code is changed and trigger full build.
 2. Skip unit test execution when code is not changed.

Both of the these features can help in reducing compilation time significantly because most of the time only few modules are changed and previous build output can be used. You can get blazing fast builds by enabling this plugin.

How to use plugin
This plugin is added at pre-clean stage, add below entry to pom.xml and use "mvn pre-clean install" 


Plugin code is available @ compilerplugin github repo

sandbox code using plugin is available @ compilerplugintest github repo

Always collect metrics on build like how long it takes to compile , time taken by test , package size, dependency etc. Once you start measuring then you will notice how slow builds are and that also need same love as code.

Fast build is first step that enable continuous delivery.

Saturday, 14 March 2020

Hands on Optional value

Optional is in air due to coronavirus, everything is becoming optional like optional public gathering , optional work from home, optional travel etc.

Image result for option scala

I though it is good time to talk about real "Optional" in software engineering that deals with NULL reference.

Tony Hoare confessed that he made billion dollar mistake by inventing Null. If you have not seen his talk then i will suggest to have look at Null-References-The-Billion-Dollar-Mistake.

I will share some of the anti pattern with null and how it can be solved using abstraction like Optional or MayBe.

For this example we will use simple value object that can have some null values.

This value object can have null value for email & phone number.

Scenario: Contact Person on both email and phone number

Not using optional
First attempt will be based on checking null like below

This is how it has been done for years. One more common pattern with collection result.

Use optional in wrong way

This is little better but all the goodness of Optional is thrown away by adding if/else block in code.

Always Happy optional

It is good be happy but when you try that with Optional you are making big assumption or you don't need optional.

Nested property optional
For this scenario we will extend Person object and add Home property. Not everyone can own home so it is good candidate that it will be not available .
Lets see how contacting person scenario work in this case

This is where it start to become worse that code will have tons of nested null checks.

Priority based default
for this scenario we first try to contact person on home address and if it is not available then contact on office address.
Such type of scenario require use of advance control flow for early return and makes code hard to understand and maintain.

These are some of the common pattern where optional are not used or used in wrong way.

Optional usage patterns
Lets look at some of good ways of using optional.

Make property optional based on domain knowledge
It is very easy to makes property optional.

Yes it is allowed to make get Optional, no one will hang you for that and feel free to do that without fear. Once that change is done we can write something like below

It looks neat, first step to code without explicit if else on application layer.

Use some power of Optional

Optional is just like stream, we get all functional map,filter etc support. In above example we are checking for OptIn before contacting.

Always happy optional
Always happy optional that calls "get" without check will cause runtime error on sunday midnight, so it advised to use ifPresent

Nested Optional

Flatmap does the magic and handles null check for home and convert  insurance object also.

Priority based default

This example is taking both home & office address and pick the first one that has value for sending notification. This particular pattern avoids lots of nested loops.

Else branch
Optional has lots of ways to handle else part of the scenario like returning some default value(orElse) , lazy default value (orElseGet) or throw exception(orElseThrow).

What is not good about optional
Each design choice has some trade off and optional also has some. It is important to know what are those so that you can make careful decision.
Memory overhead
Optional is container that holds value, so extra object is created for every value. Special consideration is required when it holds primitive value. If some performance sensitive code will be impacted by extra object creation via optional then it is better to use null.

Memory indirection
As optional is container , so every access to value need extra jump to get real value. Optional is not good choice for element in array or collection.

No serialization
I think this is good decision by Jdk team that does not encourage people to make instance variable optional. You can wrap instance variable to Optional at runtime or when required for processing.

All the example used in post are available @ optionals github repo

If you like the post then you can follow me on twitter.

Monday, 20 January 2020

Better product by documenting trust boundary

Have you ever faced issue when you trusted system, team or product and that resulted in failure of feature or system?

In fast changing progressive delivery, definition of trust keeps on changing. Some of the trust issues that happens are.
  • Other system sends junk data.
  • Other system does not maintain constraint of data like unique , null , referential integrity.
  • Third party library causing unknown side effect.
  • Nitpicking user or QA.
  • Ignorant users.
  • Demanding product owner.
As engineer you have to continuously revisit trust boundary otherwise someone else will find gap and exploit system. Trust issues happens between teams or within single team, so it is important to establish trust boundary to smooth function of teams. 

Knowledge gaps plays big role in trust assumption that teams or individual make, best way of preventing trust issues are to document it as group that includes developers , QA and business. Once line is drawn then it becomes easy to take decision on different part of system.

The best part of agreeing on trust boundary is that it gives common framework for discussion and also avoids unproductive discussion.

So next time before starting to build any new feature or system , list down how much you trust other system or teams. Create triggers or alerts for every trust boundary violations ,so that you can react when something unexpected happens.

Bugs report also has lots of clue about trust boundary violation, so before fixing it revisit the trust boundary and make adjustment. 

Friday, 15 November 2019

Complexity Accidental vs Essential

Today it is hard to find team or organization that is not following agile but building software has not become easy, projects are missing schedule , over budget and it is also flawed.

Image result for software complexity"
Why it is so hard to build software ?

If you ask this questions to any engineer then 90%+ will say requirement, but is that the full truth ?

Lets try to decompose software construction. Every feature has 2 important component that decides whether feature will be successful or not.

  • Essential complication (ec)
  • Accidental complication( ac)

We will do some Functional programming refresher.

Feature = f( Essential Complication) + f(Accidental Complication )

Essential complication comes from domain like if you are building software for medical industry then it is complex. Accidental complication is complexity added by engineers, process & management to build feature.

Essential complication are hard to reduce because of domain, but to some extent it is possible to reduce by using good decomposition techniques. Decomposition is hard skill and comes from experiment of some fail projects.

Accidental complication can be controlled but it is not linear function, accidental complication is not same in every part of system and gets more complex over time. This also gives feedback on how much bad job we have done as engineer or product team.
Complexity comes in various forms like communication in team, less understanding , difficulty reusing some feature , extending program to new function, management problems etc.

Now we know accidental complication is exponential, so lets write formula again.

Feature = f( Essential complication) + f(Accidental complication * Unknown)) 

Now it will become clear why something takes many times longer than estimate or guess. Product owner also has part to play in adding accidental complication by marking assumption on importance of feature.

What can be done ?
If we need some predictability or consistence in delivery then we have to continuously work on reducing accidental complexity. Lets look at ways to keep to keep this in control.
  • Using higher level languages.
  • Incremental development by growing the software not building it.
  • Good buy vs build decision. 
  • Unified programming environment. 
  • Raid prototype to refine requirement.
  • Listen to design pressure.
  • Test driven development.
  • Stop "Get it out of the door" mindset.
  • Reduce "surgical strike effort" in delivery.
Very insightful quote from Frederic Brooks, both customer and engineer has to learn what to ask, expect, and commit otherwise only option is broken system.

“An omelette, promised in two minutes, may appear to be progressing nicely. But when it has not set in two minutes, the customer has two choices—wait or eat it raw. Software customers have had the same choices. The cook has another choice; he can turn up the heat. The result is often an omelette nothing can save—burned in one part, raw in another.”
― Frederick P. Brooks Jr., The Mythical Man-Month: Essays on Software Engineering

Mythical-Man-Month by Mr Brooks is must read for every product owner , project manager and engineer.

If you like the post then you can follow me on twitter.

Tuesday, 12 November 2019

All models are wrong

Image result for all models are wrong
Above quote by George Box is commonly used in Statistic and Machine learning but this post has nothing to do with Machine Learning!

I am going to share some thing about leadership models not machine learning model.

Some of the extremely successful leaders like Steve JobsSteve BallmerJeff bezosLarry Ellison , Elon Musk , Jack Mamark zuckerberg has played key role in transformation of our industry and life but do you want to use their leadership style or want to work with them ?

All of these leader are very successful and they came up with ideology,principles that worked in the system they were in and as engineer we know each system is optimized for very specific goals.

So before you start taking some of the traits from any leader you have also think about the system in which they we using that style and if you are not in that type of system then this leadership is going to do more damage than help.

With that we have to rephrase George Box statement to include leadership 

All leadership models are wrong but some are useful.

Today leadership has to be more open,collaborative and inclusive. 

Leadership comes after followership but many time we think that it is boolean, you are either leader or follower, being good follower is first stage of leadership and you have work on continuous improvement of followership skills.

Leadership and followership are two side of same coin but many leadership discussion, books etc puts very less focus on followership, so as individual you never deliberately practice follower skill. 


Image result for followership"

Followership has been studied for years and has many models, wikipedia link has good summary of the common ones and remember each model is optimized for system in which it works.
One of the model proposed by Ira Chaleff is very interesting, it divide followers in 4 types and later 5th one was also added.

As per him model looks like this
Image result for ira chaleff followership"

It has 2 dimension support & challenge.

Resource is low support and low challenge. They do what is asked for, they do enough to retain position.

Implementer is high support and low challenge. This is 10x engineer in team, always up for anything required to meet deadline. Leaders love this type of follower but side effect is that they will not speak up when things does not make sense or are not strategic. 

Individualists is low support & high challenge. This type has "Me" mindset and shows hostility towards someone or something. This type of follower are difficult to manage and needs constant realignment. 

Partners is high support & high challenge. This is sweet spot and very hard to find. This type of follower challenge leaders to make sure they understand big picture of team or organization. Once they are aligned then they take full responsibility and work whole heart to make it happen. This type of individual are good candidate for moving into leadership role.

Both leadership and followership are important to be successful and having self awareness of what type of leader & follower you are is very important. This gives nice model to understand people and their behavior. 

Sunday, 3 November 2019

Don’t comment bad code—rewrite it

In this post i will share my experience of "comments in code" that i have got by reading code , writing code and reading books.

Lets start with famous quote
“Don’t comment bad code—rewrite it.”
  —Brian W. Kernighan and P. J. Plaugher

Lots of comment in code looks just like above image and it is distraction.

Comments are lie 
Most of the time comments are not in sync with code, no one cares about it so no one maintains it.

Code is refactored
Code is under constant improvement, it changes , evolves and dies but code comments just sticks and become orphan.

Bad code has more comments
Once we write code that is hard to understand then we try to explain it by comments and it tells lot about our failure to write expressive code.

Noisy comment
Lets look at at some code+comment from public domain.

* Create a new HdfsAdmin client.
* @param uri the unique URI of the HDFS file system to administer
* @param conf configuration
* @throws IOException in the event the file system could not be created
public HdfsAdmin(URI uri, Configuration conf) throws IOException {
FileSystem fs = FileSystem.get(uri, conf);
if (!(fs instanceof DistributedFileSystem)) {
throw new IllegalArgumentException("'" + uri + "' is not an HDFS URI.");
} else {
dfs = (DistributedFileSystem)fs;

This code snippet is from popular open source java project, many comment in enterprise is also like this. This adds more noise than value.

Journal comments
Many scripting code, data base code, python code is based on this pattern, this looks like team has no trust on source control and they have taken over this responsibility. It looks something like below

01-Apr-2019 - First version with skeleton code by someone
10-Apr-2019 - Config support added  by someone

Marker comments
This starts with something like below

// Starting init

//Make DB call 

this is also very common in scripting, DB stored procedure or in real enterprise type programs. This patterns tell that it is shouting to extract small function but gets overlooked.

Commented code
I am sure you have seen this many time and got frustrated that why this dead code exists.
It looks something like this.

//Keeping this for backup


Author code
IDE adds block of comments to every file that is added to project, it looks like below



Author names is just at the top of the file but he no more maintains it and does not work for team or organization but occasionally gets credit/blame for his creation.

Code Comment mostly adds no value, it comes between you and code, it is one extra distraction.It also promotes bad habit for young programmer because they think "adding comments makes code better".

In some case comments are useful when you are writing public APIs but still be careful on what you write, software engineers are smart they can understand if code does not have accidental complexity.

I will leave you with another famous tweet by Kevlin Henney

Uncle bob has whole chapter in Clean Code: A Handbook of Agile Software Craftsmanship about comments. I recommend to read chapter, it covers things in more details.

If you like the post then you can follow me on twitter.

Wednesday, 2 October 2019

Progressive delivery

Coupling is one of the important pattern for building software and it is recommended to build loosely coupled systems. 
Lets talk about loose coupling pattern in software deployment context.

Image result for evolution

Industry has seen evolution in Devops space in form of continuous integration,continuous delivery, automated testing etc but world is moving fast and we need CI/CD++.  Next step after CI/CD is Progressive delivery.

Deploying  != Release
Progressive delivery could sound like cool tech jargon but it simply comes down to drawing line between Deploying in productions vs Activating in production.

This new way of deploying software gives control on how do you want to roll out new features to customers. It gives control on whether to upset all users vs small set of users to get some feedback before whole world knows about it.

A version of progressive delivery is also called Blue-Green deployment, A/B testing , canary testing, feature toggle etc.

Benefit of progressive delivery 
- Low ceremony way of enabling features to user.
- Safe way of testing things in production.
- Less git conflicts to manage because code does not sit in git after it is developed.
- Better operational management because deployment is done in Parts. Your support team will love it.
- Avoid frustration of working on noisy Pre production.  
- Product team are empowered to decided when they want to open it for external users.

Last with not least that Progressive delivery methodology allow team to control blast radius when rolling out new software.

Share your progressive delivery experience. 

If you like the post then you can follow me on twitter.