Complexity Accidental vs Essential

Today it is hard to find team or organization that is not following agile but building software has not become easy, projects are missing schedule , over budget and it is also flawed.


Why it is so hard to build software ?

If you ask this questions to any engineer then 90%+ will say requirement, but is that the full truth ?

Lets try to decompose software construction. Every feature has 2 important component that decides whether feature will be successful or not.

• Essential complication (ec)

• Accidental complication( ac)

We will do some Functional programming refresher.

Feature = f( Essential Complication) + f(Accidental Complication )

Essential complication comes from domain like if you are building software for medical industry then it is complex. Accidental complication is complexity added by engineers, process & management to build feature.

Essential complication are hard to reduce because of domain, but to some extent it is possible to reduce by using good decomposition techniques. Decomposition is hard skill and comes from experiment of some fail projects.

Accidental complication can be controlled but it is not linear function, accidental complication is not same in every part of system and gets more complex over time. This also gives feedback on how much bad job we have done as engineer or product team.
Complexity comes in various forms like communication in team, less understanding , difficulty reusing some feature , extending program to new function, management problems etc.

Now we know accidental complication is exponential, so lets write formula again.

Feature = f( Essential complication) + f(Accidental complication * Unknown)) 

Now it will become clear why something takes many times longer than estimate or guess. Product owner also has part to play in adding accidental complication by marking assumption on importance of feature.

What can be done ?
If we need some predictability or consistence in delivery then we have to continuously work on reducing accidental complexity. Lets look at ways to keep to keep this in control.

• Using higher level languages.
• Incremental development by growing the software not building it.
• Good buy vs build decision. 
• Unified programming environment. 
• Raid prototype to refine requirement.
• Listen to design pressure.
• Test driven development.
• Stop "Get it out of the door" mindset.
• Reduce "surgical strike effort" in delivery.

Very insightful quote from Frederic Brooks, both customer and engineer has to learn what to ask, expect, and commit otherwise only option is broken system.

“An omelette, promised in two minutes, may appear to be progressing nicely. But when it has not set in two minutes, the customer has two choices—wait or eat it raw. Software customers have had the same choices. The cook has another choice; he can turn up the heat. The result is often an omelette nothing can save—burned in one part, raw in another.”
― Frederick P. Brooks Jr., The Mythical Man-Month: Essays on Software Engineering

Mythical-Man-Month by Mr Brooks is must read for every product owner , project manager and engineer.

If you like the post then you can follow me on twitter.

All models are wrong

Above quote by George Box is commonly used in Statistic and Machine learning but this post has nothing to do with Machine Learning!

I am going to share some thing about leadership models not machine learning model.

Some of the extremely successful leaders like Steve Jobs , Steve Ballmer, Jeff bezos, Larry Ellison , Elon Musk , Jack Ma , mark zuckerberg has played key role in transformation of our industry and life but do you want to use their leadership style or want to work with them ?

All of these leader are very successful and they came up with ideology,principles that worked in the system they were in and as engineer we know each system is optimized for very specific goals.

So before you start taking some of the traits from any leader you have also think about the system in which they we using that style and if you are not in that type of system then this leadership is going to do more damage than help.

With that we have to rephrase George Box statement to include leadership 

All leadership models are wrong but some are useful.

Today leadership has to be more open,collaborative and inclusive. 

Leadership comes after followership but many time we think that it is boolean, you are either leader or follower, being good follower is first stage of leadership and you have work on continuous improvement of followership skills.

Leadership and followership are two side of same coin but many leadership discussion, books etc puts very less focus on followership, so as individual you never deliberately practice follower skill. 

Followership

Followership has been studied for years and has many models, wikipedia link has good summary of the common ones and remember each model is optimized for system in which it works.

One of the model proposed by Ira Chaleff is very interesting, it divide followers in 4 types and later 5th one was also added.

As per him model looks like this

It has 2 dimension support & challenge.

Resource is low support and low challenge. They do what is asked for, they do enough to retain position.

Implementer is high support and low challenge. This is 10x engineer in team, always up for anything required to meet deadline. Leaders love this type of follower but side effect is that they will not speak up when things does not make sense or are not strategic. 

Individualists is low support & high challenge. This type has "Me" mindset and shows hostility towards someone or something. This type of follower are difficult to manage and needs constant realignment. 

Partners is high support & high challenge. This is sweet spot and very hard to find. This type of follower challenge leaders to make sure they understand big picture of team or organization. Once they are aligned then they take full responsibility and work whole heart to make it happen. This type of individual are good candidate for moving into leadership role.

Conclusion

Both leadership and followership are important to be successful and having self awareness of what type of leader & follower you are is very important. This gives nice model to understand people and their behavior. 

Don’t comment bad code—rewrite it

In this post i will share my experience of "comments in code" that i have got by reading code , writing code and reading books.

Lets start with famous quote

“Don’t comment bad code—rewrite it.”
  —Brian W. Kernighan and P. J. Plaugher

Lots of comment in code looks just like above image and it is distraction.

Comments are lie 
Most of the time comments are not in sync with code, no one cares about it so no one maintains it.

Code is refactored
Code is under constant improvement, it changes , evolves and dies but code comments just sticks and become orphan.

Bad code has more comments
Once we write code that is hard to understand then we try to explain it by comments and it tells lot about our failure to write expressive code.

Noisy comment
Lets look at at some code+comment from public domain.

https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java

/**
	* Create a new HdfsAdmin client.
	*
	* @param uri the unique URI of the HDFS file system to administer
	* @param conf configuration
	* @throws IOException in the event the file system could not be created
	*/
	public HdfsAdmin(URI uri, Configuration conf) throws IOException {
	FileSystem fs = FileSystem.get(uri, conf);
	if (!(fs instanceof DistributedFileSystem)) {
	throw new IllegalArgumentException("'" + uri + "' is not an HDFS URI.");
	} else {
	dfs = (DistributedFileSystem)fs;
	}
	}

This code snippet is from popular open source java project, many comment in enterprise is also like this. This adds more noise than value.

Journal comments
Many scripting code, data base code, python code is based on this pattern, this looks like team has no trust on source control and they have taken over this responsibility. It looks something like below

/*
01-Apr-2019 - First version with skeleton code by someone
10-Apr-2019 - Config support added  by someone
....
...
*/ 

Marker comments
This starts with something like below

// Starting init

//Make DB call 

this is also very common in scripting, DB stored procedure or in real enterprise type programs. This patterns tell that it is shouting to extract small function but gets overlooked.

Commented code
I am sure you have seen this many time and got frustrated that why this dead code exists.
It looks something like this.

//Keeping this for backup
//fastCode()


superFastCode() 

Author code
IDE adds block of comments to every file that is added to project, it looks like below

/*

@Ashkrit
*/

Author names is just at the top of the file but he no more maintains it and does not work for team or organization but occasionally gets credit/blame for his creation.



Conclusion
Code Comment mostly adds no value, it comes between you and code, it is one extra distraction.It also promotes bad habit for young programmer because they think "adding comments makes code better".

In some case comments are useful when you are writing public APIs but still be careful on what you write, software engineers are smart they can understand if code does not have accidental complexity.

I will leave you with another famous tweet by Kevlin Henney

Uncle bob has whole chapter in Clean Code: A Handbook of Agile Software Craftsmanship about comments. I recommend to read chapter, it covers things in more details.

If you like the post then you can follow me on twitter.

Progressive delivery

Coupling is one of the important pattern for building software and it is recommended to build loosely coupled systems. 

Lets talk about loose coupling pattern in software deployment context.

Industry has seen evolution in Devops space in form of continuous integration,continuous delivery, automated testing etc but world is moving fast and we need CI/CD++.  Next step after CI/CD is Progressive delivery.

Deploying  != Release

Progressive delivery could sound like cool tech jargon but it simply comes down to drawing line between Deploying in productions vs Activating in production.

This new way of deploying software gives control on how do you want to roll out new features to customers. It gives control on whether to upset all users vs small set of users to get some feedback before whole world knows about it.

A version of progressive delivery is also called Blue-Green deployment, A/B testing , canary testing, feature toggle etc.

Benefit of progressive delivery 

- Low ceremony way of enabling features to user.

- Safe way of testing things in production.

- Less git conflicts to manage because code does not sit in git after it is developed.

- Better operational management because deployment is done in Parts. Your support team will love it.

- Avoid frustration of working on noisy Pre production.  

- Product team are empowered to decided when they want to open it for external users.

Last with not least that Progressive delivery methodology allow team to control blast radius when rolling out new software.

Share your progressive delivery experience. 

If you like the post then you can follow me on twitter.

Tracer bullet software development

Software development methodology is evolving very fast and every team has found version that works well in current context.

Software development methodology is going through continuous improvement.

The_Pragmatic_Programmer book talks about Trace bullet approach and as per that it comes down to feedback. The more quickly you get feedback, less change is required to hit the target.

I have used Tracer Bullet approach through out my career with good success.

I will share another approach or mindset of software development that can be used in some scenario to build better product and create value.

Explore or Discovery 


Many time as team you have to find what is the next feature you should be building that will help in increasing adoption of product.
So i call this as Explore or Discover phase and this phase of development has very different goal and trade off. This phase means that you have to move fast, cut some corners to get feedback. This phase will not have enough tests, documentation, code quality is not good etc.

Important thing about this phase is you are actively collecting feedback on whether this is the next big idea your team will be investing.
In this phase you have to make sure you gain more than loose, so timebox discovery phase to keep track on resource that is consumed.

Outcome of successful discovery puts you in Growth phase and in many case this will be steep growth, but every unsuccessful outcome has lots of signal on next exploration.

 You Build to learn in this phase and once you learned then move to next phase or start another discovery.

Growth Phase


This phase is outcome of successful discovery and now you have found the next feature that market or your target audience need.
Trade off for this phase is very different from Explore phase, you have to stabilized feature , do changes based on feedback so users who have shown interest in idea are still engaged. Users who got on-boarded work like your sponsor, so keep them in loop.

Interesting things about this phase is your team will be in war room type of situation, everyone is trying to get over huddles and get feature out.

Word of caution this phase very intense and demanding. This phase is the real "Sprint" phase not the agile sprint! putting extra hours has good returns.
Another thing to watch out is to be persistent in exploiting maximum of new feature but many time team drops the ball and go back to discovery phase. 

I would say this phase also puts design pressure on team and you can refer to design-pressure-on-engineering-team post that talks about it.

Successful outcome of this phase is Expand phase. Team is exhausted after this phase but very motivated.

Expand Phase


Welcome to phase that requires building software in the way we learn in text book, this is the phase where engineering discipline are very important because solution has to be scalable, maintainable , reliable etc.
Now you can go to management and ask for more funds to get servers , expand team etc because this idea will generate some profit.

Don't build product like discovery phase in this stage otherwise you will become victim of your own success, below image should be good example of it.



Conclusion
All of these phases has very different constraint and trade off and it is very important to know that, expand phase can't be managed like explore phase or vise versa and i have also found that engineering disciplined are very very different in these phase, so you need team that is aligned with mindset. If you get your team wrong then it can really very challenging to execute phase.

It is fascinating to see that our industry keeps on evolving and new development methodology are found.

It is much more than Agile or scrum.

I want to end with quote
As engineering team we should do continuous exploration and exploration is not linear process.

Some of other post on software development that you might interesting

need-driven-software-development-using
broken-promise-of-agile
cargo-cult-innovation-center

I will be happy to learn about new ways of building software, so please share it!

If you like the post then you can follow me on twitter.

frontend vs backend development

I am sure you might have got into discussion of frontend vs backend development or some engineers want to do only one type of development.




I am passionate engineer for more than decade and have got opportunity to see full spectrum.  I have done development on extreme end of both the side.

In this post i will share something on front-end engineering that is hidden and things that is not discussed openly.

Lets start ..

Back-end 
In this part of the work every one understands that it is a world of data structure , algorithm and writing bare metal code, handling scale etc.


Front-end
In this part of world most of us think is only about building engaging and secure User Interface, but i will say it is just one part of it.
As a frontend developer you are exposed to so many engineering challenges. Lets discuss about it.

Getting started
For any backend related task you can write the quick code or that code can come from Stackoverflow and go and run from IDE.

Frontend getting code is just small part but after that you need some webserver/container to host the code and then know the browser/client that you want to use and then finally code runs.

This is just small example and you can get idea of extra number of steps required to see your code running.

Synchronous vs Asynchronous 
On backend system you have option to choose if code is Sync vs Async but on frontend most of the operations has to be asynchronous otherwise end user experience is very bad.

On frontend you are exposed to this on day one but on backend it will take months or year before you get to state where start thinking about sync vs async .

I am sure if you have done any concurrent programming then you know how hard it is to coordinate async tasks.

Distributed Computing
 Now distributed computing has become so common that now it is hard to think of system that is not making distributed calls.

As a backend developer you are guarded or gets late exposure to distributed computing but on frontend every call to get data is remote call, so you have to aware about failures that can happen when remote call is made and slowness it adds.

Proper error handling becomes optional in backend system but on frotnend it is not the option because user will noticed it and complain about it.

Frontend is the last gate so it has to handle all the errors that are thrown or  suppressed by backend systems, so end user experience is smooth.

You experienced distributed computing very early once you are on frontend side.

Network 
We read many text book that "network is not reliable" and as a backend engineer you don't get directly exposed to this because library and framework handles it for you but on frontend you get first hand experience to deal with and come up with strategy

All the backend application gets benefit of fast network ("100 GBs network") because it runs in data center but for frontend application is data center is end user device which will be browser/handheld device.
Many time network is dial up( i.e KBs) and application has to work on slow network.

Compute and In-memory 
This one is interesting because when backend program is slow first option is increase compute or memory because elastic infra allows to do that that but on end user side no elasticity, so first option is no option for front-end friends.

Approach taken to optimize frontend is very creative and innovative as compared to backend. This also put design pressure on front application.

Algorithm 
Backend systems has more options on algorithm that can be used to solve program for example Disk based algorithm are very common for many data intensive backend application but on frontend side this option is not available or in very limited way and you have to very creative in how do you use it.

I think many chapters of algo book is Not Applicable for frontend.

Patterns
On frontend side industry is inventing new patterns every day but backend side does not move at that pace for eg functional composition , incremental rendering , state management using immutable DS , event driven systems, chunking of requests , late arrival of information due to slow network etc


Artifact/Packaging
Backend system are never seen from lens on how big jar/exe/dll is but this is first challenge to be solved on front-end side because package must be small so that it can be downloaded quickly by clients.
Network play role in this remember 100GBs vs Kbs ?

Requirements
This can be little controversial but in many case frontend are built with fuzzy or no requirement and later requirement is added. Requirement is must for backend! 

Testing
This is the hardest part for frontend. I will leave this for now because it needs multi series blog just for this topic.

Conclusion

I know it might look like i am just trying to sell frontend development is more complex but my point is you become better engineer when you move to frontend.
If you are not doing any frontend then find way to do that or learn about these hard problem from frontend engineers and incase they say "i don't think about these challenges" then educate & help them.

If you like the post then you can follow me on twitter .

JVM with no garbage collection

JVM community keeps on adding new GC and recently new one was added and it is called Epsilon and is very special one. Epsilon only allocates memory but will not reclaim any memory.



It might look like what is use of GC that does not perform any garbage collection. This type of Garbage Collector has special use and we will look into some.

Where this shinny GC can be used ?
Performance Testing

If you are developing solution that has tight latency requirement and limited memory budget then this GC can be used to test limit of program.

Memory Pressure Testing
Want to know extract transient memory requirement by your application. I find this useful if you are building some pure In-Memory solution.

Bench marking Algorithm.
Many time we want to test the real performance of new cool algorithm based on our understanding of BIG (O) notion but garbage collector adds noise during testing.

Low Garbage
Many times we do some optimization in algorithm to reduce garbage produced and GC like epsilon helps in scientific verification of optimization.

How to enable epsilon GC

JVM engineers have taken special care that this GC should not enabled by default in production , so to use this GC we have to use below JVM options

-XX:+UnlockExperimentalVMOptions -XX:+UseEpsilonGC -Xlog:gc


One question that might be coming in your mind what happens when memory is exhausted ? JVM will stop with OutofMemory Error.

Lets look at some code to test GC

How to know if epsilon is used in JVM process?

Java has good management API that allows to query current GC being used, this can also be used to verify what is the default GC in different version of java.

Run above code with below options
-XX:+UnlockExperimentalVMOptions -XX:+UseEpsilonGC VerifyCurrentGC

How does code behave when memory is exhausted. 

I will use below code to show how new GC works.

Running above code with default GC and requesting 5GB allocation causes no issue (java -Xlog:gc -Dmb=5024 MemoryAllocator) and it produces below output

[0.016s][info][gc] Using G1
[0.041s][info][gc] Periodic GC disabled
Start allocation of 5024 MBs
[0.197s][info][gc] GC(0) Pause Young (Concurrent Start) (G1 Humongous Allocation) 116M->0M(254M) 3.286ms
[0.197s][info][gc] GC(1) Concurrent Cycle
[0.203s][info][gc] GC(1) Pause Remark 20M->20M(70M) 4.387ms
[0.203s][info][gc] GC(1) Pause Cleanup 22M->22M(70M) 0.043ms
[1.600s][info][gc] GC(397) Concurrent Cycle 6.612ms
[1.601s][info][gc] GC(398) Pause Young (Concurrent Start) (G1 Humongous Allocation) 52M->0M(117M) 1.073ms
[1.601s][info][gc] GC(399) Concurrent Cycle
I was Alive after allocation
[1.606s][info][gc] GC(399) Pause Remark 35M->35M(117M) 0.382ms

[1.607s][info][gc] GC(399) Pause Cleanup 35M->35M(117M) 0.093ms
[1.607s][info][gc] GC(399) Concurrent Cycle 6.062ms

Lets add some memory limit ( java -XX:+UnlockExperimentalVMOptions -XX:+UseEpsilonGC -Xlog:gc -Xmx1g -Dmb=5024 MemoryAllocator)

[0.011s][info][gc] Resizeable heap; starting at 253M, max: 1024M, step: 128M
[0.011s][info][gc] Using TLAB allocation; max: 4096K
[0.011s][info][gc] Elastic TLABs enabled; elasticity: 1.10x
[0.011s][info][gc] Elastic TLABs decay enabled; decay time: 1000ms
[0.011s][info][gc] Using Epsilon
Start allocation of 5024 MBs
[0.147s][info][gc] Heap: 1024M reserved, 253M (24.77%) committed, 52640K (5.02%) used
[0.171s][info][gc] Heap: 1024M reserved, 253M (24.77%) committed, 103M (10.10%) used
[0.579s][info][gc] Heap: 1024M reserved, 1021M (99.77%) committed, 935M (91.35%) used
[0.605s][info][gc] Heap: 1024M reserved, 1021M (99.77%) committed, 987M (96.43%) used

Terminating due to java.lang.OutOfMemoryError: Java heap space

This particular run caused OOM error and is good confirmation that after 1GB this program will crashed.

Same behavior is true multi thread program also, refer to MultiThreadMemoryAllocator.java for sample.

Unit Tests are available to test features of this special GC.

I think Epsilon will find more use case and adoption in future and this is definitely a good step to increase reach of JVM.

All the code samples are available Github repo

If you like the post then you can follow me on twitter .

Need driven software development using Mocks

Excellent paper on mocking framework by jmock author. This paper was written in 2004 that is 18 years ago but has many tips of building maintainable software system.



In this post i will highlight key ideas from this paper but suggest you to read the paper to get big ideas behind mocking and programming practice.

 Mock objects are extension of test driven development.

Mock objects can be useful when we start thinking about writing test first as this allows to mock parts that is still not developed. Think like better way of building prototype system.

Mock object are less interesting as a technique for isolating tests from third-party libraries.

This is common misconception about mock and i have seen/written many codes using mock like this. This was really eye opening fact that comes from author of mocking framework.

Writing test is design activity

This is so much true but as engineer we take shortcut many time to throw away best part of writing test. Design that is driven from test also gives insights about real problem and it lead to invention because developer has to think hard about problem  and avoid over engineering

Coupling and cohesion 

As we start wiring test it gives good idea on coupling & cohesion decision we make. Good software will have low coupling and high cohesion. This also lead to functional decomposition of task.
Another benefit of well design system is that it does not have Law_of_Demeter, this is one of the common problem that gets introduced in system unknowingly. Lots of micro services suffer from this anti pattern.

Need driven development
As mocking requires explicit code/setup, so it comes from need/demand of test case. You don't code based on forecast that some feature will required after 6 months, so this allows to focus on need of customer. All the interfaces that is produce as result of test is narrow and fit for purpose. This type of development is also called top down development.

Quote from paper
"""
We find that Need-Driven Development helps us stay focused on the requirements in hand and to develop coherent objects.
"""


Programming by composition

Test first approach allows you to think about Composability of components, every thing is passed as constructor arguments or as method parameter.
Once system is build using such design principal it is very easy to test/replace part of system.
Mock objects allows to think about Composability so that some parts of system are mocked.

Mock test becomes too complicated
One observation in paper talks about complexity of Mock Test.
If system design is weak then mocking will be hard and complicated. It does amplification of problems like coupling, separation of concern.  I think this is best use of mock objects to get feedback on design and use it like motivator to make system better.

Don't add behavior to mock
As per paper we should never add behavior to stub and in case if you get the temptation to do that then it is sign of misplaced responsibility.

If you like the post then you can follow me on twitter to be notified about random stuff that i write.