Complexity Accidental vs Essential

Today it is hard to find team or organization that is not following agile but building software has not become easy, projects are missing schedule , over budget and it is also flawed.


Why it is so hard to build software ?

If you ask this questions to any engineer then 90%+ will say requirement, but is that the full truth ?

Lets try to decompose software construction. Every feature has 2 important component that decides whether feature will be successful or not.

• Essential complication (ec)

• Accidental complication( ac)

We will do some Functional programming refresher.

Feature = f( Essential Complication) + f(Accidental Complication )

Essential complication comes from domain like if you are building software for medical industry then it is complex. Accidental complication is complexity added by engineers, process & management to build feature.

Essential complication are hard to reduce because of domain, but to some extent it is possible to reduce by using good decomposition techniques. Decomposition is hard skill and comes from experiment of some fail projects.

Accidental complication can be controlled but it is not linear function, accidental complication is not same in every part of system and gets more complex over time. This also gives feedback on how much bad job we have done as engineer or product team.
Complexity comes in various forms like communication in team, less understanding , difficulty reusing some feature , extending program to new function, management problems etc.

Now we know accidental complication is exponential, so lets write formula again.

Feature = f( Essential complication) + f(Accidental complication * Unknown)) 

Now it will become clear why something takes many times longer than estimate or guess. Product owner also has part to play in adding accidental complication by marking assumption on importance of feature.

What can be done ?
If we need some predictability or consistence in delivery then we have to continuously work on reducing accidental complexity. Lets look at ways to keep to keep this in control.

• Using higher level languages.
• Incremental development by growing the software not building it.
• Good buy vs build decision. 
• Unified programming environment. 
• Raid prototype to refine requirement.
• Listen to design pressure.
• Test driven development.
• Stop "Get it out of the door" mindset.
• Reduce "surgical strike effort" in delivery.

Very insightful quote from Frederic Brooks, both customer and engineer has to learn what to ask, expect, and commit otherwise only option is broken system.

“An omelette, promised in two minutes, may appear to be progressing nicely. But when it has not set in two minutes, the customer has two choices—wait or eat it raw. Software customers have had the same choices. The cook has another choice; he can turn up the heat. The result is often an omelette nothing can save—burned in one part, raw in another.”
― Frederick P. Brooks Jr., The Mythical Man-Month: Essays on Software Engineering

Mythical-Man-Month by Mr Brooks is must read for every product owner , project manager and engineer.

If you like the post then you can follow me on twitter.

Don’t comment bad code—rewrite it

In this post i will share my experience of "comments in code" that i have got by reading code , writing code and reading books.

Lets start with famous quote

“Don’t comment bad code—rewrite it.”
  —Brian W. Kernighan and P. J. Plaugher

Lots of comment in code looks just like above image and it is distraction.

Comments are lie 
Most of the time comments are not in sync with code, no one cares about it so no one maintains it.

Code is refactored
Code is under constant improvement, it changes , evolves and dies but code comments just sticks and become orphan.

Bad code has more comments
Once we write code that is hard to understand then we try to explain it by comments and it tells lot about our failure to write expressive code.

Noisy comment
Lets look at at some code+comment from public domain.

https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java

/**
	* Create a new HdfsAdmin client.
	*
	* @param uri the unique URI of the HDFS file system to administer
	* @param conf configuration
	* @throws IOException in the event the file system could not be created
	*/
	public HdfsAdmin(URI uri, Configuration conf) throws IOException {
	FileSystem fs = FileSystem.get(uri, conf);
	if (!(fs instanceof DistributedFileSystem)) {
	throw new IllegalArgumentException("'" + uri + "' is not an HDFS URI.");
	} else {
	dfs = (DistributedFileSystem)fs;
	}
	}

This code snippet is from popular open source java project, many comment in enterprise is also like this. This adds more noise than value.

Journal comments
Many scripting code, data base code, python code is based on this pattern, this looks like team has no trust on source control and they have taken over this responsibility. It looks something like below

/*
01-Apr-2019 - First version with skeleton code by someone
10-Apr-2019 - Config support added  by someone
....
...
*/ 

Marker comments
This starts with something like below

// Starting init

//Make DB call 

this is also very common in scripting, DB stored procedure or in real enterprise type programs. This patterns tell that it is shouting to extract small function but gets overlooked.

Commented code
I am sure you have seen this many time and got frustrated that why this dead code exists.
It looks something like this.

//Keeping this for backup
//fastCode()


superFastCode() 

Author code
IDE adds block of comments to every file that is added to project, it looks like below

/*

@Ashkrit
*/

Author names is just at the top of the file but he no more maintains it and does not work for team or organization but occasionally gets credit/blame for his creation.



Conclusion
Code Comment mostly adds no value, it comes between you and code, it is one extra distraction.It also promotes bad habit for young programmer because they think "adding comments makes code better".

In some case comments are useful when you are writing public APIs but still be careful on what you write, software engineers are smart they can understand if code does not have accidental complexity.

I will leave you with another famous tweet by Kevlin Henney

Uncle bob has whole chapter in Clean Code: A Handbook of Agile Software Craftsmanship about comments. I recommend to read chapter, it covers things in more details.

If you like the post then you can follow me on twitter.

Design pressure on engineering team ?

How many time you are supporting or developing system and felt it could have been better designed.

Move fast and break things culture in software has allowed engineering team to get product early to market but has created huge Tech Debt that team has struggled to come over.

Engineering team is put under feature pressure to get the things done and other important things are left as nice to have.

Important things in system are put as "Non Functional" requirement but in reality these are the requirement that are must for system to be functional in long run.

In real engineering project like civil engineering people are trained to think about 

• What material to use ?
• How many people it can accommodate ?
• When maintenance must happen ?
• Where are the emergency exit ? 
• How to evacuate in case of emergency ? 

But many Software system are build without thinking about basic Operations or Support requirement.

Software development has become trend or resume driven to use machine learning, Deep learning, latest java script , latest NO-SQL or Stream processing,Blockchain etc.

These cutting edge fails to delivery to basic engineering needs if no upfront thought is put in.

As a software engineer we have to think about Design Pressure as first thing. What are these design pressure ?

• How service executes ?

• What load it can handle ?

• It is possible to run it on developer laptop ?

• Is it possible to debug it ?

• How Coupling is used or Cohesion is used ?

• Does it support automated packaging or deployment ?

• Is it testable or maintainable ?

• Single responsibility or multiple responsibility ?  

• Separation of concerns ?

• Can you observe system from outside ?

• Does system has pipeline or chokeline architecture 

• What are system dependency or it needs whole world

• Does it capture metrics or you have to make a guess.

• Does it fit in your HEAD or you need BIG head or multiple HEAD

These are things that are very difficult and expensive to add later.

The reason why we don't look at these things first is because "We get bored" or "We like complex" or "We want to copy what google or facebook does"

Not every company has problem that google or facebook is facing so we should be careful in picking tools/tech that is used by big internet company.

I want to end with with "tale of two value" of software system.

Behavior 

Software developer are hired to build new features and stakeholders work closely to get this thing implemented. This is the "what" part of Software.

Architecture

If this part is done properly then Software remains "Soft" or it becomes Big ball of mud.

This is "How" part and engineer has to take all the responsibility of this. 

Both the value of system are important and urgent, but more focus is put on behavior because it becomes Urgent as soon as project starts.

If we ignore "Design Pressure" then system will become costly to develop & making changes will be impossible.

Let me know about your experience with Design pressure 

Unix design patterns and philosophy

Design patterns got very famous after release of Design-Patterns-Elements-Reusable-Object-Oriented book and it became one of the advance topic of software engineering.

It become one of the must book to have it on desk, it also gave common vocabulary to talk about design. Design pattern idea was extended to capture solution to recurring problem. 

Linda Rising took same idea and wrote book on Fearless Change Patterns Introducing Ideas , she wrote part2 of book More Fearless Change and i am sure lot more books will come in future which will present patterns idea in different shapes & forms.

In this blog i will share design patterns used in UNIX.

All the patterns are 30+ year old but are so much relevant in microservices time.

Lets start with famous quote from  "C. A. R. Hoare"

"There are two ways of constructing a software design. One is to make it so simple that there are obviously no deficiencies; the other is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult."

The Emperor’s Old Clothes, CACM February 1981
—C. A. R. Hoare

Bottom up design

First design philosophy is unix is build using bottom up approach, it is build from lego blocks, built small pieces and then compose them to create something more useful. Functional programming is based on this principal !

Make each program do one thing well 

    This is such a simple rule but not used enough, developer loves complexity and this rule is violated everyday and results in software that does too many things but many features are not implemented properly.

Some of the example from UNIX of one thing well are grep, diff , patch , Yacc.
These programs are bugfree and so much that functionality is taken for graunted.

Design programs to be connected with other programs(Composition)

This is the rule that invented PIPE (i.e | ) , if our program are build using this rule then lot of time/money spent on integration can be saved.

In Unix world binary input/output is avoided and this makes integration so easy.
Unix program separate interface from logic, interface is "Text" stream and produces "Text" stream.

Each program is independent and they can be composed together because they follow simple rule of "text" interface.
  
As a developer we like to do premature optimization and start with binary input/output, use binary format only when bottleneck is proved.

 

Build software that can be tried early may be in in few days or 1 week. 

This philosophy is also applicable for complex software like Operating System/ Device Driver etc, today we have to take help of Agile to build software that can be tried in 2 or 3 weeks.

Unix guys were doing Agile in 1970.

Avoid Fancy algorithm

Fany algorithm are complex to implement and have bugs and gives dividend only when N is above some threshold, by using simple algo and data structure lot of complex problem can be solved.

Data dominates 

Spend more time on building data structure that will organize data properly.

Write simple parts connected by clean interface 

This is the most difficult on to get right, we have seen different programming paradigm starting form assembler , structural , object oriented , functional etc but all of them have failed, as soon as program turns from POC to real it can't fit in head of human brain.

Clarity over Cleverness 

I am sure every developer will have some war story to tell about debugging Clever program.

Rule of Transparency

This is one the things that is thought  on last day of development in-spite of knowing that this rule is to make inspection and debugging of program easier.

Rule of Repair

When you fail, fail noisily and as soon as possible.

Single Point of Truth(SPOT) rule

This rules says that every piece of knowledge must have single, unambiguous representation in system. Repetition leads to inconsistency so use Constants, tables, metadata, code generator . Complicity is cost , don't pay it twice.  

All the philosophy boils down to