Machine for Killing Your Best Ideas

You built a model of your executive’s judgment so you could pre-clear every idea before it reaches them.

Take an executive whose yes or no gates everything, watch enough of their decisions, and fit a model to the pattern. Now you have an oracle. Run every idea past the clone, advance the ones it approves, and never burn a real meeting on a doomed proposal again. Faster loops, less friction, a happier executive.

It works and that create new problem if you think as System Thinker.

How IT WORKS

Strip the magic away and the clone is a filter: a function from an idea to a predicted approval. To see why that’s dangerous, ask what approval was ever for. Nobody wants approvals. They want outcomes. 

Approval is a stand-in — a lossy compression of “this will probably work, and it fits where we’re going” down to a single bit. The executive is already a proxy for the goal, real test has multiple dimensions and latent factors that you never know.

The clone is a proxy for the executive. You are now optimizing two hops from anything real, and every hop throws away signal.

Upside is real ( Illusion)

Which is exactly what makes this hard to argue against. Cycle time collapses — you get an instant read and iterate against it instead of waiting days for a verdict. 

Executive’s attention stops being spent adjudicating obvious non-starters. And building the clone forces something genuinely useful: to fit the model, someone has to name what actually drives a yes. The criteria come out of the executive’s head and onto the page, and the model is often clearer about the real rules than the person is.

That legibility is worth having on its own. If the story ended here, this would simply be a good tool. It doesn’t end here.

A clone can only recognize the future by its resemblance to the past.

How clone fails

Once clone is the gate that every idea must clear, and few things happen. None of them is the thing you or company wanted.

GoodHART Law

As per GoodHart law ,the moment “looks approvable” becomes the target, it stops measuring quality. You are now running gradient descent on the proxy — producing proposals optimized to please a model, which is a different thing from proposals that work. We see this in every industry.

Variance Collapse

Clone recognizes the future only by its resemblance to the past. Genuinely novel bets — the ones that don’t pattern-match a prior yes — die before a human sees them. You strip out the asymmetric upside and keep the safe, legible middle. In a fast domain, the middle is where returns go to die.

Atrophy ( RubberStamp)

If the clone is good, the real approvals become rubber stamps, because you only ever surface what it already predicts a yes on. The executive stops seeing raw idea flow — the exact input that kept their judgment calibrated. The map replaces the territory, and then no one is watching the territory.

Frozen Judgement

Last year’s executive approved last year’s ideas. The clone encodes that snapshot and enforces it going forward. You become faster at executing a judgment at the precise moment it goes stale — efficiency pointed the wrong way. Real world connection breaks.

Gaming

People learn what the clone likes and write to it. Proposals converge on a template. The organization gets fluent in “approvable” and forgets how to say “true.” Generative AI is amazing at cloning. I wrote post on how easy it is clone software now.

Lets get real

Everything above assumed the clone sits in front of a human — a filter you consult on the way to a real decision.

Now change one detail. The model isn’t hypothetical, and it isn’t waiting to be built. It’s installed. Decision-makers are already using it to say yes and no — on live proposals, today. The clone isn’t a draft of the gate. It is the gate.

Executive was a proxy for the goal so you cloned the executive. Now the executive consults the clone before deciding — so the human sits downstream of the model that was supposed to sit downstream of the human. 

We are moving towards world where clone has veto.

How AI Coding Assistants Reshape Productivity

 

The Jevons Paradox in Software Engineering: How AI Coding Assistants Reshape Productivity

Imagine this: You've just installed the latest AI coding assistant. The marketing promised to cut your coding time in half. Six months later, you're writing more code than ever before, tackling increasingly complex problems, and somehow still working the same hours. What happened?

Welcome to Jevons Paradox in the age of AI-assisted software development.

The Curious Case of Efficiency That Doesn't Save Time

In 1865, a British economist named William Stanley Jevons noticed something counterintuitive about coal consumption. When more efficient steam engines were introduced, logic suggested coal use would decrease. Instead, it skyrocketed. The more efficiently coal could be used, the more applications people found for it.

Fast forward to 2025: Your AI coding assistant is the modern-day steam engine, and your time and mental energy are the coal.

My Journey With AI Coding Assistants: A Personal Story

When I first integrated an AI coding assistant into my workflow last year, I had visions of shorter workdays and more time for strategic thinking. The reality? I found myself saying "now I can finally tackle that refactoring project I've been putting off" and "let's add those extra test cases we've been skipping."

Sound familiar?

The Numbers Don't Lie: The Productivity Paradox in Action

Recent industry surveys reveal a fascinating pattern:

Metric
Without AI Assistant
With AI Assistant
Change
Lines of code written/week
1,200
1,560
+30%
Tickets closed/sprint
8
10
+25%
Languages/frameworks used regularly
2-3
4-5
+67%


Below Metrics are subjective.
I loved writing more code








Average hours worked/week
40 to 50
70+
50%+

Why We Keep Consuming Our Efficiency Gains

The Expanding Possibility Frontier

As our tools improve, our concept of what's possible expands with them. It's human nature. When we suddenly have the capacity to do more, we don't pocket the difference—we expand our ambitions.

This cycle isn't unique to software development, but our field experiences it more intensely than most because of how quickly our tools evolve.

The Four Types of Productivity Consumers

In my observation, there are four ways engineers typically "spend" their AI-driven productivity gains:

1. The Depth Diver – Uses efficiency to create more robust solutions with better error handling, edge case management, and performance optimization
2. The Breadth Explorer – Leverages AI to work across more languages, frameworks, and systems than previously possible
3. The Quality Enhancer – Invests saved time in better documentation, more comprehensive tests, and cleaner code
4. The Volume Maximizer – Simply produces more features, closes more tickets, and ships more code

Which one are you? Most of us are a blend, shifting between these archetypes depending on project requirements and personal interests.

The Great Capability Expansion

What makes AI coding assistants particularly powerful is how they expand what individual developers can accomplish:

This expansion means junior developers can contribute to complex systems earlier in their careers, while senior developers can focus more on architecture and innovation.

Reimagining Productivity in the AI Era

From "Doing Things Faster" to "Doing Better Things"

The most successful teams I've observed aren't just using AI to speed up existing processes—they're rethinking what processes should exist in the first place.

Consider this reimagined development workflow:

The key shift: humans focus on the parts of the process where creativity, judgment, and contextual understanding matter most.

How to Thrive in the Age of AI-Assisted Development

1. Embrace Strategic Inefficiency

Not everything should be optimized for speed. Sometimes, diving deep into a problem without AI assistance builds fundamental understanding that pays dividends later.

2. Set Clear Boundaries

Establish team norms around when and how to use AI assistants. Some projects benefit from exploration and creative generation; others need careful, methodical human reasoning.

3. Measure What Matters

If you're still measuring productivity by lines of code or tickets closed, you're missing the true impact of AI assistance. Consider metrics like:

• Time to validated solution (not just working code)
• Reduction in production incidents
• User-reported satisfaction with features
• Knowledge dissemination across the team

4. Continuously Reskill

The skills that make developers valuable are evolving rapidly. The future belongs to those who can:

• Clearly articulate problems for AI to solve
• Evaluate and refine AI-generated solutions
• Understand and communicate system-level concerns
• Apply deep domain knowledge to technical decisions

Looking Ahead: The Co-Evolution of Engineers and Their Tools

As our relationship with AI coding assistants deepens, we're not just changing our tools—our tools are changing us. The software engineers of 2030 will approach problems differently than those of 2020, just as today's engineers think differently than those of the pre-internet era.

The most exciting part of this journey isn't just what we'll build—it's who we'll become as builders.

---

What's your experience with AI coding assistants? Are you saving time, doing more, or both? Share your thoughts in the comments below!

Measuring Developer Productivity in Age on GENAI

The GenAI Revolution: Two Years Later

November 30, 2022 marked a pivotal moment when ChatGPT was released, sparking excitement and optimism about increased efficiency across industries. Now, with over two years of GenAI integration, the industry has matured enough to properly evaluate the impact and value of these tools on various aspects of business. In this post, I'll focus specifically on measuring developer productivity.

Measuring Impact: Output vs. Outcome

The impact of any change—whether new tools, processes, or methodologies—can be measured in terms of both output and outcome.

As a product organization, outcomes are ultimately the metrics that deliver revenue or customer growth. However, this same model cannot be directly applied when measuring the impact of GenAI on developer efficiency.

A Framework for Measurement

In this post, I'll share several approaches to measure productivity with GenAI tools, focusing on a progression from:

Output → Outcome → Growth

This framework will help organizations better understand how GenAI affects developer productivity in ways that eventually translate to business value.

Developer productivity can be measured on multiple dimensions

	How Fast (Output)	Is effective  (Output)	Impact (Outcome)	Growth (Outcome)
Primary Metrics	# PR per Engineers # Test Coverage per PR	# Engineering time Index # Non Engineering time index	Failure Rate of Change Usage of Feature	Time spent on new capability/products  Time spent on R&D
Secondary Metrics	Cycle Time for PR Deployment Frequency Perceived rate of productivity	Time on PRs per sprint  Friction in  delivery Code tech Debt Index Code Security Debt Index	Last minute change  Operational & Security Health	ROI on new features  Revenue per Engineers New Products/Segments

 

Finding the Right Mix of Developer Productivity Metrics

The table above outlines four key dimensions for measuring developer productivity in the GenAI era. These dimensions incorporate both quantitative and qualitative metrics, collected through various methods:

Balanced Measurement Approach

Each dimension contains metrics that vary in nature:

• Quantitative metrics provide objective, numerical data that can be tracked over time
• Qualitative metrics capture subjective experiences and insights that numbers alone cannot reveal

Lets start with category of metrics 

How Fast ( Output)

This metric provides a straightforward measure of how effectively development teams leverage generative AI tools to produce code and the rate at which they do so. It serves as an excellent starting point for analysis and can be fully automated for continuous monitoring.

Is Effective ( Output)

This category assesses the quality of output by analyzing the ratio of time spent on engineering versus non-engineering tasks. It also incorporates lagging indicators such as sprint-level pull request review times, code technical debt indices, and security vulnerability indices. These metrics, largely automated, provide insights into both positive outcomes and potential side effects.

Impact ( Outcome)

This category marks the initial phase of measuring the impact of generative AI-assisted work. It focuses on evaluating delivery quality, product usage, and overall product health.

Growth ( Outcome)

This final category focuses on quantifying the tangible value generated by new features, specifically in terms of return on investment (ROI) and revenue. While direct revenue impact may not be immediately apparent in short development cycles, the focus shifts to measuring the time freed up for new capability development and the potential for new product or market segment expansion.

Things to watch while you measure developer productivity. 

Measuring productivity can lead to misleading signals. Organizations should be wary of:

• Spikes in Lines of Code (LOC) that don't mean better output.
• High Commit/PR counts without real progress.
• Long hours, which often signal burnout, not efficiency.
• Burning through story points too fast, which can mean poor planning.
• Focusing only on individual metrics, not team success.
• Using gamification that hurts collaboration.
• Too many unfinished POCs or WIP projects.
• Thinking Generative AI fixes everything
• A pattern of implementing new Generative AI tools at an unsustainable frequency, such as weekly or more

Conclusion

Metrics shared in this post are in between DORA and SPACE and gives holistic view of team productivity gain. 

If you are early in journey then refer to Implementing-genai-in-engineering-teams post that talks about how to implement transformation.

Implementing GenAI in Engineering Teams - System Thinking Approach

In the rapidly evolving landscape of software development, Generative AI represents not just another tool, but a fundamental shift in how engineering teams operate. However, successful implementation requires more than just access to the latest AI tools—it demands a systematic approach to change management and team adaptation.

Challenges In GenAI Adoption

Many engineering teams rush to adopt GenAI tools like GitHub Copilot or Claude etc, hoping for immediate productivity gains. Yet, without a structured approach, these implementations often fall short of expectations or, worse, create new inefficiencies. The key lies in understanding that GenAI adoption is a systems challenge, not just a technical one.

A Systematic Framework for Implementation

Drawing from Donella Meadows' "Leverage Points" model, here's a practical framework for implementing GenAI in engineering teams, organised from foundational elements to transformative changes.

Start with the Foundations (Parameters & Buffers)

Before diving into complex transformations, establish your baseline:

- Set clear metrics for current development speed and quality

- Allocate 20% of team time for AI tool learning

- Maintain manual coding capabilities for A/B test

- Track costs and benefits per developer

Build the Structure

Structure your implementation around:

- A pilot team with clear objectives

- One primary AI tool (e.g., GitHub Copilot, Cursor , Aider , Windsurf etc)

- Specific use cases (test generation, documentation, new code , refactoring , code review etc)

- Regular feedback mechanisms

Optimize Information Flow

Success depends on effective knowledge sharing:

- Create an internal prompt library

- Document successful patterns and anti-patterns

- Establish clear guidelines on AI capabilities and limitations

- Regular updates on new AI features and best practices

Establish Clear Rules and Processes

Protect quality and security with:

- Mandatory review processes for AI-generated code

- Security scanning protocols for work produced by AI

- Data privacy guidelines

- Clear escalation paths for AI-related issues

Foster the Right Mindset

The most crucial transformation happens in how teams think about their work:

- Position AI as an augmentation tool, not a replacement

- Focus on high-value problem solving

- Encourage experimentation and learning

- Build confidence through small wins

Measuring Success

Track progress through:

Speed Metrics

   - Code completion time

   - Time saved on repetitive tasks

   - Documentation generation speed

Quality Indicators

   - Code review feedback

   - Bug detection rates

   - Technical debt metrics

Team Adaptation

   - AI tool usage rates

   - Prompt effectiveness

   - Knowledge sharing participation

Common Pitfalls to Avoid

Tool Overload: Starting with too many AI tools simultaneously

Unrealistic Expectations: Expecting perfect code from AI

Neglecting Training: Not investing in team AI literacy

Ignoring Process: Bypassing code review for AI-generated code

Poor Documentation: Not capturing lessons learned

The Path Forward

Successful GenAI implementation is a journey, not a destination. Start small, focus on concrete wins, and build momentum through systematic change. Remember that the goal isn't to replace human developers but to augment their capabilities and free them to focus on more complex, creative problem-solving.

Key Takeaways

1. Start with clear metrics and baseline measurements

2. Focus on one team and one tool initially

3. Build strong feedback loops and learning mechanisms

4. Maintain high quality standards

5. Foster a culture of experimentation and learning

The future of software development lies in the effective collaboration between human creativity and AI capabilities. Teams that can systematically implement these tools while maintaining their engineering excellence will find themselves at a significant advantage in the evolving technological landscape.

Remember, the goal isn't to completely transform overnight, but to build a sustainable, efficient system that leverages AI to enhance human capabilities rather than replace them.

Top large language model to watch

The LLM landscape is exploding! With the immense potential of large language models, competition is fierce as companies race to develop the most powerful and innovative models. Training these models presents a lucrative business opportunity, attracting major players and startups alike.

Keeping track of the leaders is challenging. The LLM space is highly competitive, making it difficult to identify a single frontrunner. New versions are released constantly, pushing the boundaries of what's possible. While some might see this as a race to the bottom, it's more accurate to view it as rapid innovation that will ultimately benefit everyone.

Top company as of July,2024

Above diagram is in 2 groups , one for commercial ones and other one for hybrid(commercial/open weights) 

Commercial

OpenAI

This is poster child of LLMs, it has series of GPT* models. First large scale provider consumer LLMs.

GPT4-O is flagship model and all the models are available via API. This is very well funded and microsoft is behind this.

More details about model can be found at Open AI Model 

Research paper talking about GPT4 Model is available at 

GPT-4 Technical Report 

 GPT 1.0

GPT 2.0

Language Models are Few-Shot Learners

Evaluating Large Language Models Trained on Code

Amazon

Amazon has family of models called "Titan". Amazon Titan family of models incorporates Amazon’s 25 years of experience innovating with AI and machine learning across its business. Amazon Titan foundation models (FMs) provide customers with a breadth of high-performing image, multimodal, and text model choices, via a fully managed API.

More details about model can be found at Amazon Models

No research papers are available about amazon LLM model details. It is all propriety to keep competitive edge.

Antropic

Antropic is cofounded by some of ex Open AI employee. 

Anthropic's latest offering, Claude 3.5 Sonnet, has generated significant buzz. This powerful language model builds upon their previous success with Claude 3 Opus and is claimed to outperform OpenAI's GPT-4o, particularly in coding tasks.

Antropic is also very well funded, Amazon and google are major investor.

More details about model can be found at Antropic Models

Antropic models will be based on Open-AI type of architecture but they are focused on few research principal like AI as Systematic Science , safety and scaling 

One of the popular research paper from antropic is mapping-mind-language-model

MoasicML

MosaicML, co-founded by an MIT alumnus and a professor, made deep-learning models faster and more efficient. It was acquired by Databricks. 

Mosaic Pretrained Transformers (MPT) are GPT-style models with some special features -- Flash Attention for efficiency, ALiBi for context length extrapolation, and stability improvements to mitigate loss spikes.

More details about model can be found at mosaic ml

Some popular research papers are Train Short, Test Long and Flash attention

InflectionAI

Inflection AI focuses on developing a large language model (LLM) for personal use called Inflection.

Not much details is available about how model was trained but they claim - world's top empathetic Large Language Model (LLM)

More details about model can be found at inflection-2-5

Hybrid/Open Source

Google

Google inventor of famous paper Attention Is All You Need that became kernel of all the LLMs we see today. 

Google has been releasing LLM to community before Chatgpt came, Bert was one of the first model based on encoder/decoder and become foundation for many LLM that we see.

Google offers large language models (LLMs) across a spectrum of availability. Some models are fully commercial with open weights, meaning the underlying code is proprietary but the model outputs are accessible.

The Gemini family exemplifies this, with variants like Ultra, Pro (introduced in v1.5), Flash, and Nano catering to different needs in terms of size and processing power.

In contrast, Gemma is Google's open-source LLM family. It's designed for developers and researchers and comes in various sizes (e.g., Gemma 2B and 7B) for flexibility

Lots of reading material is available from google on LLM and Gemma models, some of the popular ones are 

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

PaLM: Scaling Language Modeling with Pathways

Scaling Instruction-Finetuned Language Models

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Gemma: Open Models Based on Gemini Research and Technology

gemma-2-report

Meta

Meta builds LLama series of model, these are open source and Meta designed Llama to be efficient, achieving good performance while being trained on publicly available datasets.

Llama3 is most recent and state of art. These models are trained by meta and made available via various hosting platform. Llama3 is is extended by other vendors like Gradient , Nvidia , dolphin etc.

Details about model is available at llama3. 

Meta has publish lots of paper from first version of model, some of the popular ones are 

LLaMA: Open and Efficient Foundation Language Models

Llama 2: Open Foundation and Fine-Tuned Chat Models

Code Llama: Open Foundation Models for Code

meta-llama-3

Mistral

Mistral is french based company and they release all model weights under Apache 2.0.

Mistral strives to create efficient models that require less computational power compared to some competitors. This makes them more accessible to a wider range of users.

Mistral innovation is around Grouped Query Attention (GQA). Some of the recent models are based on Mixture Of Expert.

More details about model is available at Mistral models

Some of popular research papers are Mistral 7B , A Closer Look into Mixture-of-Experts in Large Language Models

DataBricks

Databricks is building open source model that are based on MOE. Most recent and state of the art model is DBRX.

Details about model is available at introducing-dbrx-new-state-art-open-llm

Some of popular research papers are 

MEGABLOCKS: EFFICIENT SPARSE TRAINING WITH MIXTURE-OF-EXPERTS

PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Cohere

Cohere is canadian based company. They build model called CommandR, it is a state-of-the-art RAG-optimized model designed to tackle enterprise-grade workloads.

More details about model can be found at Command-R

Some of popular research papers are RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs

 

Microsoft

While Microsoft leverages OpenAI's powerful GPT-4 language models for some functionalities, they've also made significant contributions to open-source AI with the Phi-3 family of models.

Phi-3 models are a type of small language model (SLM), specifically designed for efficiency and performance on mobile devices and other resource-constrained environments.

 More details about model can be found at phi-3

Some of popular research papers related to Phi series model are Textbooks Are All You Need , Textbooks Are All You Need II and Phi-3 Technical Report

Conclusion

We are witnessing an interesting time where many large language model (LLM) models are available for building apps, accessible to both consumers and developers. Predicting the dominant player is difficult due to the rapidly changing landscape.

One key concept to grasp is that the GENAI stack is multifaceted. Foundation models are just one layer, and they can be quite expensive due to hardware requirements. Training a foundation model can easily cost millions of dollars, making it difficult for companies to maintain a competitive edge.

As software engineers, we need to leverage this technology by selecting the best model for each specific use case. Defining "best" can be subjective, and the answer often depends on various factors.

Here's a crucial consideration: while using the top-performing LLM might be tempting, it's vital to maintain a flexible architecture. This allows you to easily switch to newer LLMs, similar to how we switch between databases or other vendor-specific technologies.

In the next part of this blog, I'll explore the inference side of LLMs, a fascinating area that will ultimately determine the return on investment (ROI) for companies making significant investments in this technology.