How do you know if a new tool is worth it?
As professional software developers we continuously want to improve our ability to deliver high quality software in the time and resource budget we have available.
However there have always been a problems in knowing for sure which tools and processes help us towards that goal. Should we spend time writing tests rather than features? How do we know? What is the optimal number of tests? Is pair programming worth it? Is a new tool worth the financial cost and the cost of changing our workflow? And again, how do we know?
These have been perennial issues and today we're faced with a different version of the problem. Should we invest $X / month in a generative AI coding tool (co-pilot) and change our development workflow and if so what kind of return can we expect and when and by which developers? And more generally are the drawbacks of generative AI worth the potential gains?
In a previous job my manager recommended the book "How to Measure Anything: Finding the Value of Intangibles in Business" by Douglas W. Hubbard. As I remember it as an interesting book and worth a look. I also think that "Thinking in Bets: Making Smarter Decisions When You Don't Have All the Facts" by Annie Duke is relevant.
What it comes down to is you'll never have complete and accurate information so it is more about making good decisions rather than achieving ideal outcomes. And making good decisions is more about being clear on your goals, using all available information, making a smart decision and measuring results; all while knowing full-well that a great decision may have negative results. If I remember Annie Duke's examples: driving home drunk is a bad decision whether or not you make it home while taking a chance on someone (position) you are interested it can be a good decision whether or not the relationship (job) works out.
But to do this you need to:
- decide what matters
- find a way to measure it
- be aware of confounding factors and biases and
- have the courage to face a negative result
More on all that in future posts. In the mean time I plan on revisiting those books. I'd love to hear if you've read them and what you think about them.