The Art of Picking the Right Problems

The core function of a Software Development Manager (SDM) is to choose the right projects and problems for their teams to tackle. Sometimes we call this skill “product sense”. SDM can decide the problems to solve with a portion of their team’s time, on top of fulfilling requirements from product management. This is especially so for teams that have high operational excellence requirements. The freedom turns every SDM into a product manager. However, not all problems are created equal. I have seen many teams devoting excessive time to resolving issues, either from customers or from daily operations. This "one damned thing after another" approach, lacks strategic vision and forethought. The work becomes reactive instead of proactive, with the team sprinting from one firefighting situation to another, leaving little to no time to invest in initiatives that have true long-term value.

I had a recent conversation with an SDM about a new project they will launch. The team was contemplating fixing their "log loss" problem - a situation where, due to the inability to stream logs quickly enough, some stale logs were disregarded by their metric system. Superficially, it seemed like a worthwhile problem. After all, better observability is a good thing, right?

“But who will you delight, and on what use cases if you fix this?” I asked.

The SDM was caught off guard. “The on calls I suppose?”

“But on calls were doing fine without these metrics. They could observe the system using up stream metrics already. Can you name at least one concrete use case that can benefit from this?”

“Maybe our performance investigation?” The SDM said.

“The metric system aggregates at minute level. For performance investigations we need second level aggregates. We will need to compute the measurement from raw logs.”

While the SDM initially perceived that resolving the log loss issue would universally benefit the on-call operators, a deeper dive revealed that the incomplete metrics were not critically needed. A more pressing matter was to collect CPU utilization metrics. We can fix it with much lower cost because they are low volume data stream.

The moral here is that picking the right problem, involves deep thinking and first-principle questions about who will be delighted and by what. This process helps to uncover the real problems worth solving and investing in, to deliver genuine value to the customers and stakeholders. It's about the art of picking the right problem for the right people at the right time, a skill that differentiates a good leader from a not-so-good one.

“The answer is 42… what was your question?” — The Hitchhiker's Guide to the Galaxy