You may have often heard this question: “If we listen to only this customer, how can we be sure that we are not overfitting?” This is the question that leads to the death of the project. This is such an important topic that I decided to give it a dedicated article, so today, we are going to talk about the following:

  • Why is directly building generalized products dangerous?
  • Why is overfitting for just one customer a much better way to end up with a generalized product?
  • Why do many senior leaders in the industry think that building generalized products (and not overfitting) is the right way?

To illustrate the article’s main point, I will tell a story. The story is about two tech leads (names and circumstances are fictional):

  • Let’s name the first tech lead Romeo. He is practicing “user oriented development process” UODP and knows the critical rule: Onboard one customer, build an overfitted solution, and find a way to generalize later. Romeo is prioritizing learning. Learning new information by releasing an early over-optimized solution always trumps spending resources later to rewrite the solution.
  • The second character — let’s name him James — is a classic product lead who thinks he needs to design the system correctly and build the generic solution from the beginning. James is prioritizing the reduced waste of engineering resources.

Let’s see how the industry at large has been forcing people like Romeo to become more like James.

Let’s assume that James is tasked with building an internal service for his company. This service should connect those who would love to be mentored with those who would love to be mentors. James starts with collecting requirements for the future service, meeting customers, and planning how the website would look. He talks to as many customers as possible and asks questions like the following:

  • What topics might you be interested in learning?
  • Are you interested in teaching, just learning, or both?
  • How many hours per week can you commit to this task?
  • How much time will you need to reach your goal?

After some investigation, James comes up with some high-level user journeys and requirements:

  • mentor/mentee account-hosting site
  • accounts that include these elements:
  • - fields the person wants learns about
  • - subjects the person is interested in teaching
  • - time zone
  • - time preferences
  • matching tool
  • progress-tracking tool
  • search ability

The first milestone (V0) is as simple as possible: Just have a portal for mentors and searching functionality. This still requires the following things:

  • authentification
  • personal profiles
  • searching

James estimates three months for the full V0 rollout and six to nine months for the entire project.

At the same time, Romeo’s organization has a pressing problem of ever-growing requests to match mentors with mentees. To address the situation the fastest way, Romeo creates an intake form to collect folks who want to be a mentor and Google spreadsheet for storing mentors and mentees. He shares it with everyone in his organization. A month later, this spreadsheet became several organizations’ primary source for finding mentors, but it still lacks some things. The most pressing are these:

  • pinging mentors periodically to see if they still want to mentor (and removing them from the list if the answer is no)
  • displaying mentor status (We already have enough mentors/openings for now.)
  • sending a notification when the mentor with the right skills is registered

None of these requirements have been identified by James. It is mainly because customers do not know they are critical until they start using things. To address this, Romeo introduces the first service and the first DB created to accommodate these needs. At this point,

  • Romeo creates a “horrible service” from an engineering perspective. For example, it does crazy things like joining Google spreadsheet with DB and requires complex on-call actions from the team. Romeo will definitely need to rewrite everything eventually. But he has customers and can start scaling the solution. Romeo has a service that delivers real value.
  • James does not have customers yet and is unaware that several critical requirements are not reflected in his service. He is on track to release V0 in several weeks. He slowly starts realizing a painful truth: that all his potential customers from Romeo’s organization already have a solution that looks better from a customer perspective (while definitely worse from an engineering perspective).

A quarter into this project, James has zero chance of winning any customers. Not only are prospective customers already using the solution from Romeo, but V0 also lacks critical features (while providing many additional but not critical features). Leadership reasonably decides to kill James’s project and redirect headcount (HC) to Romeo’s team. Two to three quarters later, Romeo’s team implements features that were in James’s original proposal. They implement a proper service, depreciate Google spreadsheet, and have an adequate web UI — essentially everything that James’s team initially proposed. What is especially frustrating for James’s team members is that they were well positioned to implement similar things much more quickly.

Romeo’s way provides more value to customers faster. So why do people like James keep emerging in highly influential engineering leadership positions? There are several reasons, and the number one is “hindsight bias.” To illustrate how it plays a role here, let’s assume there was no James in our story, only Romeo.

After implementing his solution, for some time, Romeo has to support a complex service to correctly map records in Google spreadsheet to the DB records. Since the tool (the spreadsheet) is highly inappropriate for the task, this service is slow and has many bugs. Later on, Romeo has to rewrite everything to remove the spreadsheet altogether. This required a costly and lengthy customer migration from one source of truth (spreadsheet) to another, DB to Web UI. Migration was not the smoothest, with constant customer complaints. All in all, since each version of the service solved the main customer problem at any point, the service always had customers, and no one turned their back and left.

In hindsight, since there was no James, it might look like directly building the “right service” would be faster than what Romeo did. In the debrief, everyone would be pointing out that:

  • At least two times, the team had to throw the solution out and rewrite it.
  • Migration was complex, and many users were unhappy during it.
  • The intermediary solution was difficult to use and hard to maintain (on call was quite intensive).

With such “lessons learned,” it might seem like Romeo did everything wrong and should not be allowed to do projects like this anymore. So, when the next Romeo (let’s call him Romeo the Second) emerges in the company and suggests solving the problem in the simplest possible way, the original Romeo might start advising him in this way: “You should not think short term; let’s properly design scalable solutions so we can adequately accommodate all the required customers and not just the one. Let’s reduce waste.” This is how Romeo becomes just like James.

In the mind of the original Romeo, we went via the path that looks like this:

So it makes total sense for the next time to go like this:

Now, in reality, if Romeo would be honest and look back to see where we thought we should be aiming, he might find that initial ideas about a destination:

  • does not have features (like notifications) making it unusable
  • have too many features that are not critical or required

However, how often during debriefings do you hear questions like:

  • How many things have we learned?
  • How many problems are we now aware of that were hidden at the beginning?

Usually, these questions are not asked. It is almost assumed that the information we have now was always available.

This story in UODP illustrates the following rule: “It is easier to implement products/features around one concrete customer and end up building highly used generalized products/features than building generalized solutions from day one and end up with a solution that can be used by at least one customer.” In short, “Building a solution for one customer quickly is better than building a generalized solution slowly.”

But you know what? Many folks agree with the statements here but will still do otherwise. Why? Because when you optimize for learning, you assume that you do not know something, UODP is all about getting the best outcome in uncertain conditions. However, many teams think their information is sufficient to make conclusions about customers/markets/products. It takes a lot of courage from the product manager to acknowledge how uncertain our understanding is of the customer and the market. This brings us to the second problem of convincing people UODP is the right way for them. A key point of UODP is: You do not know something; you do not know what you do not know, but you must constantly search for it. This argument is not concrete (since we do not know what we do not know). Meanwhile, people will have factual arguments why we should aim for the outlined target. They will have charts, poll results, and records of the interviews with customers. It will look like we know everything and are exactly where we need to be; it makes sense to build the shortest possible path to that place. This is how reducing uncertainty translates into reducing resource waste with full certainty.

After all, one can not quantify how much information you have not learned, so the horrible downside of building generic solutions is not visible. On the other hand, it is a more or less visible small amount of waste that generic solution allows to avoid. So we optimize not what is important, but what is simpler to measure.

The best product managers I have ever worked with shared my ideas about uncertainty. We agreed that the small details we do not know can have disproportionately significant outcomes on the product’s shape (hello, Taleb). Often, these details can only be found when an actual customer is doing the last of the integration.

Instead of saying how the product should look, the best PMs focus on finding the right customer to onboard next. It should be a big customer, so we will still profit if we only onboard her. At the same time, the customer should be willing to be a partner and work with us while we build a solution. You cannot onboard a customer who only talks to you once per quarter.