Interviewing is a strange process. Most of the time you answer the same question over and over again, asked in different ways – “What kind of a role are you looking for?” “Let’s go through your job history.” “Describe the architecture of a system you’ve implemented.” But every so often there’s a new question that forces you to think through how you feel about something important.

A couple years ago I was asked by a hiring manager, “What was your worst hire?” I’ve done a lot of recruiting, hired a lot of people, and sure, not all of them have been great. There was that one toxic dude a couple of decades back, some people with low motivation, and one or two who just couldn’t perform the technical tasks. But as I considered, I realized that the people who caused the most heartache were the ones who were always on the edge of being fired.

These are the people you spend an inordinate amount of time trying to help. They’re likable, you want them to succeed, and if you just spend a little more time working with them, maybe you can help them level up. Sometimes there’s a skill gap, sometimes it’s output, or quality, or focus. This one does great work, but only on the projects they’re interested in. That one repeatedly wastes everyone’s time with pull requests they clearly haven’t tested. Another has low energy and drags down the team.

It’s a truism that you should spend the most time with your top performers, but it’s extremely hard not to keep trying to pull up your weakest team members. And sometimes you can! Sometimes you’re able to help them work through their specific issue. But a lot of the time you can’t, and you spend a lot of time and effort mentoring, sharing strategies, discussing issues, and going round and round with exactly zero net positive impact.

But back to the original question. OK, let’s talk about specific person X. I hired X, they had problem Y, and I just wasn’t able to help them improve. And furthermore, when I looked at my interview feedback, I saw that I had correctly identified Y as a potential problem during the interview! What would I have done differently?

At this point, it would be easy to say that of course I’d learned my lesson, was weighting Y more heavily now, and would avoid the same hiring mistake in the future. But that isn’t true. I would absolutely hire a similar person. I would hire ten people like this. Why?

Annie Duke is a world championship poker player. In her book Thinking in Bets, she describes how to think about decisions in which you don’t have full information. She spends a lot of time discussing poker, in which one of the key errors that amateur players make is “resulting”. I.e., focusing on the results of a particular hand, instead of focusing on the decision-making process that led to a specific choice – call, raise, fold. You might win a bad hand due to good luck, or lose a good hand due to bad luck, but poker is ultimately a game of volume, and winning means coming up with a strategy that is more likely to win than lose over hundreds of hands – and then sticking with it. You shouldn’t start betting on inside straights just because you got lucky that one time, and you shouldn’t fold your full houses just because you lost to a better hand once. This doesn’t mean that you shouldn’t seek to improve your strategy or develop new sources of information – but those improvements should be based on a better understanding of the game as a whole, not based on good or bad results that were due to luck.

This can be frustrating. In the end, the only thing that matters is results. FedEx wouldn’t exist if its founder hadn’t gambled the last of the company’s cash on a game of blackjack. But when you’re building systems, you have to build them with processes based on your best understanding of reality, not individual lucky results (good or bad). This is hard – it’s easy to believe that your good results were based on skill, and your bad results were bad luck – but it’s essential.

Back to hiring. Every hire is a probabilistic risk. And depending on the number of hires and the type of role, your risk profile will differ. You probably want to accept less risk for senior leaders, architects, and SREs, and be more risk-tolerant for junior engineers. A larger organization can afford more risk because you’re generally increasing capacity, not fundamentally altering the engineering organization. A smaller organization might have to accept more risk due to timing and concerns over business continuity (e.g., if your last engineer unexpectedly quits, you’re going to need to scramble to hire, and quickly – you need to balance risk of a bad hire vs. risk of no hire).

One way in which companies go from small and nimble to big and slow is by the accretion of time-consuming rules to prevent one-off outages or failures. An engineer did something bad once, so now all engineers have to follow a process that imposes a small tax. Not a big deal individually, perhaps, but in aggregate the epsilons of time and frustration add up until suddenly you realize that you can’t get anything done. Or you’re so scared of a false positive that you create an extraordinarily restrictive hiring process that wastes time and money, and prevents you from reaching your hiring goals.

So where does that leave you? The point isn’t that you should hire everyone, or that you should get rid of process, or delete your automated tests. The point is that you should approach the problem with a systems mindset. When you get a bad result, you need to understand when it was due to a fault in the system, and when it was due to the risk that you intentionally accepted. Or when it was due to a misaligned level of risk tolerance, or the impact was worse than planned.

Every hire is a bet. And bets imply limited information. The question isn’t whether you have an ironclad guarantee of success, but how good a bet you think someone is. At the time, I didn’t know how to interpret the information I saw in the interview with the toxic dude, but have since used the experience to make the system better. Conversely, if 90% of the borderline cases turn out well, it would be a bad strategy to rule them all out to avoid the one case that was mediocre – especially if I can develop a system to address poor performance after the fact.[1][2]

---

[1] What about “bar raisers“?

Amazon includes an additional interviewer to make sure that every hire “raises the bar” in one or more competencies. The goal is to make sure that teams don’t hire to meet short-term needs, while lowering the overall quality of the organization. To the degree to which this increases the objective data about a candidate (reducing the error bars), encourages teams to maintain a high level of quality (because there’s an external observer), and results in a more effective process, this is great! But it doesn’t change the underlying probabilistic nature of the problem.

The specifics of Amazon’s hiring process and compensation structure are waaaay outside the scope of this blog post. I don’t know enough about them to comment on their fairness, or the degree to which the “bar raiser” performs a useful role. But the goal is in line with trying to improve information gathering and reducing the error bars.

[2] Toxicity is something you should never accept risk on.