Goldilocks process

For the last six and a half years, I’ve worked in a “speed wins” culture of quick iterations, speed of execution, done is better than perfect. A culture that believes that the faster you go, the more you can iterate, the closer you can curve-fit to the user’s needs. Of course, it doesn’t always work out that way. Sometimes, you break things. Sometimes you break important things, and the company loses money. Sometimes – not often, but sometimes – you break things so badly that servers go down and the site is put at risk. And so we’ve worked hard to create a process and infrastructure which prevent or mitigate the vast majority of problems, and provide an escape hatch in the worst case scenario.

We have several staging environments, each of which runs a different set of automated/load tests. When we release new code, we release to a limited number of servers, watch for new errors and system instability, and only roll out to the full site when we’re convinced things are reasonably safe. We release twice daily, so there are always opportunities to fix emergent issues. And if all else fails, we can fail over to a secondary data center.

Any company with an effective process will be thinking about these kinds of tradeoffs. For us, the natural antagonism lies between moving quickly, stability, and technical debt. We emphasize speed, but still need to balance the needs of stability and long-term code maintainability. You can easily imagine a company that went in the opposite direction, with a “slower is faster” culture that focused on stability, maintainability and long-term development speed – at the price of fewer iterations, slower product cycle, etc. Or a company that focused on design and polish over schedule and cost. And so on.

Each of these is a choice, and each has its place – you wouldn’t want Boeing to “move fast and break things,” for instance, and Apple wouldn’t be Apple without their design focus. The danger comes when there’s an unwillingness to identify, admit to, and mitigate the tradeoffs in culture and process. Moving quickly only works if you understand the dangers and prepare for the inevitable breakages.

Even as an individual contributor, it’s important to sit down and think through these tradeoffs. It’s not enough to take a mantra at face value – as our CEO likes to remind us, “speed wins” doesn’t mean it’s ok to ship crappy code as long as you do it quickly. Everyone wants to have it both ways, and it’s precisely your ability to work according to the cultural norm, while managing the downside risk, that will determine your success in any organization.

2 thoughts on “Goldilocks process

  1. I agree. I’ve also noticed that there is a difference, across companies, in the extent to which tradeoffs can be explicitly discussed. At one large internet company where I worked, there was a big cultural emphasis on code quality and extensive reviews. My opinion was that much of this effort was bike-shedding that provided a poor ROI. However, I quickly discovered that I couldn’t raise questions about the cost/value out loud, because professing agreement was actually a way of demonstrating one’s membership in the group, and by questioning it I was branding myself an outsider.

    I remember you wrote another piece a while ago about the different types of conversations which happen at different levels; it’s possible that these conversations about tradeoffs were in fact occurring, but at the time I was not at the correct level to be participating in them.

  2. Completely agreed. Keeping a balance between speed release and robust release is important. It also boils down to the product, domain and competition. I once worked on for a travel company where many a times a new feature with high priority meant a source of revenue.
    In one of the new requirements, the product team asked for a quick dirty solution in a week. We delivered it but its scope increased to other lines of business which needed a re-design and development work of extra 2 weeks.

    A quick approach can sometimes lead to dirty patchwork in the code. Due to speed releases, there’s not much time for peer review. This reduces the code quality and maintainability.

Leave a comment