This is the fourth part in a series about achieving mastery as a software engineer. The first part described senior software engineers. The second part discussed common flaws that, quite apart from coding skills, could undermine an engineer’s professional and personal growth. In the third part I got down to business and talked through the foundational skills necessary for starting on the path toward mastery.

In this post I’ll discuss how one moves from being a competent mid-level engineer to a more senior level. Of course there are many paths up the mountain, and different problem domains will require different specific skill sets. But in the same way that all black belts need to know how to block a punch – regardless of the martial art – there are some skills and approaches that are required for mastery, no matter the domain. This isn’t just a matter of coding a lot – though that’s certainly critical. You also need to have the underlying knowledge and habits of thought to benefit maximally from the experience. If not, it’s all too easy to get stuck, and fail to gain the full benefit from the work you’re doing.

Context

When you first move to a new town, you quickly learn how to get from your house to the store, to school, work, the gym, and any other frequent destination. In the beginning, you don’t know how these interconnect, and wouldn’t be able to get from one to another without a map – you just know the specific series of turns to get from home to each place, and back. Over time, you develop a mental map of the city – where things are in relation to each other, alternate routes, etc. The more context you have, the easier it is to fit new things into the map (“oh, it’s two blocks down and to the right from where the supermarket used to be”), and to improvise when there’s traffic or a street is blocked.

Similarly, the more context you have about how things work, the more you’ll learn from your projects, even absent feedback. Without context, you can write the same bad code ten times. When you understand how things work at a deep level, you may make design tradeoffs due to time or resource constraints, but you understand what you’re doing, and have insight into the results when you’re done. There are a couple of key components to developing this kind of contextual awareness.

Do it yourself

When you work within a framework set up for you by someone else, your understanding of the choices implicit in that setup is necessarily limited. Maybe they were good, maybe bad, but ultimately you’re a tourist, not a native, and you can’t truly understand the tradeoffs at a deep level. Throwing something into the microwave isn’t cooking, and using a cake mix isn’t baking. You have to be comfortable setting things up on your own, doing everything from scratch. Doing so gives you a deeper understanding of the consequences of your choices, and the confidence to start over if need be.

• Set up your own machine

It used to be a rite of passage – buy all of the components individually, assemble them at home, and hope that things worked when you turned on the machine. Nowadays, the Raspberry Pi is probably about as close as most people come to this – it’s too easy to buy something off the shelf – but you should be comfortable moving around or replacing hardware components.

When you do get a new computer, max out the memory, erase the current OS and install a new one from scratch (also a good way to get rid of bloatware). Install your own development environment, IDE, source code repository, and custom tools. Know where everything on your computer comes from. Build open source software from source, instead of using pre-compiled binaries. Setting everything up yourself will teach you things that you’ll never learn if you just accept the defaults.

• Assembly language

You don’t have to be able to write complex programs in assembly language, but a couple of small projects will dramatically improve your understanding of pointers, registers, the stack, loops, and the importance of initializing variables. When you program in another language, you’ll automatically think about what it’s actually doing under the covers. There are lots of ways to optimize assembly code, from paying attention to the number of cycles different commands take, to thinking about instruction pipelining and branch prediction, to decrementing instead of incrementing a loop counter. As Donald Knuth said, “premature optimization is the root of all evil” – most of this won’t (and shouldn’t) make a difference on a day-to-day basis, but it will be there in the background, context that will help you understand your code at a deeper level.

• Start from scratch

When I was trying to break into the video game industry, I wrote a 3D engine from scratch (including writing the rasterizer in assembly). This was in the days before widespread custom rendering hardware, and before DirectX was a reasonable choice. And besides, it was fun. But more important to this discussion, it meant that when I did get a job writing 3D video games, I could look at someone else’s render engine, compare it to what I’d done in the same situation, and learn¹.

When I started working at TripAdvisor, I was coming from a startup in which I’d built a website from the ground up – chosen (and learned) the technologies, set up the servers, negotiated with the hosting facility, organized the code tree, and written the code. Looking at the TripAdvisor code base was an incredible eye-opener and education, which I would have benefited from less if I hadn’t had the experience of having rolled my own.

Obviously, these are examples, but the basic principle holds true – building things from scratch forces you to confront all the ugly little details that you can ignore when working in someone else’s framework. This, in turn, gives you the context with which to understand when you see how others have faced the same problem. Also, the more you do things yourself, the less you fear it, and the more confident you are when trying something new.

Know your Domain

It would be nice if there were a single canonical set of knowledge necessary for achieving mastery, but every domain has its own quirks and requirements. The more senior you get, the more depth of domain knowledge matters. I’ve worked with brilliant college grads who knew their theory, coded fluently, and could solve any coding problem they ran into – but there’s something fundamentally different between them and someone who’s been working with the JVM for 15 years, knows the differences between point versions, is familiar with the various command line options, knows how just-in-time compilation works, and closely follows updates to the garbage collection framework. Both of these kinds of programmers are great to work with, and a team needs as many of both as it can find – but where the former will do great work, the latter will be able to solve problems that seem beyond mortal ken.

• Frameworks

Part of being effective in a particular technology is knowing the different frameworks (open source or otherwise) that have already been developed to solve common problems. This includes knowing which are in active development, have good support/communities, are more feature-rich vs. performant, are better for this use or that, and of course which to avoid.

• Tools

When watching a virtuoso violinist play a difficult piece, one has the sense of watching someone with two elbows per arm, not one. The violin and bow are natural parts of her body, moving under complete, almost (it seems) unconscious control. You need to have the same level of familiarity with and control over all of your most frequently used tools, including installation, usage, configuration, options. Tools are usually far more powerful and flexible than their users realize, typically because their creators tried to solve a variety of related use cases, not just the most common one (this is particularly true of *nix commands – e.g., how many people only use sed for search-and-replace, or find for finding files?). Understanding their full capabilities can dramatically improve your effectiveness.

• Industry knowledge

Working in an industry for a while will teach you a lot about the basic parameters for dealing with common problems. This helps develop your ability to evaluate technical ideas, as well as your capacity for being impressed. The former for when someone suggests an idea that’s impractical, inefficient, or just plain wrong. The latter for when someone demonstrates something that shouldn’t work, but does. Wow! How did they do that? Conventional wisdom can be a trap, of course, but making decisions without knowing how things are typically handled means that you have to invent everything from scratch, a time-consuming and error-prone business.

Furthermore, your industry’s tools and conventional wisdom are constantly evolving. It might mean reading HackerNews, related blogs, periodicals, or going to meetups, user groups, or conferences, but keeping up-to-date on industry trends is an important part of being an expert.

• Have a long tail of knowledge unrelated to your specialty

It may seem strange to put this under the category of “knowing your domain,” but it’s the flip side of knowing a lot about a specific thing. Knowing a little about everything else is hugely valuable, as you’ll be able to make connections that would be completely opaque to a single-subject zealot. It also removes the psychological barrier to diving into a completely different area – if you have to set something up in framework Foo using language Bar, it’s a lot easier to get started if you’ve already fiddled around with one or both in the past.

Advanced Topics

In the same way that being comfortable with technologies like OOP and SQL is an important basic requirement for moving past the most junior stage of your career, intermediate mastery requires a solid understanding of more difficult ideas. Theory and praxis are both required at this point – you need to understand things abstractly, as well as to be able to implement them using the concrete toolset at your disposal.

• Concurrency

Whether it’s to queue asynchronous operations, organize parallel tasks, or efficiently utilize multiple cores, being able to write concurrent code needs to be a well-oiled part of your programming toolbox. Different languages have different ways of handling concurrency, and you should know and be comfortable with the syntax, libraries, and best practices in your language of choice.

• Memory Management

As the problems you face get more complicated, the more you have to worry about how memory management actually works. Sometimes this is simple (as in understanding the difference between malloc, alloca, and calloc in C), sometimes it’s more complicated (e.g., having deep understanding of your language’s garbage collection mechanism). Most modern languages don’t allow you to explicitly free memory, which actually makes this more difficult, not less. You have to worry about memory leaks in C/C++, but understanding the subtleties of garbage collection is a discipline in and of itself.

• Architecture

You need to be able to take a large project, break it down into components, organize the code hierarchy at a high level, and structure things so that other engineers can work independently on smaller pieces of the problem within the larger framework. This can include choosing technologies, defining interfaces, thinking through memory footprint, performance criteria, automated testing, metrics/reporting, how to phase the project, and so on – all while communicating with the business partners to ensure the plan matches the requirements.

• Debugging

It may seem odd to include “debugging” in the list of “advanced topics”, but one thing I’ve noticed about very strong developers is that they have more tools at their disposal when tracking down a problem. Junior developers may diligently go through the paces to try to find a bug, but when they get stuck, they frequently devolve to randomly changing variables without any logical pattern, hoping that something gives them a clue. Sometimes this works, but usually it doesn’t – at which point they have to get help from someone more senior. Senior engineers are much more methodical and relentless, have more places to look for clues, more tools to analyze data or profile code, and more strategies for dissecting a problem. Debugging skill is one of the key differentiators between junior and senior engineers, and a crucial skill to develop on the way to mastery.

Electives

It’s important to have one or more less common areas of expertise. Not just because it’s interesting or fun, but also because opportunities open up when you have a super power. No one of these is required per se, but not having expertise in anything other than your core skills is surprisingly limiting. You don’t have to be a world-class expert, but even a little real-world experience will be useful in evaluating problems, determining approaches, and giving you a leg up when coming up to speed on a new project.

• Statistics, Math, Physics

3D graphics is basically a lot of linear algebra. A lot of data analysis comes down to statistics. Physics engines are used in everything from 3D games, CAD, and visual effects, to Cut the Rope and Angry Birds. Knowing even the basics of one or more mathematical disciplines can be very useful – because they provide tools that go beyond the basic algebra that most engineers are limited to.

• Advanced data structures

In Mastering the Basics, I talked about how understanding the core data structures at a deep level would make you a dramatically better programmer. There are many, many more data structures, from suffix trees and tries, to a menagerie of different heap types, k-d trees, and so on. The reason you don’t know most of them (unless you’re a researcher) is that, for the most part, you’ll never use them. However, knowing what’s out there, and understanding the basics of their use cases, will give you a powerful tool when you run into one of those situations.

• Foreign languages

Foreign language expertise comes in handy more often than you’d expect. Of course, fluency is preferable, but even just knowing about different languages (e.g., this language has declensions, that one tends to have long words, another doesn’t have spaces, and yet another is right-to-left) can be helpful. These are all pretty basic facts, and are easy to learn even if you flunked out of high school French.

• Operations
• Networking
• NoSQL
• Big data
• 3D graphics
• AI
• Machine learning
• And many, many more…

What’s left?

There’s a lot here, and for the most part it isn’t simple “learn over the weekend” type stuff. Building your skills and getting to mastery takes years. To be honest, achieving the “intermediate mastery” described in this post is pretty rare – most people stop somewhere along the way – and a handful of engineers at this level are at the heart of most successful companies.

In the next post, we’re finally going to get to the top of the mountain, and discuss the difference between experienced, highly capable, superior engineers, and the freakishly amazing superstars that make the world run.

[1] I remember picking up Michael Abrash’s Graphics Programming Black Book, and being dumbstruck when I realized he’d done in a dozen lines of code what had taken me hundreds of lines of special cases. Stay humble.