As I’ve moved into my new operations role, I’ve been continually struck by just how much everyone around me knows. They all seem like polymath geniuses, weaving together a dozen different knowledge domains to come to startlingly insightful diagnoses of intractably obscure problems. Coding a clever algorithm is all very well and good, but that doesn’t get you very far when dealing with the subtle interplay between apache, rsync, a point version of the Linux kernel, and Java garbage collection.
There’s a lot to learn – which is a good thing, since that’s a major reason why I wanted to make this change.
Still, there’s this awful feeling in the pit of my stomach – how long will it take? I distinctly remember having the same feeling after taking three years of Japanese, a moment of clarity in which I realized I’d have to study – hard – for another three years to get to fluency. The learning process can be rewarding, but it’s only when you’ve truly mastered a subject that the really interesting experiences open up. You can have fun as a beginning chess player, rock climber, pianist, programmer, but the most exciting and rewarding experiences are only open to the adept. You won’t be joining any epic quests with your 11th level orc shaman. You won’t be unlocking the deep mysteries of the Sicilian Dragon as a 1600-level player.
Logic, knowledge, and complexity
But what does it mean to really get to expert level? What secrets does the Grand Master of Flowers hold? In general, there are two kinds of problems, and it’s been interesting to see how product development and ops require very different types of skill sets to succeed.
The first kind of problem is algorithmic. There’s a logical puzzle, inside a functional conundrum, wrapped in a recursive enigma. You need to figure out a good schema, architect your class structure, pull in appropriate libraries and data structures, write your code with an eye toward time and space complexity, think through various tradeoffs, and so on. You go through an iterative code-and-debug cycle, and eventually you end up with something that works, more or less. Anyone can look at your code and, with enough work, follow the flow of control.
The second kind of problem is knowledge-based. You have to know the precise incantation to set up the reverse proxy in the public-facing DNS server, and more importantly, to know that that was actually the solution to the problem in the first place. Knowing the deep details of diverse sets of arcane tools, being able to tie them together, and understanding how and why they might break, requires years of wading through incomplete, out-of-date, poorly written documentation (when it even exists), old forum posts, frustratingly vague (or completely useless) error messages, change logs, security alerts, config files, RFCs, and errata.
I’ve met college students who could solve any algorithmic problem you set to them. But even among senior engineers, it’s the rare bird who understands the interconnections between the technologies in their stack at a deep level. They’re the ones who’ve been running their own DNS servers for years because, well… BECAUSE. They built their own machines and used Gentoo when it was a thing because that was the nerdiest distro at a time when only nerds had even ever heard of Linux. They play with Arduino and Lego MindStorm and MakerBots and Raspberry Pis. And like magpies, they gather bits and pieces of info from a hundred different places, until over time it starts to fit together and make sense.
When you move to a new town, you know how to get from one place to another, but you don’t know how things relate. Over time, you learn alternate routes, and where the different discrete destinations are in relation to each other. Likewise, when you know one particular technology, you may be able to complete specific types of tasks, but it’s understanding how multiple technologies interrelate that allows you to address novel issues.
The problem in ops is that when things blow up, it’s rarely one technology, library, or point release that’s at fault. Each functions perfectly well in isolation – bad things only happen when they come together, and suddenly an obscure bug in one interacts with an undocumented feature in another, which eats up a resource needed by a third. BOOM. There’s no shortcut to this knowledge – smarts are a base requirement, not a fast-track to success. That’s why the idea of a secret guild is so attractive, however seriously (or not) it was intended.
Being a pleb
We all have excessive demands on our time – I have small kids, trust me, I know – and it’s hard to prioritize learning when you can get by on existing skills. It’s easiest, of course, when you can create forcing functions – e.g., when I was 23, I took a job as a technical trainer in Tokyo because I figured that teaching eight hour seminars in Japanese would be a good way to polish my language skills. My first car had a manual transmission – even though I didn’t know how to drive stick – because I didn’t want to pay money to avoid learning how (manual transmissions are cheaper). More recently, I switched from a middle management role to an individual contributor operations role. It was a lot easier to throw myself into something new as a 23-year old – I didn’t have a career, a position, a reputation, expectations. I wasn’t starting over – I was just starting. I didn’t have any pride tied up in what I was already doing.
It’s hard to start over again as a pleb. Older, more expensive, and less useful than the junior members of the team. Trying to make up for a lack of domain knowledge with an understanding of software engineering principles, planning, and communication, while scrambling to come up to speed. It isn’t comfortable, and it’s a challenge not to fall back on well-worn skills.
Forgetting, remembering
Even worse than not knowing is having forgotten. To know that you used to be better, to compare your current performance to an idealized past, and to know that you’d have to expend a lot of time and energy to become almost as good as you used to be. Not as good. Not better. Almost as good. If you put in a major effort. Ex-musicians know this better than anyone, I suspect, though I’ve certainly experienced a good deal of it with Japanese. Most people never give it more than a token attempt, and I’ve certainly met my fair share of senior architects and technical managers who couldn’t remember how to write a single line of code in any language.
Strangely enough, interviewing has been one of the ways I’ve kept my programming skills intact. Speaking with my (two year old) daughter in Japanese has been a way to revitalize my Japanese skills. My son’s interest in climbing has gotten me back to the rock gym. You use the tools at hand.
Eminence
There’s one other curious thing about knowing. People who specialize for a long time, who fall too deep into the gravity well of a particular domain, ultimately become incapable of escape. You know your tools so well that it’s easier to force a problem to conform to them than to learn new ones. It’s easy to feel this in oneself – I’ve used an open source stack for the last decade – could I really imagine myself working on IIS and .Net, even if that were the right choice for a particular project? Could I choose the best tool, even if it were one for which I felt a deep personal distaste? It’s easy to get caught in local maxima – knowing just isn’t enough.
The goal, clearly, isn’t to stop knowing – it’s to not stop learning. And more specifically, not to stop learning about things that are way outside the bounds of anything you currently need to know. Jim Brikman recently wrote a great article on programming languages and paradigms you’ll almost certainly never use, or even want to. So why spend valuable time learning them? Why learn a functional language if you work at a Java shop, or a rapid web development framework like Rails if you’re working on enterprise software? Why learn Erlang, or Angular, or Go, when they don’t have any relation to your current job? When they’ll never have a relation to your current job?
Maybe because you’re trying to stretch your brain. Or because you’re trying to develop a deep understanding of the connections between different technologies. Or maybe, just because.
Dan Dreams of Coding 