Consider the generalist software engineer. She has a solid grasp of CS fundamentals, knows several programming languages, can write scripts, program user-facing front-end code, and work on deep algorithmic problems in the backend. For her, coding is a set of logic puzzles, debugging, patterns and anti-patterns, architecture, and tradeoffs between fast, cheap, and good. Most developers fit into this mold – experience and skill levels vary, of course, but given a novel problem, they can usually solve it from first principles.
This is all well and good, but some problems require a specific expertise. When I was working at Angel Studios, I spent a fair amount of time working on the physics engine – but my work was limited to architecture, optimization, tools, and ray/polygon intersection. The hard part – figuring out what would happen when two solids crashed into each other, or how a car with shock absorbers should bounce on uneven terrain – was handled by physics PhDs. It isn’t that they could do it faster, or come up with a better solution – it’s that they could do it at all.
The physics example is a bit on the extreme side, but in a similar vein, over the past couple years several “special” skills have come into high demand in the web industry. These don’t necessarily require PhDs (though in the case of data science, it helps), but they’re deep fields, with a lot of domain-specific knowledge that simply must be learned. Getting past beginner takes a while, and that expertise is valued accordingly.
This has been growing for a while, with iOS developers being historically prized. What’s been interesting is the degree to which Android developers have also been targeted over the past year or two. This isn’t just a matter of knowing Objective C, Swift, or Java – rather, knowing effective patterns, the hidden gotchas, how to architect an app for simultaneous development by multiple developers, how to develop for multiple screen sizes, OS versions, and configurations, familiarity with key open source packages, knowing how a platform feels, etc. – all have continued to increase in value alongside the ongoing explosion in app traffic and commerce.
While the definition may vary depending on whom you talk to, the basic idea of taking an agile, software engineering approach to ops, and treating infrastructure as code, are pretty standard features. At the same time, the rise of virtualized environments (AWS, Heroku, DigitalOcean), and open source and SaaS-based tools for monitoring, configuration management, continuous integration/delivery, container-based deployment, dynamic capacity scaling, etc., have created a need for experts who can architect flexible, robust, reasonably priced operational environments at a fraction of the effort required in the old days of dedicated and colocated servers.
Data is pretty much useless unless you know what to do with it. This isn’t restricted to the nuts and bolts of setting up an analytics infrastructure, logging, archiving, ETL, data warehousing, and hooking up visualization/query tools to your warehouse – these can, by and large, be handled by generalists. Data scientists create value by thinking through what data needs to be captured, analyzing the data in a statistically significant manner, creating taxonomies to annotate and add meaning to the data, and most critically, by knowing how to ask the right questions and run experiments to achieve business objectives. Driven by massive data sets, cheap computational power, and cheap disk space, machine learning has also taken off as an important sub-specialty of data science.
What this means for you
Hiring is not a buffet, where employees can be constructed to order with the precise skills desired. Being a generalist with a high-value add-on skill gives you a tremendous advantage, especially in smaller companies where full-time single-skill specialists are an unaffordable luxury.
Larger companies may have teams of people working in each of these areas, but smaller companies will frequently need someone who can do some initial set up, fold back into main product development, then take on additional special projects as they get prioritized. This is far better than relying on a contractor – the engineer is embedded in the team, diffusing knowledge, sharing best practices, giving code reviews, and shifting her workload as necessary to accommodate changing or emergent priorities. As the company’s needs grow, this creates opportunities for her to continue to build skills, and to take a leadership role in building a team of specialists, or stay on as an ambassador in the main product group.