I’m lucky enough to have interviewed thousands — and hired hundreds — of the best engineers in the world. When I took the VP of Engineering job at Context Relevant, my mandate was to put everything I’ve learned from interviewing and working with these talented people into practice, and build the best engineering organization on the planet. So, how do I do it?
The key to hiring great engineers, is to, not surprisingly, hire people that are great engineers! Ok, that’s a tautology, how does one actually determine if someone is a great engineer? That’s what I’ll go into detail on. In this post I won’t cover how to find, or attract, or sell, or determine roles, or any of those other critical aspects of hiring and building a team.
So what are the attributes of a great engineer? There are of course many, but I believe that the key attributes, at least as can be discovered in a reasonable interviewing and hiring process, are:
1) Intellectual horsepower. Mental quickness. Smarts.
2) Solid working knowledge of software fundamentals
3) Ability to formalize abstract knowledge
4) Passion to create systems that have impact
5) Interpersonal skills
6) (Optional) Specific domain expertise
7) The value of experience
8) Team strength – the value of diversity
I’ll go into a little bit of detail on each one, why I believe it’s critically important in a great engineer, and a little bit about how I look for these things while interviewing.
Intellectual horsepower
This is absolutely critical, non-negotiable. Writing software requires making a huge number of decisions – it’s creating a specific product and solution of maximum value to a large set of people, with a literally infinite realm of possible choices. New information must be assimilated, new tools and techniques and processes constantly understood (and evaluated, for whether they make sense to use or not), various possible solutions proposed and weighted and discussed and improved, etc. An engineer’s day is spent doing mental work. A great engineer is able to do that mental work better.
How do I look for intellectual horsepower in an interview? At some point in the interview I’ll ask a discrete question that I expect the candidate to solve. Frankly, for engineers, I’ll ask them to design and code up the solution to a relatively simple problem – after all it has to be something can be done on a whiteboard in 30 or 40 minutes. The question is designed to build, and have multiple potential directions. I might purposefully leave some aspects ill-defined or incomplete, and see if they realize it’s missing and prompt me for what’s missing. I might propose or word the problem in a way that leads them to a straightforward, but non-optimal solution, and see if they realize this. I try to ask a question outside their domain of knowledge, to see how quickly they can learn and apply some new concepts, rather than test specific facts or memory. What I don’t do is ask trick questions – there shouldn’t be an ‘aha’ moment that they either get or don’t get.
Solid working knowledge of software fundamentals
Note that I did NOT say ‘solid working knowledge of technology X or language Y or operating system Z’. A smart, passionate person can learn any of those things quickly. In every single software course they took in college, they learned new technologies, and often a new language, and often working knowledge of a new operating system. So I’ll assume that working fulltime in an environment surrounded by experts in those areas, they’ll be able to quickly learn what they need. However, if they don’t have a strong working knowledge of basic algorithms, a solid understanding of how a computer system works, understanding of performance characteristics and tradeoffs etc., then I figure they’re likely not a great engineer. The great engineers know that stuff inside out.
How do I look for this in an interview? With the same question as before. It will be a pretty straightforward problem (I don’t want to spend more than 30 or 40 minutes tops on a single question), but designed in such a way that solving it effectively will demonstrate that working knowledge of software fundamentals.
Ability to formalize abstract knowledge
What do I mean here? I mean that I expect people to go from an abstract, high-level solution, or discussion of a number of possible solutions, and make all of the specific decisions necessary to turn it into a discrete answer. Writing software in a particular language is, at its essence, manipulation of a formal system. Let’s take any common procedural language. You have variables, types, control statements, function prototypes, data structures, etc. and there are specific rules for syntax, operator precedence, and so on. It’s a formal system, encompassing other formal systems (such as basic mathematical and logical operations, etc.) Can the candidate take an abstract solution (“I’d put those things into a list, then sort the list, then pull the right elements out, then use them to lookup elements in this other table..”) and turn it into real, working code? Some people are great at coming up with a high level solution, but struggle at formalizing it, turning it into something absolute and discrete, that fully does what they need it to do.
And of course, this is something that’s clearly demonstrated (or not) in the discrete question that I expect them to solve. By now you’ve probably figure out that it’s a programming question, and the solution will be working code. I don’t really care which language they use, hopefully it’s one that I have working knowledge of (I might suggest one, from the list of those that their resume says they’re ‘expert’ in, but I’ll always ask if they’re comfortable with that choice, and allow them to choose a different language if they prefer).
Passion to create systems that have impact
Software development is about creating systems. A software company is in building to build software systems. If they’re not systems that actually work to solve a real problem for a real customer, the company is going to have problems staying in business, as for some crazy reason, people only want to pay money for something that does something useful for them, and in particular, more useful than however they’re currently doing it, and enough more so that they’re willing to take the time and trouble and effort to evaluate and buy and install and learn and train and utilize this fabulous new system. The people building that system, need a ‘North Star’, they need a goal, they need a reason to get up in the morning (or maybe early afternoon after a late-night coding session, these are software engineers we’re talking about J), and do mentally exhausting work that often takes months and months, and sometimes years, before it gets in the hands of customers. A great engineer has passion about building software, about getting things done (and I mean done, i.e. completed, fully working, tested), and into the hands of customers who are going to be amazed and will do amazing things with this product, that will change their world, and the world of all the people that they interact with.
This can be a little trickier to interview for. Part of it is in seeing the sparkle in someone’s eye, and their excitement in talking about something really cool that they worked on in the past, whether at a previous company, or in a college project. Part is in evaluating whether they have the discipline and desire to complete things, and ensure that it’s done end-to-end, or do they enjoy learning and trying something new, and getting it partway done, but then losing interest and leaving it to others to finish (or perhaps just leaving it unfinished if there’s nobody to pick up the pieces).
Interpersonal skills
These are software engineers. Isn’t it fine for them to live in a cave and eat take-out sushi and guzzle coffee and green tea, and check in awesome code to the source control system, that someone else will then take and sell and deliver and install for the customer? Well maybe, if it doesn’t take a team of people to build the software, and they can infer all the requirements without talking to anyone about them, and don’t have any questions that they need answered, etc.
The reality is that building commercial quality and scale software is a team exercise. Engineers are part of a team, they have peers, they have a manager, there are customers whose requirements need to be met, even though perhaps another team member (the program manager or product manager) might be responsible for understanding those and turning them into software requirements. A great engineer knows that they don’t have complete knowledge, so knows when (and how) to ask questions when they need more info. For the team to function well, team members need to provide input, and support, and mentoring for others on the team. They need to ensure that different parts of the product interact well, and that they fully understand the interfaces and data structures and purpose of all the components that they interact with. They need to work with engineers on the QA team that will torture the software in unnatural ways and ensure that keeps functioning.
There is plenty of interaction with others required in software engineering, and though an engineer doesn’t need to be able to successfully run for elected office, they do have to be able to interact with others in a way that ensures that everyone enjoys working together. People often spend as much or more of their time at work as they do with their families, and for all kinds of reasons, people need to enjoy doing what they’re doing, and enjoy the people they’re doing it with.
Domain specific expertise
I listed this as optional. People often interview for specific knowledge. Specific languages, specific tools, detailed knowledge of specific areas. I’m not saying it’s not important, for some roles it’s of course critical. You can’t hire a data scientist that doesn’t have strong working knowledge and understanding of statistics and machine learning. But I’ll hire a super smart, passionate, easy to work with software generalist that can write great code, straight out of college with no industry experience, any day of the week. They can’t do all the jobs, there will always be specific jobs and roles that require specific domain expertise. But many engineering jobs really don’t, I might even say that most don’t.
The value of experience
Ok, so other than domain expertise, listed as optional, none of these aspects appear to value experience and knowledge. This is crazy, we all know that experience and knowledge are super valuable, right? Of course they are! However, I don’t consider them to be the primary selection criteria of a great engineer. What experience and expertise does, is make a great engineer even better. It does this by allowing them to make better decisions faster, and of course speed up implementation. It doesn’t however obviate the need for the other aspects listed, it just enhances them, it turbo-charges them. Of course a great engineer with ten years’ experience in distributed systems will tend to outperform a great engineer straight out of college with one or two courses they may have touched on the subject, when working on a complex distributed system. There can be downsides to experience also – people can tend to frame everything in terms of their prior experience, and fail to consider or value ‘outside the box’ thinking, which is often best done by people without experience, they aren’t familiar with the ‘established methods’.
The primary benefit of experience is that it allows people to fail faster. They can discard ideas that they know won’t work, without having to implement them and see the problems. They can see the scalability pitfalls of a particular design, because they’ve seen previous designs fail, and know they need to consider that aspect. They can make appropriate tradeoffs between implementation time and product performance, as they’ve done and seen multiple situations in the past where less than optimal choices were made. So of course a great engineer with tons of experience has more value than a great engineer with little experience, but at the end of the day, they’re still both great engineers, and will do great work. In building a team and organization and product and company that will do amazing things and last for years and years, I’ll take a great engineer with little experience over an average engineer with tons of experience, any day of the week.
I founded and ran a team that built what was then, and still is now, one of the largest distributed data storage and analysis systems on the planet, which is being constantly extended without ever having had to revisit or materially change any of the foundational principles or components. There was precious little previous experience on the team in complex distributed systems, and most of the team was mid-level to senior, with only a few super senior members. They did a fabulous job, building a better system, faster, than anything else that existed at the time. How? Because it was an amazing team of wicked smart engineers that met all of the criteria 1-6 above, and though there was little domain expertise in distributed systems, everyone was very solid overall systems programmers, and we were willing and able to reconsider the problems from first principles – what were the requirements of the system we needed to build, what were the priorities of that system, etc. I strongly feel that not having worked on other distributed storage and analysis systems, and not doing extensive analysis of research and competitive or alternate systems and solutions, was a significant benefit in coming up with a system that met all of our criteria, and avoided a lot of the pitfalls and problems of existing systems. And of course, there was no existing system available that met all of our criteria, or we wouldn’t have had to build it in the first place. Thus, rather than trying to ‘fit’ or ‘modify’ existing systems or methods, we started from first principles, and created a plan to build something from scratch that met all of the requirements, and was extensible to meet what we expect will be future requirements. And then we executed on that plan.
Team strength – the value of diversity
I didn’t list this above as an attribute of a particular engineer, as it’s of course an aspect of an overall organization. I’ve travelled literally around the world to hire great people, and I’ve hired and worked with literally dozens of nationalities. I’ve worked multiple times for women managers, and for managers of different races. One of the best engineers I ever worked with graduated with a music degree, and I’ve worked with great engineers that never graduated college.
To build a fabulous product, there are thousands of decisions to be made and optimized, thousands of conversations about the best way to do something, of what a future customer might want to do, or the easiest way for them to do it. Diversity gives us different perspectives, and different ways of thinking. Engineers are often role-playing – what would I want if I was the customer, what if I was the IT administrator, what if I was the manager of a team within the customer’s organization using the product. What if I was a novice user? What if I found it easier to use and learn visual tools?
I spent a couple of years leading an effort to take a product and service with millions of users from the U.S., literally around the world – which data centers would we deploy in, what tax and regulatory issues would we have, how do we ensure as great an experience in China or Japan or India as we have in the U.S., when the majority of the engineering team isn’t able to read those languages or speak to many of those users? What different expectations might they have about what a web page or software product should look and act like? Diversity – open your mind to differences, and don’t assume that your experience or expectations are representative. And of course, surround yourself with people who will ensure that varied opinions and voices are heard and integrated into the best possible product, and best possible company culture.
Diversity also means having an interdisciplinary team – the product team is not just classic software engineers. I’ve had amazing discussions and insights from our staff sociologist. Sure he’s also an expert data scientist, and helps customers best use our product to solve their problems and discover new insights they can get from their data, and ensures that our product is capable of delivering everything the customers might require, but I’ve never worked with someone with such deep insights into human behavior, and both our team and product are clearly better for it. Plus, the lunchtime conversations on our team are always incredibly interesting and entertaining!
Jim Walsh Blog 