Reinventing the SDE Interview: An Hour-by-Hour Breakdown

In the first three parts of this series, we talked about how to approach the interview process as a candidate, how to study, and then how to stay within the current conventions of how an interview is conducted while still getting the clearest hire/no-hire signal.

In the fourth part of this series, I’m going out on a limb. It’s a pretty thin limb, and it cracks a bit under my own weight, but I’m going out on it anyway. I want to talk about what my concept of an ideal SDE interview is, from the perspective of a hiring manager.

This article is hyper-focused on software developers, but I hope you will be able to easily see how to adapt it for your own needs. Keep in mind, too, that this article is longer, and more in-depth than the earlier ones were. It’s intended for those well versed in the interview process who look to improve their hiring pipeline.

Note that here I’m focusing on an on-site interview. Hopefully, it should be obvious how to translate this to a virtual setting, but please hit me up with questions if it isn’t.

Mindset

An interview should try to replicate a real work environment as much as possible. After all, the whole point of an interview is getting an idea of the capabilities you can expect from your potential hire. Those capabilities are made up of (in priority order):

1. Team Fit

• How well does the candidate fit within the culture of the team?
• Is this someone the team would welcome? (Modulo unconscious bias – that needs to be considered and mitigated.)

2. Engagement

• Is the candidate passionate about their field, or are they only in it for the money?

3. Communication

• How clearly does the candidate communicate their thoughts, in both verbal and written form?
• How well are they able to understand the way your team communicates? Do they understand concepts explained to them? Are they able to pick up on others’ points of view?

4. Dealing with ambiguity

• How well does the candidate handle poorly-defined, ambiguous problems?

5. Intellectual horsepower

• How well does the candidate solve clearly-defined but complex problems?

So these are the questions we want to structure our ideal interview around. So now let’s break down the 8-hour interview loop into individual hours, starting at the standard 9am first-hour.

9am-10am

Participants:

• Hiring manager, key members of the team

Goals:

• Putting the candidate at ease
• Answering their questions
• Setting expectations about the rest of the day

Format:

This hour is focused completely on the candidate’s needs. The aim here isn’t to learn about any specific aspects or traits they may have, but to start getting to know them as a person, and give them a chance to get to know you and the folks they’ll be spending the rest of the day with. Naturally, the sort of questions they ask and how they engage with the interviewing group is a signal in itself. This is also an opportunity to soft-sell the team, though really, it’s the rest of the interview, and the interactions the candidate will have with your interviewers, that will do that.

10am – 3pm (1 hour lunch break)

Participants:

• Key members of the team

Goals:

• Simulate an actual work environment to the maximum possible extent.
• Evaluate the candidate’s Team Fit, Communication skills, Dealing with Ambiguity, Intellectual Horsepower

Format:

The candidate is given a desk and a computer next to the rest of the team. They are then given a project. The nature of the project will depend on the role they’re interviewing for but must be large in scope, have real-world significance, while remaining ambiguous as stated. Here are some examples:

• Design a License Plate Management system for the Department of Licensing in California.
• Design and implement a CMS for your local school
• Design and implement a recommendation system for Netflix

The key thing to understand and communicate to the candidate is they’re not actually expected to complete the task. Just as with any other interview question, tasks like this must be calibrated through multiple interviews with a wide swatch of candidates, so you get an idea of what an excellent performance looks like, and what a poor one does.

The next key thing to understand and communicate is that the candidate is allowed to both engage and collaborate with any member of your team. This is something super-important to mention in the first hour, and, as candidates won’t be used to this sort of an interview format, you’ll have a bit of a job convincing them that there’d be no points docked for this choice. If they want to spend the four hours in a pair design/programming session with a team member, great, they can do so. Or if they want to go at it alone, they can do that too. Your goal is to see them function in a setting as closely resembling your day-to-day team environment as possible.

Okay, okay, I can feel the mob forming. Questions? You have questions. Let’s hear them.

Q: Do you mean every time I do an interview loop I must disrupt my whole team’s day??

A: This is a common concern I hear about this method, but in practice it’s unfounded. It is highly unlikely the candidate will engage anyone but the key members of the team they met in their first hour onsite. This is why that first hour is so important.

And, yes, there is a real possibility that one or two of them will be randomized or even completely pulled away from their work for those 4 hours. So what? In a standard interview loop we randomize 3-5 members of the team for an hour of the interview plus another half hour to write the interview feedback. If you combine that with interview prep time and context shifting costs, their day is largely shot anyway.

Q: How do I evaluate the candidate’s performance?

A: Glad you asked! Before I answer this question, let’s agree on one thing:

When you collaborate closely with someone, you get a good idea of their capability. When collaboratively working on system design or code, you gain insight into each other's thought patterns, logic, and intellectual capacity. You also get a feel for how easy they are to work with, how quickly they pick up on someone else’s ideas, how well they can make themselves understood, etc.

To put it in simpler terms, a capable engineer recognizes capability or lack thereof in others primarily through collaborating with them or examining their output.

This is the environment we’re trying to create. Assuming you’ve put your more capable team members on the interview team, trust them to determine capability in your candidate.

The key is to make sure the interviewers are aware of their own unconscious biases and are taking them into account when evaluating. The hiring manager must ensure that the interviewing team values diversity. Being truly bias-proof can be a challenge here, though it’s undeniably a challenge in any interview format.

Q: What if they spend the whole time pair-programming with someone on the team? How do I know they can contribute something useful on their own?

A: Two-fold answer here. First, if pair-programming isn’t a practice on your team, don’t offer it as an option during the interview. Remember, the goal is to replicate the actual work environment on your team. So, give the candidate access to the processes your team uses, and withhold those it does not.

That said, if you’ve ever done true pair programming you’ll know that it’s one of the most high-bandwidth collaborative activities possible in the software engineering field. In other words, whether you function as the driver or the navigator, the impact of the person next to you is felt very deeply on a minute-by-minute basis. In the chance case that the candidate does spend the whole time pair-programming, I would be very comfortable with their pair’s hiring recommendation.

Q: How do you ensure the candidate remains comfortable and at ease during the interview process?

A: The key is to make this a collaborative experience. Treat the candidate as you would a newly-joined team member. Offer help. Engage in a discussion. Brainstorm ideas. Since this interview time will span lunch, use this as an opportunity to check in with the candidate’s experience. Ask what’s working, what could work better. Then adjust for the second half of this period. Don’t expect that the candidate will be completely at ease – this is an interview, after all, but if they’re someone excited by their work, as the hours wear on you should start seeing them relaxing into the problem.

Think about it this way: what would you, as the manager, do if your direct appeared stressed? You’d try to figure out why and help, right? Since we’re trying to simulate the real work environment on your team, why not show the candidate how good a manager you can be? Remember, the interview is a two-way process. Instead of telling the candidate about your team’s awesome culture and your capabilities as a manager, show them!

Q: How would you evaluate the candidate’s ability to deal with ambiguity via this process?

A: Remember, the question initially asked is purposefully ambiguous. What you’re looking for is for the candidate to ask clarifying questions, just as they should in a standard interview. But here, because you have more time, the question can be even more ambiguous, and the candidate has the chance to dive deeper before working on solutions. This gives you more of an opportunity to evaluate their critical thinking skills.

Q: How do you ensure that the project given to the candidate during this time period is appropriately scoped and ambiguous?

A: Just as with any interview question, you must make sure to calibrate the project in advance of the interview. The easiest way to do this is have one of your existing team members take it on, but really, expect it to take anywhere from 10-15 tries before you have a good feel for scoping and evaluating your question.

Specifically, if you find multiple candidates spend the first hour of these four floundering and your team members handhold them through the first part of the project, that could be a sign that the project is too ambiguous to make meaningful forward progress. Scope it down a little.

If, on the other hand, your candidates are finishing early, or, really, finishing at all, that’s probably a sign that the project is too scoped. Ideally, the project should be big enough that it cannot be completed by 95% of your candidates, and the candidate should be told so explicitly in hour 1. Again, remember, the point isn’t to complete the project. The point is to get a feel for how the candidate functions on your team to the best extent possible. Make sure the candidate understands this before they start, or they’ll stress needlessly about how much they may or may not have getting done. This stress will then get in the way of their actual performance, muddling your hiring signal.

3-4pm

Participants:

• At least one non-technical team stakeholder. (e.g. PM, Marketing, BizDev, etc).

Goals:

• Get a non-tech stakeholder perspective on the candidate. Get a feel for their prior work.

Format:

This is the standard behavioral question part of the loop. The candidate should be asked to tell 2-3 stories exemplifying areas of concern identified during the project.

For example, if the candidate appears conflict-averse, a question could be asked like “Tell me about a time you disagreed with a colleague and how you went about getting consensus.”

If the candidate struggled with a part of the project but didn’t ask for help promptly, you could ask “Tell me about a time when you struggled to deliver on a project and how you went about getting help.”

There’s a huge interview bank of behavioral-based interview questions on the net. Pick ones that will round out your understanding of the candidate’s capabilities.

Naturally, this will require coordination between the interview team during the interview and I know some companies prohibit that in the spirit of interrupting bias. Personally, I’ve always thought it a mistake. Good interviewers know they should draw their own conclusions. Bad interviewers shouldn’t be interviewing.

4-5pm

Participants: Hiring Manager

Goal: Get the candidate’s perspective on the day.

This is just an informal, unwinding-style discussion, but it’s extremely important. Ostensibly, this is a chance for the candidate to talk about the day: what went well, what they wish could’ve gone differently. For the hiring manager this is also an excellent opportunity to evaluate how self-aware the candidate is. Have they noticed their own mistakes during the loop? Have they thought about what they might do differently next time?

Mistakes are expected. They’re inevitable. In fact, if the candidate truly made no mistakes during the project, the project hadn’t been complex enough. What’s key is learning from these mistakes. This is a chance for the candidate to show how they do this.

Debrief

At the end of the interview loop, each member of the hiring team should write up their impressions of the day. The hiring manager should also capture the candidate’s impressions as related in the last hour of the interview.

The next day the team should get together for 30-60 minutes to discuss the interview. How well do the interviewers’ impressions correlate with those of the candidate? What challenges did the candidate encounter during the day and how did they approach those challenges?

The bottom line is, is this someone the interview team sees as adding value to the team? If so, that’s the hire signal.

Summary

The key goal we’re trying to achieve in this format is gauging the candidate’s performance in an environment as similar as possible to your actual team dynamics. To do this, we’re approximating the type of work the candidate will be doing, and access to the people they’ll be working with to the maximum extent we can.

They say traditional interviewing is a skill separate from those needed for successful on-the-job performance. Let’s break this pattern while improving the candidate experience in the process!

Call to Action

Look, I know this is a very non-standard way to interview, and, as with anything non-standard, it feels scary. I would love to have a discussion with you all on ways to improve this format, or what other concerns you might see with it. Please drop your thoughts in comments, and let’s talk!