Bayle Shanks's website: notes-business-startups-startups-stHiring

hiring

interviewing software engineers

from a company that has studied interviewing: " 1. Decide what skills you're looking for

...ask yourself the following questions (these are questions we ask when we onboard a new company at Triplebyte).

    Do you need fast, iterative programmers, or careful rigorous programmers?
    Do you want someone motivated by solving technical problems, or building product?
    Do you need skill with a particular technology, or can a smart programmer learn it on the job?
    Is academic CS / math / algorithm ability important or irrelevant?
    Is understanding concurrency / the C memory model / HTTP important?

There are no right answers to these questions. We work with successful companies that come down on both sides of each one. But what is key is making an intentional choice, based on your needs. ... 2. Ask questions as close as possible to real work ... caveat...an interview question (((should))) be free from external dependencies. For example, asking a candidate to write a simple web scraper in Ruby might seem like a good real-word problem. However, if a candidate needs to install Nokogiri (a Ruby parsing library that can be a pain to install) and they end up burning 30 minutes wrestling with the native extensions, this becomes a horrible interview.

3. Ask multi-part questions that can't be given away

Another good rule of thumb for interview questions is to avoid questions that can be “given away”, i.e. avoid questions where there's some magic piece of information that the candidate could have read on Glassdoor ahead of time that would allow them to answer easily. This obviously rules out brain teasers or any question requiring a leap of insight. But it goes beyond that, and means that questions need to be a series of steps that build on each other, not a single central problem. Another useful way to think about this is to ask your self whether you can help a candidate who gets stuck, and still end the interview with a positive impression....To give examples, asking a candidate to implement the game Connect Four in a terminal (a series of multiple steps) is probably a better question than asking a candidate to rotate a matrix (a single step, with some easy giveaways). And implementing k-means clustering (multiple operations that build on each other) is probably better than determining the largest retangle that can fit under a histogram.

4. Avoid hard questions

If a candidate solves a really hard question well, that tells you a lot about their skill. However, because the question is hard, most candidates will fail to solve it well. The expected amount of information gained from a question, then, is heavily impacted by the difficulty of the question. We find that the optimal difficulty level is significantly easier than most interviewers guess. ... The rule of thumb we now follow is that interviewers should be able to solve a problem in 25% of the time they expect candidates to spend. So, if I'm developing a new question for a 1-hour interview, I want my co-workers (with no warning) to be able to answer the question in 15 minutes. Paired with the fact that we use multi-part real-world problems, this means that the optimal interview question is really pretty straightforward and easy. ... To be clear, I am not arguing for lowering the bar in terms of pass rate. I am arguing to ask easy questions, and then including in your evaluation how easily the candidate answered the questions. I'm arguing for asking easy questions, but then judging fairly harshly. This is what we find optimizes signal. ... To give examples, asking a candidate to create a simple command line interface with commands to store and retrieve key-value pairs (and adding functionality if they do well) is probably a better problem than asking a candidate to implement a parser for arithmetic expressions. And a question involving the most common data structures (lists, hashes, maybe trees) is probably better than a question about skiplists, treaps or other more obscure data structures.

5. Ask every candidate the same questions

Unfortunately, the only answer here is for the interviewers to put in the effort. Consistency is key to running good interviews, and that means asking every candidates the same questions, and standardizing delivery. There's simply no alternative.

6. Consider running multiple tracks

In conflict with my previous point, consider offering several completely different versions of your interview. The first step when designing an interview is to think about what skills matter. However, some of the answers might be in conflict! It's pretty normal, for example, to want some really mathy engineers, and some very productive / iterative engineers (maybe even for the same role). In this case, consider offering multiple versions of the interview. They key point is that you need to be at enough scale that you can fully standardize each of the tracks. This is what we do at Triplebyte. What we've found is that you can simply ask each candidate which type of interview they'd prefer.

7. Don't let yourself be biased by credentials

Credentials are not meaningless. Engineers who have graduated from MIT or Stanford, or worked at Google and Apple really are better, as a group, than engineers who did not. The problem is that the vast majority of engineers (myself included) have done neither of these things. So if a company relies on these signals too heavily, they will miss the majority of skilled applicants. Giving credentials some weight in a screening step is not totally irrational....Letting credentials sway final interview decision, however, does not make sense... The most obvious way is just to strip school and company names from resumes before giving them to your interviewers. Some candidates may mention their school or company, but we do all our interviews without knowing the candidates' backgrounds, and it's actually pretty rare for a candidate to bring it up during technical evaluation.

8. Avoid hazing

One of the ugliest ways interview can fail is that they can take on an aspect of hazing. They're not just about evaluating the skill of a candidate, they're also about a group or team admitting a member. In that second capacity, they can become a rite of passage. Yes, the interview is stressful and horrible, but we all did it so so should the candidates. ... As an interviewer, it can be frustrating to watch a candidate beat their head against a problem, when the answer seems so obvious! You can get short tempered and frustrated...This is something you want to stay a mile away from. The solution is talking about the issue and training the interviewers. One trick that we use is, when a candidate is doing really poorly, to switch from evaluation mode, where the goal is to judge the candidate, to teaching mode, where the goal is to make the candidate understand the answer to the question. Mentally making the switch can help a lot. When you're in teaching mode, there no reason to withhold information or be anything other than friendly.

9. Make decisions based on max skill, not average or min skill

So far, I've only talked about individual questions, not the final interview decision. My advice here is to try to base the decision on the maximum level of skill that the candidate shows (across the skill areas you care about), not the average level or minimum level.

This is likely what you are already doing, intentionally or not! The way hire/no hire decisions are made is that everyone who interviewed a candidate gets together in a meeting, and an offer is made if at least one person is strongly in favor of hiring, and no one is strongly against. To get one interviewer to be strongly in favor, what a candidate needs to do is ace one section of the interview. Across our data, max skill is the attribute that's most correlated with acing at least one section of a company's interview. However, to be made an offer, a candidate also needs no one to be a strong no against them. Strong noes come when a candidate looks really stupid on a question.

Here we find just a great deal of noise. There are so many different ways to be a skilled engineer, that almost no candidates can master them all. This means if you ask the right (or wrong) question, any engineer can look stupid. ...The best solution, I think, is for companies to focus on max skill, and be a little more comfortable making offers to people who looked bad on parts of the interview. This is, looking for strong reasons to say yes, and not worrying so much about technical areas where the candidate was weak. I don't want to be absolute about this. There are of course technical areas that just matter to a company. And deciding that you want to have a culture where everyone on the team is at a certain level in a certain area may well make sense. But focusing more on max skill does reduce interview noise.

Why do interviews at all?

A final question I should answer is why do interviews at all? I'm sure some readers have been gritting their teeth, and saying “why think so much about a broken system? Just use take-home projects! Or just use trial employment!” After all, some very successful companies use trial employment (where a candidate joins the team for a week), or totally replace in-person interviews with take-home projects. Trial employment makes a lot of sense. Spending a week working beside an engineer (or seeing how they complete a substantial project) almost certainly provides a better measure of their abilities than watching them solve interview problems for 1 hour. However, there are two problems that keep trial employment from replacing standard interviews:

    Trial employment is expensive for the company. No company can spend a full week with every person who applies. To decide who makes it to the trial, companies must use some other interview process.
    Trial employment (and large take-home projects) are expensive for the candidate. Even when they are paid, not all candidates have the time. An engineer working a full-time job, for example, may simply not be able to take the time off. And even if they can, many won't. If an engineer already has job offers in hand, they are less likely be willing to take on the uncertainty of a work trial. We see this clearly among Triplebyte candidates. Many of the best candidates (with other offers in hand) will simply not do large projects or work trials.

The result of this that trial employment is an excellent option to offer some candidates. I think if you have the scale to support multiple tracks, adding a trial employment track is a great idea. However, it's not viable as a total replacement for interviews.

Talking to candidates about past experience is also sometimes put forward as a replacement for technical interviews. To see if a candidate can do good work in the future, the logic goes, just see what they've done in the past. We've tested this at Triplebyte, and unfortunately we've not had great results. Communication ability (ability to sell yourself) ended up being a stronger signal than technical ability. It's just too common to find well-spoken people who exaggerate their role (take credit for a team's work), and modest people who downplay what they did. Given enough time and enough questioning, it should be possible to get to the bottom of this. However, we found that within the time limits of a regular interview, talking about past experience is not a general replacement for interviewing. It is a great way to break the ice with a candidate and get a sense of their interests (and judge communication ability and perhaps culture fit). But it's not a viable total replacement for interviews

...

Programmers also choose interviews. While this is a very controversial topic (there are certainly programmers who feel differently), when we've run experiments offering different types of evaluation, we find that most programmer still pick a regular interview. And we find that only a minority of programmers are interested in companies that use trial employment or take-home projects. For better or worse, programming interviews seem to be here to say.

" -- https://triplebyte.com/blog/how-to-interview-engineers

" pklausler 87 days ago [-]

The sad reality of programming interviews is that it's absolutely necessary to ask several near-trivial questions in order to flush out the candidates with awesome resumes and impressive degrees who simply have no idea how to analyze a simple problem and solve it using a computer.

Lately, I've been asking "given the starting and ending times of two calendar appointments, determine whether or not they conflict." No loops, no fancy algorithms, no tricks... and asking it has not been a waste of time. I get people writing doubly-nested loops over all of the seconds in the two intervals, comparing for equality; I get multi-line predicates full of redundancy that still yield false positives and negatives; it's depressing as hell. "

 rjkennedy98 1 day ago [-]

I don't think this is as rare as you might imagine. When I started doing engineering interviews at our last company I was startled about how obsessive they were about doing video interviews because I hadn't ever remember doing them in previous jobs. If video resolution was bad they would simply cut the interview and ask to reschedule.

When I asked why they did this, to my surprise they said they had been scammed by people who did phone interviews well, and in the on site interviews were completely incapable of answering the same questions at all. They we sure they had been catfished by multiple recruits.

binarytox1n 1 day ago [-]

This has happened to us many times and we now do the same for remote interviews. No video? No interview. And even then, if you do great in the video we fly you in for an in person interview just to be sure. There's clearly a market for stand-in phone interviewers. I can't imagine that jobs acquired this way ever last, but that doesn't stop people from trying.

---

Paul Graham seems to suggest that if a startup gives stock in lieu of salary, it should take the amount of salary+overhad that it is saving per year, then give 1.5x that much stock (at current valuation) [1]

---

https://web.mit.edu/e-club/hadzima/how-much-does-an-employee-cost.html

---

https://wasp-lang.dev/blog/2022/08/15/how-to-communicate-why-your-startup-is-worth-joining

---