.png)
Four interviews gets you to 86% accuracy on a hiring decision. Every interview after that adds about one percent. Todd Carlisle ran the study at Google that produced that number, and of roughly 5,000 interviewers in the data, not one could beat the average of the group.
Who is on this episode
Todd Carlisle is Chief People Officer at Bessemer Venture Partners, one of the oldest venture firms in the world. He spent a decade at Google running some of the most famous people analytics experiments in corporate history, including Project Cerebro, and went on to HR leadership roles at Twitter and Bay Area startups. He holds a PhD in organisational psychology, and he has worked in organisations from a hundred people up to 70,000.
What they covered
Project Cerebro started as a scaling problem. Google was sometimes hiring a hundred people a week while still assembling enormous packets on every candidate, and recruiters were going back to candidates for one more data point, one more interview. One person was interviewed twenty times before an offer. Todd tested around 150 candidate variables against 25 to 30 measures of actual performance. Most of them predicted nothing, which matches the published literature: even structured interviews correlate around 0.3 with performance, which accounts for roughly 9% of it. Interviews do work, but only under specific conditions.
The follow up study, run with a Stanford professor, looked at every interview the company had ever done and found the curve flattens hard at four. Todd also checked whether tenure or individual skill beat the crowd, correcting for harsh and lenient raters, and it did not. Tenured interviewers got slightly better and still lost to the mean.
"Of the I think it was 5,000 interviewers, no one could beat the mean of that 86%."
Todd Carlisle
The rest is operator territory. Google eventually published some of the research and also wrote a list of banned interview questions, with most of the famous brainteasers on it, including how many golf balls fit in a 747. Todd's four assessment categories survived every attempt to improve on them: general cognitive ability, role related knowledge, culture fit and leadership. At small scale, culture fit carries more weight, because one person in fifty moves the whole organisation.
"If you are less than fifty people, each person you're adding is like creating the DNA of the organization like one person at a time."
Todd Carlisle
He and Imran also get into when to hire your first people leader, which Todd splits into two phases, the metric he thinks is actively misleading, and what he is doing now at Bessemer: collapsing 26 separate HR tools, from hiring through performance management to learning, down towards one, and working out how much of it AI can absorb while keeping a high touch concierge team for the conversations people still want with a human.
Key insights
- Four interviews gets you to 86% accuracy. Each one after that buys about one percent, so interview five onwards is mostly cost: your time, the recruiter's time and a candidate experience that loses you the hire.
- Trust the crowd over the expert. None of the roughly 5,000 interviewers in the data beat the group mean, so a diverse panel asking the right questions beats your best interviewer.
- Measure learning agility, not just ability. Todd looks for people who moved between different jobs inside one company and succeeded at each, and the candidate who asks the CEO what the hardest problem is and pivots to it.
- Hire the first people leader as a high agility generalist who can run HRIS, payroll and recruiting and is happy below 200 people. Expect to add a second, different leader around 200 to 250.
- NPS is the people metric Todd says drives him crazy. It is fine for judging an event, dangerous when a whole engagement survey gets narrowed down to one question.
- The 70-20-10 finding from the Center for Creative Leadership flipped how he thinks about learning and development: 70% of what Fortune 500 leaders credited for making them leaders came from on the job experience, 20% from the network they built doing it, 10% from courses.
Watch the full conversation on YouTube, or listen to the audio version on Apple Podcasts.
