Lenny's Newsletter · Product & Work
TIER 5 2022-03-22
> ## Q: How should I structure my PM interview process? I can understand why so many people are befuddled by interviewing product managers—the role is so endlessly undefined, how can non-PMs possibly know what to interview for? As [Gergely Orosz](https://twitter.com/GergelyOrosz/) (ex-Uber, Microsoft, Skype engineer) confessed in [a tweet](https://twitter.com/GergelyOrosz/status/1443275923843862536), “we EMs have no clue how to do these. At first, you wing it; by the third time, everyone has their own, completely different setup.” In my [chat with Harry Stebbings last week](https://www.thetwentyminutevc.com/lenny-rachitsky/), I shared a bit of advice on this, but it was quite brief. So let’s get much deeper. Below, I’ll share suggestions for: 1. Structuring your PM interview process 2. What specific skills to test for 3. My favorite interview questions 4. How to structure an interview project (plus a bunch of real-life examples) 5. Tips to get them to take your offer *Disclaimer: People have strong opinions about how to interview, but very few people have enough data (or have looked at their data) to know if their interview assumptions are actually predictive of performance. Interview best practices are generally anecdotal, so definitely do your own research.* With that, let’s get into it. ### How to structure your PM interview process High-level, I’d set up your interview process like so: 1. **Recruiter phone screen [30 minutes, optional]:** Make sure the candidate meets the minimum requirements for the role and is likely to be a fit for the organization. Focus on their expectations for the role, basic skill questions, personality fit (e.g. [not an asshole](https://www.amazon.com/Asshole-Rule-Civilized-Workplace-Surviving-ebook/dp/B000OT8GV2/ref=tmm_kin_swatch_0?_encoding=UTF8&qid=&sr=)), and their interview timelines. About 50% of candidates should make it through this step. 2. **Hiring manager phone screen [30 minutes]:** Make sure there is a strong chance that the candidate is a fit for the role. Focus on getting to know the person as a human, the role, and a couple skills or attributes that you believe to be most important to this role (more on this below). About 40% to 50% of candidates should make it through this step. 3. **A full-day interview [4-5 hours]:** The day is made up of two parts: 1. **A project** (done at home before arriving, in the office, or live), sharing what they came up with, with a group of potential colleagues 2. **Three to six 1:1 interviews** with potential colleagues, each testing for a key PM skill 4. **Post-interview panel discussion [30-60 minutes]:** Each interviewer blind votes Strong No, No, Yes, or Strong Yes. Share your vote, discuss, and then the hiring manager makes a decision. 5. **Reference checks:** Ping their references (and backchannel) to make sure you aren’t missing anything. 6. **Make an offer:** Put together a compelling offer, share it with them,and try like hell to get them to accept. To benchmark a typical hiring funnel, I asked [Benji Encz](https://www.linkedin.com/in/benjaminencz/), CEO of the recruiting platform [Ashby](https://ashbyhq.com/), to share stats he had on conversion rates at each step, and wow, did he deliver. Here are hot-off-the-presses stats his team pulled for me, based on over 75,000 recent applications across high-growth tech companies. Takeaway: You need to talk to an average of 23 PM candidates to hire one great one.  > #### “Many early-stage teams underestimate how many candidates they need to speak to to make one hire. In our data we also saw that only 13% of inbound applicants even make it to the first screen. You will either need a lot of inbound, or more likely, you will have to fill the funnel with sourced candidates and referrals.” > > #### —[Benji Encz](https://www.linkedin.com/in/benjaminencz/), CEO of [Ashby](https://ashbyhq.com/) I would be remiss if I didn’t plug my new [Talent Collective](https://www.lennysjobs.com/talent/) here, which is quickly becoming the single best place to find and hire product managers—and help you find your next dream gig. [Learn more here](https://twitter.com/lennysan/status/1503437573263282178). ### What skills to interview for There are [10 core skills of product management](https://www.lennysnewsletter.com/p/jobs-of-product-manager?s=w), but assuming you’re interviewing an IC PM (i.e. not a manager), you can whittle it down to these six: 1. Communication 2. Collaboration 3. Execution 4. Strategy 5. Impact 6. Product sense  Interestingly, product sense isn’t ever on PM career ladders, but [it is something that companies interview for](https://www.lennysnewsletter.com/p/what-is-product-management?s=w#:~:text=The%20most%20important%20PM%20skills%20when%20interviewing), so I’ve included it in the list above.  ### What interview questions to ask I wrote a whole post sharing [my favorite PM interview questions](https://www.lennysnewsletter.com/p/my-favorite-pm-interview-questions?s=w), so definitely go read that, but here’s a sampling: - **Collaboration:** Tell me about a time you disagreed with an engineer on your team and how you resolved it. - **Execution:** Pick a project you’re proud of that took 3-9 months. Walk me through it from beginning to end. I’ll ask questions along the way. [Give this ~7-10 minutes] - **Strategy:** Pick a product you worked on in the past year—talk me through your strategy for it. - **Customer insights:** Tell me about a time you did user research on a product/feature and that research had a big impact on the product. - **Impact:** What’s the most important or impactful product you shipped? What made it so important or impactful? Would it have been as impactful without you, and why? - **Product sense:** How would you improve feature x in our product? [This full post](https://www.lennysnewsletter.com/p/my-favorite-pm-interview-questions?s=w) includes what to look for in the answers, red flags, and a ton more great interview questions to inspire you. Seriously, go check it out. And don’t forget to keep your questions consistent, with predefined good and bad answers, to [avoid bias](https://hbr.org/2016/04/how-to-take-the-bias-out-of-interviews). > #### “Companies should rely on a structured interview that standardizes the process among candidates, eliminating much subjectivity. These interviews pose the same set of questions in the same order to all candidates, allowing clearer comparisons between them.” > > #### —Iris Bohnet, *Harvard Business Review* ### The project I’ve found the project portion of the PM interview process to be the most informative component of the interview. It’s the only time you’ll get to see the candidate tackle a chunky new problem, be able to watch them in action, and have a discussion about it. Though some people don’t include it, I couldn’t imagine a PM interview without it. > #### “Years later I can see that the performance on the project was closely related to their success at Slack over the course of years.” > > #### —[Merci Grace](https://twitter.com/merci), former Head of Growth at Slack The goal of the project is to get a taste of how a candidate approaches a new problem. Most importantly, you aren’t looking at how close they got to the right answer. Instead, you are looking at how they break down the problem, how they structure their solution, and how they communicate it to you. Often the discussion afterward is even more valuable than the prompt itself. You have two options for delivering the project: 1. **Before the interview:** Give them a project before the full-day interview, to work on at home in some timeboxed amount of time. They then share what they came up with when they come in (or Zoom in) for the full-day interview. 2. **The day of the interview:** Give them a project at the beginning of the interview day, with a timeboxed amount of time for them to work on it (e.g. 90 minutes). Once they are done, they share what they’ve come up with. I did [an informal survey on Twitter](https://twitter.com/lennysan/status/1499452039541112833), and the majority (maybe two-thirds) of hiring managers prefer Option 1—to give the candidate the project ahead of time. I personally prefer Option 2, giving the project the day of the interview. Instead of telling you exactly what to do, I’ll instead share the pros and cons of each, a few people’s perspectives, and let you decide for yourself. **Option 1: Project at home, before the interview** - **Pros:** More time to think, closer to real life, and gives you more interview time during the day - **Cons:** Unfair for people with less time outside of work, some people don’t like being asked to “work” on their own time, and more confounding variables > #### “At Slack, we gave projects to be done on their own time. I’ve heard the reasons people have not to do this, but it works. It worked for the IC product writing and UX research roles I hired and managed as well. I hired a diverse team that included parents, people with student loan debt, women, men, people of color, veterans.” —[Merci Grace](https://twitter.com/merci), former Head of Growth at Slack > #### “On their own time allows them to do research if they don’t have too much background on industry or space. Plus, preparation shows care and commitment.” —[Kosta Kolev](https://twitter.com/Kosta_kolev), founder > #### “For candidates who are more interested in the domain and the problem you are solving (the ‘missionaries’), the assignment can actually generate additional excitement about the opportunity. These candidates will likely find working on the problem so intrinsically motivating that it gives them energy rather than draining it.” —[Jens-Fabian Goetzmann](https://www.jefago.com/product-leadership/homework-for-product-manager-candidates/), Head of Product at RevenueCat **Option 2: Project during the interview day** - **Pros:** Equalizes interview conditions and doesn’t suck up candidates’ personal time - **Cons:** Candidate has less time to think and is under more pressure, and it gives you less time for other interviews > #### “We found take-homes have high friction vs. signal, and the best candidates have multiple offers.” —[David Cheng](https://twitter.com/DavidPCheng/status), Senior GPM at Chime > #### “During the day of the interview, to truly equalize amount of time spent (2 hours) by each candidate.” —[Ketki Duvvuru](https://twitter.com/k3tk1/), Product Lead at Superhuman > #### “I have done it in their own time in the past but have since moved to doing it during the interview. Many reasons why, and still today kinda on the fence/grapple with it, but mainly I choose to do it as I believe it’s more equitable.” —[Ant Murphy](https://twitter.com/ant_murphy), Product Coach If you’re unsure which way to go, try both and see which feels like a better signal of the candidates’ potential. Also, as a last resort, if you can’t decide, you can also give them the choice and accommodate both. #### What should the project be? You have three options: 1. A real problem you’ve faced 2. A hypothetical problem, related to your product 3. A hypothetical problem, unrelated to your product In the same [informal Twitter survey](https://twitter.com/lennysan/status/1499452039541112833), the majority of hiring managers go with Option 1 or 2—a hypothetical problem, related to, or based on, their actual product. > #### “Closely resemble what their day-to-day work would look like.” —[Marissa Goldberg](https://twitter.com/mar15sa), founder of Remote Work Prep > #### “I typically use a case that is in the domain but doesn’t require specific product knowledge, to prevent bias.” —[David Cheng](https://twitter.com/DavidPCheng/status), Senior GPM at Chime > #### “I want the person to learn about what we do and get excited about it. Interviews are two-way.” —[Fareed Mosavat](https://twitter.com/far33d), CDO at Reforge, ex-Slack, Instacart > #### “Our prompt is venture-specific, but we explain the venture pieces. We don’t expect people to have that domain knowledge, and we generally overlook ‘mistakes’ that are simply related to that. It’s a good litmus test, though, to see whether applicants do their own research on the space.” —[Jessica Toy](https://twitter.com/jtoyPro), Director of Product at AngelList Venture I personally went with Option 1, because I found that I was able to get the best signal, but I didn’t have the chance to experiment in-depth here. Be careful about asking candidates to do actual work for you (e.g. solving a problem you’re currently facing) without paying them for the work. But do feel free to pay them for their time, especially if the project is being done on their own time. ### Examples of real project prompts To make this even more concrete, I’ve collected a handful of real-life prompts from fellow PMs. As I come across more, I’ll add them to this post. Use this as inspiration for coming up with your own custom project. #### **[David Cheng](https://twitter.com/DavidPCheng), Senior GPM at Chime** This is a take-home assignment (when he used to do take-homes), about a hypothetical product, used to hire a manager for the identity product (owning KYC, authentication, and profiles): > *The take-home should be timeboxed to 3-4 hours. You will be presenting this as part of your on-site interview, so please choose a presentation format. You will have 45 minutes to present.* > > ***Net(Income)Flix*** > > *Imagine you’re the product lead for identity at a video streaming service called Net(Income)Flix. The CEO wants to maximize financial returns and has anecdotally heard that many of our customers share their accounts with their friends and family, leading to potential lost revenue.* > > *You have been tasked to come up with a plan to identify access to these rogue accounts and shut them down. Specifically, this plan must address:* > > - *How do you measure the scale of the problem/opportunity?* > - *What other product teams may be impacted by this? How could you address their concerns?* > - *What are some features/solutions that may address this issue (just a bullet point description, nothing in-depth)?* > - *How do you prioritize and sequence your plan?* > - *How do you think about user experience?* *Things I looked for:* - *Answered the questions I posed (surprisingly, some candidates didn’t do that!)* - *I tried to make sure the candidate knew to spend at least 1/4 of the time on user experience, because oftentimes that would get overlooked, particularly given the domain* - *Understood the domain around identity (e.g. 2FA, identity signals like device, location, etc.)* - *Had success metrics that tied back to the problem/opportunity* - *Created user personas and tied them to features/user experiences. They also tied features back to the success metrics they chose* - *[Bonus] Talked about getting user feedback through quant and qual* - *[Bonus] Pushed back and said some fraud is okay to optimize for user experience* - *[Bonus] Pushed back and said there was market opportunity for “stolen” accounts (e.g. different user profiles, different pricing schemes, referrals)* #### [Merci Grace](https://twitter.com/merci), former Head of Growth at Slack This is a take-home assignment about a real Slack product opportunity, used to hire all levels of PM: > ***Background*** > > *The first part of your interview will be an assignment for members of the product, engineering, and design teams. Our goal is to see how you approach product problems in depth, with real-life situations. Please prepare a presentation that covers the topic below.* > > *We ask that you try to complete this within 72 hours of receipt. If your schedule does not permit and you need more time, just let us know.* > > *When you’re done, please send us a copy of your presentation. We’re excited to see what you come up with!* > > ***Problem statement*** > > *Every day, human beings create new Slack teams. These people are often not officially tasked with finding new enterprise software in their day-to-day role, but they have a problem that they think Slack can solve.* > > *The most successful Slack team creators are able to get several people to create accounts on their team and have a meaningful conversation within the product in the first day or so. Inviting people to a new product is fairly straightforward—we can set that aside. What is much more difficult is getting 3 or more people to have a meaningful conversation in a new messaging service.* > > *Go through the process of creating a new Slack team. Assume that you are someone with a reasonable level of intent in creating this team: you have a problem that you think Slack can solve. Identify issues or missed opportunities with the current new-user experience that you think hold people back from having meaningful conversations in Slack. Generate ideas for experiments to run that you think would positively impact this metric. Prepare materials for a plan that ranks your ideas in priority order. Flesh out at least 3 of those ideas into experiments you think the Growth team should run.* > > ***What we want to see from you:*** > > 1. *What questions do you have?* > 2. *What assumptions are you making?* > 3. *What data would help make your case?* > 4. *How would you measure success?* > 5. *How would you execute quickly on this?* *Things I looked for:* - *Their ability to structure and communicate ideas clearly and succinctly in the overall presentation. Can this person write in a compelling way? Can they present well?* - *Are they curious? I got a lot from the questions they asked either while working on the project or within the presentation itself. Are they comfortable making a gut call when there’s little or conflicting data? When in doubt, do they prioritize creating a quality customer experience?* - *Their creativity within the constraints of the product. Did they come up with a solution that’s feasible, testable, and not too over-reaching? (We’d often see more junior product managers over-rely on a hand-wavy solution, like reaching toward Slackbot for a deus ex machina conversation with the user.)* *No joke, literally no former Facebook PM I interviewed would make a decision about something without data. They actually refused. I stopped interviewing people from FB. Your success is hugely dependent on your fit at the company, which is why it can feel so arbitrary.* #### [Kevin Yien](https://twitter.com/kevinyien), former product lead at Square, currently Head of Product & Design at Mutiny This is a take-home assignment about a fictional, but realistic, product used to assess mid-senior PMs. They were given 2-3 days to work on it, and were expected to spend 2-3 hours (though everyone always spent more). > *You are the product manager for Square Training, a new product that we will build to help merchants train their staff on Square products (starting with the point of sale).* > > *Please prepare a short presentation to describe what you would build, in what order, and why.* > > *Designs are not required but may help in articulating the solution you have in mind. Be prepared to speak for 30 minutes, with 15 minutes of discussion on top.* *Things I looked for:* - *Problem context grounded in actual customer conversations (i.e. they went out and talked to real customers)* - *Solution was tangible and defined enough to have actual conversation around, able to reframe questions around a framework rather than get bogged down in an actual response (which would be unrealistic to expect)* - *Thought through the before/during/after of a solution, not just the during* - *Considered design, technical, and GTM constraints (not just one)* - *Able to improve upon the idea in the meeting, rather than try to bring back to original solution* *I experimented with both slide and doc formats, with the live discussion being both oral presentation and a silent read of the doc (which always tripped people up unless they came from a heavy doc culture).* #### **Me, while at Airbnb** This is a day-of-interview assignment about a real product opportunity the team faced, used to hire IC PMs and managers, for the host side of the product. I’m sharing this from memory (didn’t have it written down). > *You have 90 minutes to work on this project. Once you’re done, you’ll share what you’ve come up with with a small group of potential colleagues (e.g. an engineer, a designer, a researcher, etc.). Don’t worry about having the perfect answer; just do your best.* > > ***Problem:** When Airbnb guests attempt to book a place, 50% of the time they are ignored or rejected by hosts. This leads to a bad user experience and a lack of trust in the platform, and the business misses out on revenue. How would you increase this conversion rate from 50% to 90%?* > > **Context** > > - *Nearly all host rejections are for logistical reasons—their calendar is not up to date, they can’t support pets, they don’t allow parties, etc.* > - *Hosts sometimes don’t know they recieved a guest request* > - *Hosts have 24 hours to respond to a guest request before it expires* *Things I looked for:* - *How did they break down the opportunity to increase conversion—what levers did they come up with?* - *Which lever/opportunity did they see as the biggest opportunity, and why?* - *Did they think big (i.e. working backward from the ideal) or incrementally (i.e. optimizing at the edges)?* - *How they prioritized—what did they optimize for?* - *How they laid out their proposal and communicated it—did it make sense?* - *How they balanced the needs of both sides of the marketplace* #### **[Ronnie Regev](https://twitter.com/RonnieRegev),** former product leader at **Procore, AppFolio** This is a take-home assignment to be done on their own time, with guidelines to try to keep it to a few hours. It took 6-9 months of iteration just to get the exercise dialed in. > *Pick a modern collaboration tool (e.g. Miro, Asana, Mural, Slack, etc.) and design a new feature* or *improvements to the acquisition, activation, or engagement steps.* > > *Constraints:* > > - *3 months of dev time with a team of 4 eng who know the code base* > - *Present the problem, the customer value, the solution, the success criteria, a roadmap, trade-offs, release strategy, and mid fidelity mock-up* > - *A one-pager supporting the presentation* > - *No bonus points for a super-slick presentation; focus on content over form* > - *You have 30 minutes max to cover all content, and 30 minutes of discussion* > > *This approach allowed me to assess one’s product sense and execution, and how they articulate ideas visually (mocks), verbally (presentation), and in prose (one-pager). It also worked well for designers, with different constraints.* **More projects inspiration:** ### How to convince them to take your offer Finally, after you’ve found the person you want to hire, the final step is convincing them to take your offer. I wrote [about winning at hiring](https://www.lennysnewsletter.com/p/early-stage-hiring?s=w), which has a bunch of great advice from founders who are world-class at this, so here’s some tactical advice from that post: 1. [Julianna Lamb](https://www.linkedin.com/in/juliannaelamb/) (CTO of [Stytch](https://stytch.com/)): “**We see the actual offer stage as a big place you can stand out.** We do a Zoom to surprise the candidate with everyone from their interview panel to share why they’re excited about the candidate potentially joining.” 2. [Siqi Chen](https://www.linkedin.com/in/siqic/) (CEO of [Runway](https://runway.com/)): “**We treat closing a candidate almost exactly the same as closing an investor.** In fact, we’ve found the best candidates will do even more diligence than the average investor. Arguably, candidates are investing something even more valuable than capital: their time. Our pitch and demo for our Series A is pretty much exactly the pitch and demo we do for candidates.” 3. [Sam Corcos](https://www.linkedin.com/in/samcorcos/) (CEO of [Levels](https://www.levelshealth.com/)): “**Be prepared to commit ~100 hours for each hire you make.** Interview a lot of people. Ask your network for introductions. Reach out to people on LinkedIn. Get creative on how you source candidates. Write excellent [job descriptions](https://levels.link/eng). There are no shortcuts.” ## 📚 Further study 1. [My favorite PM interview questions](https://www.lennysnewsletter.com/p/my-favorite-pm-interview-questions?s=w) 2. [Constructing an effective phone screen](https://kevinyien.com/blog/phone-screen.html) 3. [Winning at early-stage hiring](https://www.lennysnewsletter.com/p/early-stage-hiring?s=w) 4. [Product management career ladders](https://www.lennysnewsletter.com/p/jobs-of-product-manager?s=w) 5. [What is product management](https://www.lennysnewsletter.com/p/what-is-product-management?s=w) 6. [A Twitter thread with dozens of PM interview process examples](https://twitter.com/lennysan/status/1443264202831896578) 7. [Product Interviewing by Hadar Dor](https://www.hadardor.com/product-interviewing) 8. [Take-home assignments for product management candidates](https://www.jefago.com/product-leadership/homework-for-product-manager-candidates/) 9. [Bias in Interviews: Eliminate It Fast with These 6 Easy Strategies](https://codesubmit.io/blog/eliminate-bias-in-interviews/) 10. [10 Ways to Reduce Interviewer Bias](https://www.linkedin.com/business/talent/blog/talent-strategy/ways-to-reduce-interviewer-bias) *Have a fulfilling and productive week 🙏* ## **🔥 Featured job openings** 1. **Fractal Software:** [CEO/Founder](https://www.lennysjobs.com/jobs/bbb7f335-4841-48fc-aff8-702a83f26f35) (SF, NYC) 2. **Perfect Venue:** [Founding Team Full Stack Engineer](https://www.lennysjobs.com/jobs/461940f7-6f12-42af-a0d7-8fca71b2c693) (Remote-US) 3. **Snackpass:** [Senior Product Manager](https://lennys-jobs.pallet.com/jobs/3e5c5552-cb71-4b23-9773-95f21ab14dfc) (NYC, LA, SF) 4. **Silo Finance:** [Technical Product Manager](https://lennys-jobs.pallet.com/jobs/dceac63a-d817-47e7-a4a5-5f653f298bdd) (Remote-Global, Remote-EU, Remote-US) 5. **Donut:** [First Product Manager](https://lennys-jobs.pallet.com/jobs/6259d930-475f-4ed4-82a3-fea29273b40a) (Remote-US, Remote-EU) 6. **Mindbloom:** [Product Designer](https://lennys-jobs.pallet.com/jobs/0db1f9c0-266c-4d77-9852-df27b58f8779) (Remote-US, Remote-Canada) 7. **Finch Care Inc.:** [Product Designer](https://lennys-jobs.pallet.com/jobs/18311f14-d482-48a1-808e-3716a7f1aabe) (Remote) 8. **Mos:** [Product Manager](https://lennys-jobs.pallet.com/jobs/86e76563-7237-4a39-8465-3682823f7506) (Remote-US) 9. **Airhouse:** [Senior Product Manager](https://lennys-jobs.pallet.com/jobs/1739a7ee-41ab-43fb-8ac9-788738baad22)(SF, Oakland) 10. **CommerceHub:** [Director of Product Management, Delivery](https://lennys-jobs.pallet.com/jobs/5e4c79e3-9c75-4cd3-aa6f-0b9fc40636bf) (Remote-US) 11. **CommerceHub:** [Sr. Product Manager](https://lennys-jobs.pallet.com/jobs/125eafba-2e93-41ee-98aa-abaae84c1b35) (Remote-US) 12. **Elevate Labs:** [Senior Product Manager, Balance](https://lennys-jobs.pallet.com/jobs/54802bbd-3ad4-4abf-bd0f-308e86d52586) (Remote) 13. **Oath Care:**[UX Designer](https://lennys-jobs.pallet.com/jobs/0e35e470-6b81-4dae-a706-33139219fa68) (Remote-US) *Browse more open roles, or add your own, at [Lenny’s Job Board](https://lennysnewsletter.com/jobs).* ## **🧠 Inspiration for the week ahead** 1. **Listen:** [20 Product: Lenny Rachitsky and Harry Stebbings](https://www.thetwentyminutevc.com/lenny-rachitsky/?v=2) 1. **Read:** [Can’t cook, won’t cook? Here are the tips that saved me from a life of terrible meals](https://www.theguardian.com/food/2022/mar/02/cant-cook-wont-cook-here-are-the-tips-that-saved-me-from-a-life-of-terrible-meals) by Elle Hunt 2. **Watch:** *[The Rescue](https://www.disneyplus.com/movies/the-rescue/6u1VV7RFRiRz)*. From the Oscar-winning team behind *Free Solo*, it chronicles the daring rescue of 12 boys and their soccer coach from deep inside a flooded cave in Thailand. [Watch on YouTube](https://www.youtube.com/watch?v=raVceiFQSjw) **If you’re finding this newsletter valuable, consider sharing it with friends, or subscribing if you haven’t already.** Sincerely, Lenny 👋