Lenny's Newsletter · Product & Work
TIER 5 2024-07-02
You’re either building AI into your product now or you will be soon. And you’re probably already swimming in advice on the subject. But most of the advice out there is full of big, lofty ideas and light on tactics rooted in actual experience that you can implement today. So I teamed up with frequent collaborator [Kyle Poyar](https://www.linkedin.com/in/kyle-poyar/) to interview more than 20 successful builders and founders—people who have learned about building AI products the hard way—to share their biggest surprises and counterintuitive lessons. Many of these insights surprised me and got me thinking in a different way. I hope they will for you too.  Let me guess: You have a high-priority project on your roadmap right now to add (more) AI features to your product. You’re in good company. A recent [survey](https://docsend.com/view/pi629vaxy57u9qvb) by Emergence Capital found that 60% of companies have already integrated generative AI into their products, and another 24% have it on their roadmap. AI is quickly eating the world. Unfortunately, many of these efforts will end in failure. Most early AI apps have a [“tourist”](https://twitter.com/omooretweets/status/1770470084601114837) problem: they get a lot of traction quickly but have [shockingly low retention and engagement](https://www.bvp.com/atlas/seven-product-strategies-to-prevent-churn-for-b2b-ai-app-leaders). And according to that Emergence survey, [two in five](https://docsend.com/view/pi629vaxy57u9qvb) gen AI products still haven’t made a single dollar despite companies spending millions (or even [billions](https://www.nytimes.com/2024/04/24/technology/meta-profit-stock-ai.html)) to build and support them. The goal of this post is to help you avoid wasting your precious time and resources by pointing your team in the wrong direction. I polled more than 20 of the sharpest AI product builders asking them for the most counterintuitive and surprising things they’ve learned about building AI into their products. These leaders have built many of today’s most loved and successful AI products, including products at Adobe, GitHub, Intercom, Perplexity, Canva, Runway, HeyGen, and Superhuman. Here’s what I learned:  ### **1. You first need to learn to think differently** > *“It takes time to think AI-native. The first-pass product is often a bolt-on or simple chat experience. **The high-value experience is a deeper rethink** once you have played with the technology, understood what it really provides more deeply, and then integrated it into a key part of your product experience.”* > > —[Elad Gil](https://www.linkedin.com/in/eladgil/), technology entrepreneur and investor [Watch on YouTube](https://www.youtube.com/watch?v=xcvIGJ3_H_k) > *“**It’s actually easier and safer for startups to work on hard problems, problems that cannot quite be solved with today’s foundation models.** We’re excited about riding the capability curve of improving models, instead of fighting that progress.”* > > —[Sarah Guo](https://twitter.com/saranormous), startup investor and founder at Conviction > *“Over the past 10 years, for most companies (with the exception of some hard-infra projects), it is assumed that what you want to build can be fairly easily built. You start by deeply understanding the customer problem and opportunity, design what you believe to be a great solution, and build it. **AI is different.*** > > *With AI, it is wholly unclear if something is possible to build. And when it is built, it is wholly unclear if it is any good, even if it appears to be good. Often the best place to start projects is by asking, **‘What is technologically possible?’ and prototyping.** This is a big mindset shift for anyone who has built software for the past decade and followed standard best practice.”* > > —[Paul Adams](https://www.linkedin.com/in/pauladams/), chief product officer at Intercom [Watch on YouTube](https://www.youtube.com/watch?v=4fJ3rYVlfYE) ### **2. Though it still comes down to who can solve real problems for people best** > *“**Demo value isn’t user value.** Building a cool AI demo doesn’t mean we have a product that customers love and is useful.”* > > —[Joshua Xu](https://www.linkedin.com/in/buffxz/), co-founder and CEO at HeyGen [Watch on YouTube](https://www.youtube.com/watch?v=SEE875kutbk) > *“I spend a lot of time thinking about adoption curve segmentation—identifying who adopts a new product quickly and who does not, and what distinguishes these groups. Historically, I’d focus on understanding the value a new product provides to people with different functional needs. **AI changes this dynamic because the most meaningful segmentation often depends on attitudes toward the technology itself: AI embracers versus AI skeptics.*** > > *Many have discussed the AI ‘phantom PMF’ phenomenon, where novelty-driven acquisition leads to a steep churn cliff, but the reverse is also true. I frequently speak with customers who reject AI products that meet their needs simply because they don’t trust or want to embrace AI. With the right messaging and onboarding, these skeptics can become superusers! But they behave very differently from AI embracers. I sketched this out in the chart below.* > >  > > *As a result, I’ve had to rethink our user testing approach from the ground up. When testing AI products, I focus on the following:* > > 1. *Longitudinal validation: Are we testing for a long enough period to understand how engagement changes once the novelty wears off?* > 2. *High-touch testing: Are we staying close enough to users to understand how their attitudes, which drive engagement patterns, change daily? We’re experimenting with user Slack groups instead of traditional surveys and one-on-one qualitative interviews.* > 3. *Attitudinal segmentation: Are we including both AI embracers and AI skeptics in our early testing groups? Crucially, are we segmenting them carefully to avoid averaging out their engagement and creating [‘tepid tea’](https://theaccidentalmarketer.com/blog/why-your-b2b-company-should-focus-on-needs-based-segmentation/)—a product that satisfies no one?”* > > —[Hilary Gridley](https://www.linkedin.com/in/hilarygridley/), head of core app product at WHOOP [Watch on YouTube](https://www.youtube.com/watch?v=zIaNjKYImFA) > *“Building fantastic product experiences hasn’t gotten easier with AI. Sci-fi capabilities of the models are inspiring, but that’s not what makes AI products great. Good old-fashioned product engineering does. **This means homing in on real user pain points, iterating closely with customers, and holding a high bar for a delightful user experience.**”* > > —[Caitlin Colgrove](https://www.linkedin.com/in/colgrove/), co-founder and chief technology officer at Hex [Watch on YouTube](https://www.youtube.com/watch?v=Gvq7M0Cd9HQ) > *“Most people think about AI-assisted services in terms of the model quality, but model quality is just a tiny piece of the total product. It turns out that post-processing filters, contractual guarantees, data privacy, feedback loops, observable human impact, etc. are all far more important. **In other words, building AI products looks a lot like building products.**”* > > —[Ryan J. Salva](https://www.linkedin.com/in/ryanjsalva/), vice president of product at GitHub ### **3. Paired with the right product design and UX to teach people how to use it** > *“The promise of AI (LLMs in particular) is to create anything for you with just a few words. But as we have seen over the past decade with Canva, when you give people the power to do anything, it can be pretty daunting and they don’t know where to start. So—just as with the very first version of our design tool—giving people the right starting points and confidence to utilize AI is a crucial part of delivering a great AI product.* > > *The evolution of our Magic Media feature is a great example of this. Text-to-image is a magical piece of technology when you know what image you want and how to describe it. But most people don’t have the right vocabulary to properly explain what they’re looking for; or even worse, they don’t know what they’re looking for!* > > *Our iterations on Magic Media have lessened that fear-inducing empty prompt box and introduced more visual options to guide you to a great image, as well as help to get people prompting in the right way. We’re also focusing on what happens after that generation—how you tweak and adjust what an AI has given to be exactly what you want.* > > ***All of this underscored to us that AI tools require a combination of intuitive product design and broader, ongoing education to support these behavior shifts.** You can't ‘flip the switch’ with AI—society is in the midst of change at a cultural level, but well-built products can support this shift.”* > > —[Cameron Adams](https://www.linkedin.com/in/themaninblue/?originalSubdomain=au), co-founder and chief product officer at Canva > *“Experimenting to find the right UI/UX for an AI feature can have an equally big impact on conversion metrics as research updates to the AI model itself. The right UX doesn’t just make new model capabilities more discoverable—it actually improves the conversion for users who use that capability, even if they were already using the capability anyway in the original UI.”* > > —[Joel Kwartler](https://www.linkedin.com/in/joelkwartler/), head of product at Runway [Watch on YouTube](https://www.youtube.com/watch?v=nByslCkykj8) ### **4. And the (proprietary) data you have access to (and a license for) is becoming increasingly important** > *“The data and the interfaces may become more important than the models themselves, which are becoming increasingly commoditized, available via open source, and pushed to the edge (we’ll be running many models locally on devices within a few years). The AI products I am most excited about leverage a proprietary or uniquely structured set of data—for which they have a license to use rather than scrape it—and a superior interface that transforms an antiquated workflow.* > > *What are the implications of this? **Companies that have or really understand data in a deep vertical will have an advantage.** **And designers will be more important than ever before**, imagining entirely new ways to transform the interfaces of our everyday work and life with the superpowers of AI.”* > > —[Scott Belsky](https://www.linkedin.com/in/scottbelsky/), chief strategy officer and executive vice president (EVP) of design and emerging products at Adobe; founder of Behance [Watch on YouTube](https://www.youtube.com/watch?v=vpiALnqE-VQ) ### **5. Be intentional about your initial wedge workflow** > *“**Start with a core workflow that feels like a chore where the promise-to-payoff is high if you get it right.** You want to select a place where the upfront user effort (like taking the leap to try it out or customizing it) yields a big reward (like substantial time savings) and invites repeat use.”* > > —[Paige Costello](https://www.linkedin.com/in/paigecostello/), head of AI at Asana  ### **6. Just calling your product “AI” could give you a boost** > *“We talk so much in software about how users don’t care about how something is built; they care about the job software does for them. **But we’ve found that branding AI-powered products as ‘AI-powered’ (as cringe as that can be) does more than increase initial engagement—it also helps users understand what the feature is capable of and how they should interact with it.** We sell a chatbot product and typically encourage our customers to brand their chatbot as AI-powered in some way. When they do this, more users use it, and they also use it better.”* > > —[James Evans](https://www.linkedin.com/in/james-evans-7086b3126/), co-founder and CEO at CommandBar  > ***“ ‘AI mandates’ at larger companies create a window for startups to gain customers faster than they could pre-AI**,simply because the board and CEO of larger companies are pushing for something in AI to happen in their org.”* > > —[Elad Gil](https://www.linkedin.com/in/eladgil/), technology entrepreneur and investor ### **7. And even small AI-driven improvements can go a long way** > *“**The smallest (and almost invisible) features are usually the fan favorites.** Things like pre-filling names, tiny bits of UI magic, simple data transformations often have a bigger impact and more customer adoption than the Big AI Features™ like chatbots or complex agents. You don’t need a big idea to ship useful AI features into your app.”* > > —[Claire Vo](https://www.linkedin.com/in/clairevo/), chief product officer at LaunchDarkly and founder at ChatPRD > *“**Look less at ‘what cool new things could AI do’ but more at ‘what’s the thing our users do 100 times a day that AI could make better.’** An example for us is writing a summary for an incident. It turns out that users vastly, vastly prefer automatically generated summaries to writing these themselves; 75% of incident summaries are now AI-generated.”* > > —[Stephen Whitworth](https://www.linkedin.com/in/stephenwhitworth/), co-founder and CEO at Incident.io [Watch on YouTube](https://www.youtube.com/watch?v=NHUhC5vHDZk) ### **8. Anticipate the AI won’t be perfect, so give users options, and keep fine-tuning the prompts** > *“People need and want a lot of customization to make AI products work for their use case. They are willing to put in a lot of effort curating prompts, experimenting with models, and building on APIs that were never even imaginable a year ago.”* > > —[Johnny Ho](https://www.linkedin.com/in/hjohnny/), co-founder and chief strategy officer at Perplexity  > *“It may seem obvious, but traditional software is deterministic—it either does a thing or doesn’t. Services built using LLMs, on the other hand, are probabilistic. They may occasionally produce a helpful response, and other times not. **The trick is achieving consistency in the model responses so that users can generally expect a good result.*** > > *GitHub Copilot has an acceptance rate of 35%. That is to say, developers commit 35% of Copilot suggestions into their code editor. A ‘good enough’ acceptance rate will differ based on the use case and customer cohort. The best way to find out what’s right for you is to ask your customers, ‘Is this making your job easier?’ ”* > > —[Ryan J. Salva](https://www.linkedin.com/in/ryanjsalva/), vice president of product at GitHub  > *“A significant amount of AI innovation is in the prompt. Time and time again, I’ve seen teams reach this aha moment of realizing **they don’t necessarily need to build new features, but they need to spend time on making their prompts better**—prompts to improve quality, improve the results, or prompts which solve a problem for a customer in a unique way. A meaningful prompt can solve meaningful problems for customers.”* > > —[Sherif Mansour](https://www.linkedin.com/in/sherifmansour/?originalSubdomain=au), distinguished product manager at Atlassian ### **9. Also anticipate the base model constantly improving** > *“[RAG](https://aws.amazon.com/what-is/retrieval-augmented-generation/) works surprisingly well if you give it the right context and ask the model to include links to the sources. If built the right way, AI-infused products automatically get better on their own when the underlying model improves. For example, Limitless will get better overnight when we move from GPT-4 to GPT-5. So build products in a way that leverages the strengths of the underlying foundation models and doesn’t compete with those strengths. Assume the models will keep getting stronger.* > > —[Dan Siroker](https://www.linkedin.com/in/dsiroker/), co-founder and CEO at Limitless [Watch on YouTube](https://www.youtube.com/watch?v=lt_WnR_GZqs) ### **10. Watch out for scalability bottlenecks** > *“Initially an AI model might perform well with a limited set of tasks or data. **However, not planning for the scalability of both the AI system and its infrastructure can lead to performance bottlenecks as user demands increase**.This oversight can result in slow response times, decreased accuracy, and an inability to integrate new data or features efficiently. Scalability planning from the outset is crucial to support growth and changing requirements.”* > > —[Chris Lu](https://www.linkedin.com/in/chris-lu11/), co-founder at Copy.ai  > *“It’s easy to build too much ‘scaffolding’ logic on top of base models in the pursuit of optimizing quality. Fine-tuning and few-shot example training the models directly have been far more effective at improving quality for us. Beyond those, the product needs to be designed to use the raw model output as directly as possible to maximize quality, not to mention support a wide range of inputs and work reliably.”* > > —[Henri Liriani](https://www.linkedin.com/in/hliriani/), co-founder and chief product officer of Tome  ### **11. Keep an eye out for time spent in-app going down** > *“While the previous generation of AI/ML breakthroughs like recommendation or ranking systems were finding the best product-market fit with products that took time away from people (social media, etc.), this new wave of generative AI seems to find the best PMF with products that increase productivity and give time back to people. **So don’t be surprised if the best AI features reduce metrics like time spent in-app.**”* > > —[Gaurav Misra](https://www.linkedin.com/in/gamisra1/), co-founder and CEO at Captions [Watch on YouTube](https://www.youtube.com/watch?v=X9yNocxacHs) ### **12. Oftentimes, speed alone will help you win** > *“**The thing we’ve learned: speed wins.** Take, for example, Instant Reply or Auto Summarize. Gmail and Outlook have similar features, but you have to generate the replies and summaries on demand—and then wait for them to finish generating. In Superhuman, we pre-compute them so they are always instantaneous. That simple difference is a massive lever on the user experience.”* > > —[Rahul Vohra](https://www.linkedin.com/in/rahulvohra/), founder and CEO at Superhuman [Watch on YouTube](https://www.youtube.com/watch?v=OiNWdByBU98) ## **Closing thoughts** Embracing AI opportunities calls for a new approach to building products. The companies that succeed will bring different mindsets and frameworks to unlock 10x outcomes. While the tech gets the attention, the winners will still be the products that solve real problems for people the best and provide the best design and UX to train people how to use it. Be intentional about your initial wedge, and look for a workflow with a big reward and potential for repeat use. The last mile of building AI products can make all the difference. How long do people have to wait for the AI to generate? How consistently helpful are the outputs? Can people customize the prompt for their use case? And remember that we’re still in the early days of AI. Models will make step-function improvements. People will get more familiar with the technology. Keep up with changing expectations to build winning AI products that last. Your first-pass product may be a miss, but keep going and don’t shy away from hard problems. **—** *A huge thank-you to [Caitlin Colgrove](https://www.linkedin.com/in/colgrove/), [Cameron Adams](https://www.linkedin.com/in/themaninblue/), [Chris Lu](https://www.linkedin.com/in/chris-lu11/), [Claire Vo](https://www.linkedin.com/in/clairevo/), [Dan Siroker](https://www.linkedin.com/in/dsiroker/), [Elad Gil](https://www.linkedin.com/in/eladgil/), [Gaurav Misra](https://www.linkedin.com/in/gamisra1/), [Henri Liriani](https://www.linkedin.com/in/hliriani/), [Hilary Gridley](https://www.linkedin.com/in/hilarygridley/), [James Evans](https://www.linkedin.com/in/james-evans-7086b3126/), [Joel Kwartler](https://www.linkedin.com/in/joelkwartler/), [Johnny Ho](https://www.linkedin.com/in/hjohnny/), [Joshua Xu](https://www.linkedin.com/in/buffxz/), [Paige Costello](https://www.linkedin.com/in/paigecostello/), [Paul Adams](https://www.linkedin.com/in/pauladams/), [Rahul Vohra](https://www.linkedin.com/in/rahulvohra/), [Ryan J. Salva](https://www.linkedin.com/in/ryanjsalva/), [Sarah Guo](https://twitter.com/saranormous), [Scott Belsky](https://www.linkedin.com/in/scottbelsky/), [Sherif Mansour](https://www.linkedin.com/in/sherifmansour/?originalSubdomain=au), and [Stephen Whitworth](https://www.linkedin.com/in/stephenwhitworth/) for contributing their insights.* *Have a fulfilling and productive week 🙏* ## 👀 Hiring? Or looking for a new job? I’m piloting a white-glove recruiting service for product roles, working with a few select companies at a time. If you’re hiring for senior product roles, apply below. [Apply to join](https://www.lennysjobs.com/talent/) If you’re exploring new opportunities yourself, use the same button above to sign up. We’ll send over personalized opportunities from hand-selected companies if we think there’s a fit. Nobody gets your info until you allow them to, and you can leave anytime. **If you’re finding this newsletter valuable, share it with a friend, and consider subscribing if you haven’t already. There are [group discounts](https://www.lennysnewsletter.com/subscribe?group=true), [gift options](https://www.lennysnewsletter.com/subscribe?gift=true), and [referral bonuses](https://www.lennysnewsletter.com/leaderboard) available.** Sincerely, Lenny 👋