How Duolingo reignited person development
???? Hey, Lenny right here! Welcome to this month’s ✨ free version ✨ of Lenny’s E-newsletter. Every week I humbly sort out reader questions on product, development, working with people, and the rest that’s stressing you out about work.
When you’re not a subscriber, right here’s what you missed this month:
Subscribe to get entry to those posts, and each put up.
I used to be at a small occasion just a few months again the place Jorge Mazal (former CPO of Duolingo) shared the story behind Duolingo’s development reaccelerating. I used to be captivated. I’ve by no means seen a development story like this earlier than—4.5x development for a mature product, pushed by a small handful of product adjustments, rooted in an progressive development mannequin, and defined in such actionable element. I requested Jorge if he’d be keen to share (and increase on) the story with a broader viewers, and I’m so completely satisfied he agreed. Many merchandise already look to Duolingo for inspiration, and I think this story will solely enhance that pattern. Take pleasure in!
Observe Jorge for extra on LinkedIn and Twitter.
I joined Duolingo because the Head of Product in late 2017. Duolingo was already probably the most downloaded training app on the earth, with a whole bunch of tens of millions of customers, fulfilling its mission to “develop the most effective training on the earth and make it universally out there.” Nevertheless, person development was slowing down. By mid-2018, each day lively customers (DAU) have been rising at a single-digit fee year-over-year, which was troubling, given the explosive development the corporate had seen previously. This was an issue for a startup with traders anxious to see quick monetization development.
On this put up I’ll cowl a few of our early failures after which our first large wins that helped us flip round development, together with launching leaderboards, refocusing on push notifications, and optimizing the “streak” characteristic. These, along with a number of different efforts throughout Product and Advertising, helped us develop DAU by 4.5x over 4 years. Sturdy natural person development supercharged Duolingo towards its 2021 IPO.
This text is an in-depth look into that journey. It’s my hope that sharing this work will assist others discover their very own development breakthroughs sooner.
Our first try at reigniting development was targeted on bettering retention, i.e. fixing our “leaky bucket” downside. We prioritized engaged on retention over new-user acquisition as a result of all of our new-user acquisition was natural, and, on the time, we didn’t have an apparent lever to drag to supercharge that. Additionally, a few of us had a suspicion that we may enhance retention by gamification. There have been two predominant the explanation why this felt like the suitable method to me. First, Duolingo had already applied a number of gamification mechanics efficiently, such because the development system on the house display screen, streaks, and an achievements system. And second, high digital video games on the time had a lot increased retention charges than our product, which I took as proof that we hadn’t but reached the ceiling for gamification’s influence.
Armed with a brief presentation I co-created with our chief designer, we have been capable of get simply sufficient buy-in from the remainder of the chief group to create a brand new group, the Gamification Group. The group consisted of an engineering supervisor, an engineer, a designer, an APM, and me.
However there was one small concern: we had no concept which incremental gamification mechanics would work for Duolingo.
Our group on the time was hooked on a sport known as Gardenscapes, a cell, match-3 puzzle sport much like Sweet Crush. This cell sport turned our first inspiration.
As we seemed on the completely different sport mechanics in Gardenscapes, we didn’t actually know what we have been on the lookout for—we simply knew that Gardenscapes appeared stickier than Duolingo, and we noticed a number of parallels. A 3-minute Duolingo lesson felt much like a Gardenscapes match-3 degree, and Duolingo and Gardenscapes each used progress bars to supply visible suggestions on how shut the person was to finishing the session. Gardenscapes, nevertheless, paired its progress bar with a strikes counter, which Duolingo didn’t do. The strikes counter allowed customers solely a finite variety of strikes to finish a degree, which added a way of shortage and urgency to the gameplay. We determined to include the counter mechanic into our product. We gave our customers a finite variety of probabilities to reply questions accurately earlier than they needed to begin the lesson over.
It took our group a few months of labor so as to add the counter. With the discharge of the replace, I expectantly waited for an unmitigated success. Depressingly, the results of all that effort was utterly impartial. No change to our retention. No enhance in DAU. We hardly received any person suggestions in any respect. I used to be deflated. The best impact the initiative had was on our group. After the outcomes got here out, we rapidly fell into dissension. Some wished to proceed iterating on the concept, whereas others wished to pivot. The group nearly instantly (and dramatically) disbanded, and the concept was deserted. It was fairly terrible. The one silver lining of this failure was that I realized lots concerning the firm tradition and about tips on how to enhance my private management fashion—although that’s for a distinct article.
Feeling burned after our gamification effort, we utterly pivoted away from bettering retention and put collectively a brand new product group targeted on buying new customers, known as the Acquisition Group. On the time, Uber was doing properly with person acquisition and had seemingly grown largely due to its referral program. Impressed by this, we created a referral program much like Uber’s. The reward was a free month of our premium subscription, Tremendous Duolingo (on the time, it was known as Duolingo Plus). Appeared like a reasonably good supply to us!
We applied the characteristic and hoped our second try could be extra profitable. As an alternative, new customers elevated by solely 3%. It was optimistic, however not the kind of breakthrough we would have liked. Nonetheless, the group doubled down and pushed by, delivery iterations to the referral program and making another bets, however no avail.
Whereas the group continued to iterate, it turned clear to me that we needed to discover a completely different method to sort out our development downside.
The aftermath of those back-to-back failures in only some months was a interval of reflection for me about making higher product bets.
In hindsight, it turned clear why the Gardenscapes strikes counter was not a great match for our product. When you find yourself taking part in Gardenscapes, every transfer appears like a strategic determination, as a result of you need to outmaneuver dynamic obstacles to discover a path to victory. However strategic decision-making isn’t required to finish a Duolingo lesson—you largely both know the reply to a query otherwise you don’t. As a result of there wasn’t any technique to it, the Duolingo strikes counter was merely a boring, tacked-on nuisance. It was the flawed gamification mechanic to undertake into Duolingo. I noticed that I had been so targeted on the similarities between Gardenscapes and Duolingo that I had did not account for the significance of the underlying variations.
It additionally didn’t take lengthy to grasp why our referral program didn’t produce Uber-like success. Referrals work for Uber as a result of riders are paying for rides on a endless pay-as-you-go system. A free experience is a continuing incentive. For Duolingo, we have been attempting to incentivize customers by providing a free month of Tremendous Duolingo. Nevertheless, our greatest and most lively customers already had Tremendous Duolingo, and we couldn’t give them a free month after they have been already in a plan. This meant that our technique, which wanted to depend on our greatest customers, really excluded them.
In each of those conditions, we had borrowed profitable options from different merchandise, however the flawed manner. We had did not account for a way a change in context can influence the success of a characteristic. I got here away from these makes an attempt realizing that I wanted a greater understanding of tips on how to borrow concepts from different merchandise intelligently. Now when trying to undertake a characteristic, I ask myself:
-
Why is that this characteristic working in that product?
-
Why would possibly this characteristic succeed or fail in our context, i.e. will it translate properly?
-
What diversifications are essential to make this characteristic reach our context?
In different phrases, we would have liked to make use of higher judgment in adapting when adopting. Being extra systematic in simply this space would have made an enormous distinction in what gamification mechanics we selected to pursue. And we’d have most likely been dissuaded from specializing in referrals altogether. I used to be dedicated to creating certain our subsequent makes an attempt could be extra methodical. We would have liked to be higher at basing our choices on knowledge, insights, and foundational ideas.
Duolingo has at all times excelled at accumulating knowledge, particularly in help of A/B testing. However there hadn’t been a lot effort put into utilizing the information for insights technology. Having seen from the within how Zynga and MyFitnessPal used knowledge, I felt we may use Duolingo’s knowledge to discover a North Star metric and get the breakthrough we would have liked.
My time at Zynga and MyFitnessPal gave us inspiration on tips on how to phase and mannequin our customers by engagement degree. Zynga separated their customers and measured retention primarily based on the next weekly retention metrics:
-
Present customers retention fee (CURR): The possibility a person comes again this week in the event that they got here to the product every of the previous two weeks
-
New customers retention fee (NURR): The possibility a person comes again this week in the event that they have been new to the product final week
-
Reactivated person retention charges (RURR): The possibility a person comes again this week in the event that they reactivated final week
Later, after I labored at MyFitnessPal, I discovered that that they had adopted and expanded Zynga’s retention work. They not solely used CURR, NURR, and RURR to measure development but additionally to mannequin future situations. In addition they added SURR:
I hypothesized that we may use these metrics at Duolingo as a place to begin to create a extra refined mannequin, and use that mannequin to establish a North Star metric. Working with the information scientist and the engineer supervisor within the Acquisition Group, we got here up with the mannequin beneath. We used the identical retention charges as Zynga and MyFitnessPal, however we tweaked from a weekly view to a each day view and we added a number of extra metrics.
The blocks, or buckets, characterize completely different person segments with completely different ranges of engagement. And each single person who has ever used the product is in a single, and just one, bucket on any given day. Meaning the buckets within the mannequin are MECE (mutually unique, collectively exhaustive) in representing your complete base of customers who’ve ever used Duolingo. The arrows measure the motion of customers between the buckets (these embrace CURR, NURR, RURR, and SURR, however advanced into each day retention charges reasonably than weekly). Combining the buckets and the arrows, the mannequin creates an nearly closed-circuit system, with new customers being the one break.
Conveniently, the highest 4 buckets of the mannequin add as much as DAU. These buckets are outlined as:
-
New customers: first day of engagement ever within the app
-
Present customers: engaged immediately and at the very least one different time within the prior 6 days
-
Reactivated customers: first day of engagement after being away for 7-29 days
-
Resurrected customers: first day of engagement after being away for 30 days or longer
The remaining three buckets characterize customers who weren’t lively immediately and have completely different levels of inactivity.
-
At-risk WAU: inactive immediately, however lively in at the very least one of many prior 6 days
-
At-risk MAU: inactive previously seven days, however lively in at the very least one of many prior 23 days
-
Dormant customers: inactive previously 31 days or longer
The truth that DAU, WAU, and MAU can simply be calculated from these buckets made it simple to mannequin them over time. This can be a key characteristic of the mannequin. Moreover, by manipulating the charges represented by the arrows, we are able to mannequin the compounding and cumulative influence of transferring these charges over time; in different phrases, the charges are the levers product groups can pull to develop DAU.
With the mannequin created, we began taking each day snapshots of knowledge to create a historical past of how all of those person buckets and retention charges had advanced on a day-by-day foundation over the previous a number of years. With this knowledge, we may create a forward-looking mannequin after which carry out a sensitivity evaluation to foretell which levers would have the largest influence on DAU development. We ran a simulation for every fee, the place we moved a single fee 2% each quarter for 3 years, holding all the opposite charges fixed.
Under are the outcomes of our first simulation. It exhibits how these small 2% actions on every lever impacted forecasted MAU and DAU.
We instantly noticed that CURR had a huge influence on DAU—5 occasions the influence of the second-best metric. In hindsight, the CURR discovering made sense, as a result of the Present Consumer bucket has an attention-grabbing attribute: present customers who keep lively return to the identical bucket.
This produces a compounding impact, which signifies that CURR is way tougher to maneuver, however when it does, it’ll have a larger influence. Primarily based on this evaluation, we knew that CURR was the metric we needed to transfer with a purpose to get that strategic breakthrough we wished. We determined to create a brand new group, the Retention Group, with CURR as its North Star metric.
One of many greatest advantages of specializing in CURR was deciding to not work on issues that appeared paramount earlier than, particularly new-user retention. This was an enormous mindset shift for a corporation that had large success spending years working the majority of its development experiments on new customers first.
One other large lesson was seeing the large hole between how a metric may influence DAU vs. MAU; for instance, CURR’s influence on DAU was 6 occasions its influence on MAU. iWAURR (inactive WAU reactivation fee) was the second-best lever for transferring DAU however a distant fourth for transferring MAU, behind growing new and resurrected customers. This meant that, in some unspecified time in the future, we’d nonetheless want to determine new development vectors for new-user acquisition if we wished to see substantial MAU enhancements. However in the interim, our focus was solely on transferring DAU, so we prioritized CURR over all different development levers. And it turned out to be the suitable alternative.
With this clear directive, we checked out our historic mannequin knowledge and at our A/B assessments going again just a few years to see if we had inadvertently completed something that had moved CURR previously. Surprisingly, we hadn’t. In actual fact, CURR had not moved in years. We had to determine our first steps to maneuver CURR primarily based on first ideas.
I nonetheless thought gamification was a great place to begin when attempting to enhance retention. Our failure with the Gardenscapes-style strikes counter hadn’t really disproved any of the unique the explanation why we believed gamification nonetheless had upside for Duolingo—we had solely realized that the strikes counter was a slipshod try at it. This time, we’d be extra methodical and clever about options we added or borrowed. We made certain to use the teachings from our prior efforts with gamification.
After some consideration, we determined to guess on leaderboards. Right here’s why and the way. Duolingo already had a leaderboard for customers to compete with their family and friends, but it surely wasn’t notably efficient. Primarily based on my expertise at Zynga, I felt that there was a greater manner. Once I began engaged on Zynga’s FarmVille 2 sport, it included a leaderboard much like Duolingo’s current leaderboard, the place customers competed with their pals. I had hypothesized primarily based on my private expertise as a participant that the closeness of the competitor’s engagement could be extra vital than the closeness of non-public relationships. I believed this might be very true in a mature product the place many customers’ pals weren’t lively anymore. From our testing at Zynga, that concept turned out to be true. Primarily based on this, I felt a leaderboard system, much like what I had helped design at Zynga, would succeed within the context of our product.
FarmVille 2’s leaderboard additionally included a “league” system. Past attending to the highest of a weekly leaderboard, customers had the chance to maneuver by a sequence of league ranges (e.g. from the Bronze league to the Silver league to the Gold league). Leagues offered customers with a larger sense of progress and reward, an integral ingredient in sport design. In addition they elevated engagement over time, since engaged customers transfer as much as extra aggressive leagues week after week. We felt this characteristic would translate properly to Duolingo’s current product as a result of it tapped immediately into the frequent human motivators of competitiveness and development.
Not all features of the FarmVille 2 leaderboards would translate properly to Duolingo, although. We had to make use of our judgment to adapt this gaming mechanic to Duolingo’s context. In FarmVille 2, competing within the leaderboard required finishing further sorts of duties on high of the core gameplay. That was one thing that we purposefully ignored. Within the Duolingo context, extra duties would solely add pointless complexity to language studying. We intentionally made our leaderboard as informal and frictionless as potential; customers have been robotically opted in and will progress to the highest of the primary league by merely partaking persistently of their common language research. By protecting the sport mechanic thrilling, however making it easier than in FarmVille 2, we felt like we had struck the suitable stability of adopting and adapting.
The leaderboards characteristic had an enormous and nearly quick influence on our metrics. Total studying time elevated by 17%, and the variety of extremely engaged learners (customers who spend at the very least 1 hour a day for five days per week) tripled. At the moment, we hadn’t but discovered tips on how to calculate statistical significance for CURR, however we noticed that our conventional retention metrics (D1, D7, and so on.) improved materially and with statistical significance. Going ahead, the leaderboards characteristic turned a vector for bettering metrics, and groups proceed to optimize the characteristic to at the present time. Additionally importantly, the leaderboard was the Retention Group’s first breakthrough!
The Retention Group was utterly energized to search out extra mechanics to maintain present customers engaged and motivated to follow day by day. One space they began to look into was push notifications. Primarily based on substantial A/B testing in prior years, Duolingo had established that notifications is usually a large vector for development, however that influence had plateaued for us through the years. With a re-energized group full of latest concepts, we felt it was the suitable time to revisit this vector.
As we began diving into this, there was one precept that turned paramount. It got here from a cautionary story from Groupon’s CEO. He defined to Luis von Ahn, our CEO, that for a very long time, Groupon caught to at least one e-mail notification per day. However their group began questioning whether or not sending extra emails would enhance metrics. The CEO finally gave in and allowed his group to check sending yet one more e-mail to every person every day. This check resulted in an enormous enhance to their goal metrics. Inspired, Groupon saved experimenting, sending extra emails, at the same time as many as 5 a day. Then, in what felt like a change from sooner or later to the following, their e-mail channel misplaced most of its effectiveness. Over time, the buildup of Groupon’s aggressive e-mail assessments had principally destroyed their channel. One typically underappreciated danger with aggressively A/B testing emails and push notifications is that it leads to customers opting out of the channel; and even in case you kill the check, these customers stay opted out endlessly. Do that many occasions, and also you’ve destroyed your channel. This was the end result to keep away from. For our push notifications, we established one foundational rule: defend the channel.
With this constraint in thoughts, we determined to present the group lots of freedom to optimize on dimensions like timing, templates, photos, copy, localization, and so on., however they may not enhance the amount of notifications with out robust justification and CEO approval. Over time, by numerous iterations, A/B testing, and a bandit algorithm, the group was capable of generate dozens of small- and medium-size wins which have amounted to substantial good points in DAU 12 months after 12 months.
Within the seek for much more development vectors, the APM on the Retention Group began exploring whether or not there was a powerful correlation between retention and utilization of specific Duolingo options. He found that if a person reached a 10-day streak, their possibilities of dropping off have been lowered considerably. Clearly, lots of this was merely correlation and choice bias, however we felt the perception was attention-grabbing sufficient to begin investing in bettering this characteristic once more.
The idea of a streak is basically fairly easy: present customers the variety of consecutive days they’ve completed any exercise on the app. However it seems that there’s a surprisingly giant variety of optimization alternatives round streaks.
We received our first large win with the streak-saver notification—a notification that alerts customers with streaks if they’re about to lose their streak. This late-night notification proved that certainly there was appreciable upside to doubling down on streak optimizations. After this, a number of enhancements adopted: calendar views, animations, adjustments to streak freezes, and streak rewards, amongst others. Every helped enhance upon the unique streak concept and generated substantial enhancements to retention.
To this point, the streak characteristic is certainly one of Duolingo’s strongest engagement mechanics. When folks discuss their Duolingo expertise, they typically convey up their streak. I lately met one person who instructed me, “I’ve a 1,435-day streak!” and added, “with no streak freezes!” His bragging rights have been well-earned, as he had been finding out his chosen language each day for nearly 4 years.
Streaks work for quite a few causes. A kind of is {that a} streak will increase person motivation over time; the longer the streak is, the larger the impetus to maintain the streak going. In the case of person retention, that is the precise habits we would like in our customers. Every day {that a} learner involves Duolingo, they care a bit extra about coming again the following day than they did the day earlier than, therefore growing retention and DAU. As a meta-lesson, our success with the streak mechanic additional confirmed us that we may squeeze main wins from current options. We may see the worth in each large breakthroughs and in quick optimizations. And an A+ group typically has a mixture of each.
We didn’t cease at CURR; there was a really wholesome paranoia that in some unspecified time in the future CURR would hit a ceiling, so in the end we must determine development vectors for brand spanking new person acquisition. The Retention Group stayed targeted on growing CURR, however as an organization, we persistently elevated our funding in development by creating an increasing number of Product and Advertising groups to search out new vectors (for each retention and acquisition). Fortunately, a number of of those bets labored, together with increasing internationally, constructing social options (that is what the Acquisition finally group pivoted to, with nice success), accelerating course content material creation, working with influencers, growing our presence in colleges, investing (a bit bit) in paid UA, and going loopy viral on TikTok. Every of those deserves its personal case research.
By way of our efforts over 4 years, we have been capable of enhance CURR by 21%, which represents a discount within the each day churn of our greatest customers by over 40% and, along with our different profitable bets, led to a rise in our DAU of 4.5x. Final 12 months was one of many quickest development charges in Duolingo’s historical past. The standard of the person base additionally improved; the share of our DAU with a streak of seven days or longer elevated nearly 3 occasions to greater than half of our DAU. Which means not solely does Duolingo have a a lot increased variety of lively customers now, but additionally that these customers are more likely to maintain coming again, refer their pals, and subscribe to Tremendous Duolingo. This development was key to Duolingo’s profitable IPO.
I hope that this text offers you the inspiration it’s essential to discover new vectors of development to your product. When you undertake something from my expertise at Duolingo, I hope you adapt it to your personal context utilizing your greatest judgment. Don’t blindly belief what Duolingo or every other firm did. Actually that didn’t work for me. Pleased experimenting!
Gamification Group: You recognize who you’re. Thanks for instructing me a lot!
Acquisition Group: Vanessa Jameson (Engineer Director), Cem Kansu and Liz Nagler (PMs on the group, now VP of Product and Product Space Lead for Development, respectively), and the remainder of the group, who labored super-hard and finally made a sensible and profitable pivot to work on social options. Shoutout to Nico Sacheri (Principal PM) and Hideki Shima (Eng Director), who’ve been crushing it main the Connections group for the previous couple of years.
Development Mannequin: Erin Gustafson (Workers Knowledge Scientist) and Vanessa Jameson, who collaborated with me within the creation of the expansion mannequin. Study extra about how Erin is working to evolve the best way Duolingo thinks about development in her latest put up: https://blog.duolingo.com/growth-model-duolingo/
Retention Group: Sean Colombo (OG Engineer Supervisor for the group, and now Eng Space Lead for Development), Daniel Falabella (OG PM for the group, now GM for Duolingo ABC), John Trivelli (Designer on leaderboards), Anton Yu (PM who “re-discovered” streaks and a lot extra), Jackson Shuttleworth and Osman Mansur (Sr. PM and PM on the group immediately, nonetheless crushing it), Antonia Scheidel (Engineering Supervisor, additionally crushing it), and all of the fantastic engineers and designers who’ve labored and proceed to work on this group.
Gina Gotthilf, who was a complete development rock star in Duolingo’s early years.
Luis von Ahn (CEO) and Tyler Murphy (Chief Designer), with whom I reviewed each single product change for nearly 5 years.
Thanks, Jorge! You’ll be able to comply with Jorge for extra on LinkedIn and Twitter.
Have a satisfying and productive week ????
When you’re hiring, join Lenny’s Talent Collective to begin getting bi-monthly drops of world-class hand-curated product and development people who find themselves open to new alternatives.
When you’re on the lookout for a brand new gig, be part of the collective to get customized alternatives from hand-selected firms. You’ll be able to be part of publicly or anonymously, and go away anytime.
-
Athena: Head of Growth (Distant)
-
MetaMap: VP, Product (SF, Miami, Mexico Metropolis)
When you’re discovering this text priceless, share it with a pal, and think about subscribing in case you haven’t already.
Sincerely,
Lenny ????