Angela Duckworth

Katherine L. Milkman, Sean F. Ellis, Dena Gromet, Isabella M. DeMay, Heather N. Graci, Youngwoo Jung, Rayyan S. Mobarak, Ramon A. Silvera Zumaran, Mia N. Simmons, Christophe Van den Bulte, Shlomo Benartzi, Matthew Hilchey, Laura Goodyear, Dean Karlan, Nina Mazar, Daniel Mochon, Avni M. Shah, Dilip Soman, Jonathan Zinman, Angela Duckworth (2025), Can Reminder Emails Compel Americans to Save? A Two-Million Person Megastudy, Proceedings of the National Academy of Sciences Nexus. https://doi.org/10.1093/pnasnexus/pgaf280

Abstract: In the United States, 24% of adults have no savings and 39% have less than a month of income saved. We present results from a megastudy where nearly 2 million customers of a US bank were randomly assigned to receive one of seven different 2-month email campaigns, each employing a different behavioral science insight to nudge one-time and recurring savings deposits and increase savings balances or to a control condition without such messages. These campaigns increased the probability of making a one-time savings deposit, on average, by 0.05 percentage points (a 0.51% increase over control). The best-performing campaign delivered weekly messages to customers that differed depending on recent savings behavior: messages to customers who had not made a savings account deposit in the last week included a simple reminder to save, while those to customers who had made a savings account deposit in the prior week were congratulated on this accomplishment. This top-performing campaign increased the monthly likelihood that a customer made a one-time savings deposit by 0.13 percentage points (a 1.32% increase). We estimate that rolling this 2-month campaign out to everyone in our megastudy population would have led to an extra $6,123,996 to $9,910,090 in savings. Together, our findings highlight that light-touch, frequent email nudges can cost-effectively create small increases in savings deposits in the United States. Ideally, to generate meaningful benefits, behavioral science insights would be incorporated into a wider range of communications and incentives designed by financial institutions.

Erika L. Kirgios, Susan Athey, Angela Duckworth, Dean Karlan, Michael Luca, Katherine L. Milkman, Molly Offer-Westort (2025), Does Q&A Boost Engagement? Health Messaging Experiments in the United States and Ghana, Management Science. https://doi.org/10.1287/mnsc.2024.04405

Abstract: Effective information sharing is critical for the success of organizations and governments. Because information that is easy to access is more likely to be adopted, leaders often minimize friction in information delivery. However, one type of friction may increase engagement: piquing curiosity by posing relevant questions prior to sharing information. To test this, we shared identical information about COVID-19 in either question-and-answer format or via direct statements across two preregistered field experiments in Ghana and Michigan (total n = 49,395). Q&A-style communication increased information seeking about directly related topics (e.g., how to wear a mask properly) by 1.0 percentage point (216%) in Ghana and by 1.1 percentage points (19%) in Michigan (p’s < 0.001) and increased self-reported behavior change by 1.3 percentage points (4%) in Michigan (p = 0.002). However, sharing information in Q&A format did not increase interest in general COVID-19 information in either setting, suggesting that the impact of Q&A-style messaging on information seeking may be issue specific. In Michigan, both Q&A-style and direct statement messaging produced less information seeking than sending no informational messages, likely because of differential attrition: the more texts participants received, the more likely they were to opt out of receiving messages, which made it impossible for them to seek more information via text. In a follow-up implementation experiment with social media ads (a messaging strategy without attrition challenges), Q&A-style ads generated 9%–11% more unique clicks to the CDC website per dollar spent than ads that directly stated information about vaccines (p < 0.001). We speculate that Q&A-style information delivery may stimulate curiosity, driving its benefits.

Angela Duckworth, Ahra Ko, Katherine L. Milkman, Joseph S. Kay, Eugen Dimant, Dena Gromet, Aden Halpern, Youngwoo Jung, Madeline K. Paxson, Ramon A. Silvera Zumaran, Ron Berman, Ilana Brody, Colin Camerer, Elizabeth Canning, Hengchen Dai, Marcos Gallo, Hal E. Hershfield, Matthew Hilchey, Ariel Kalil, Kathryn Kroeper, Amy Lyon, Benjamin Manning, Nina Mazar, Michelle Michelini, Susan Mayer, Mary Murphy, Philip Oreopoulos, Sharon Parker, Renante Rondina, Dilip Soman, Christophe Van den Bulte (2025), National Megastudy Shows that Email Nudges to Elementary School Teachers Boost Student Math Achievement, Particularly When Personalized, Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2418616122

Abstract: In response to the alarming recent decline in US math achievement, we conducted a national megastudy in which 140,461 elementary school teachers who collectively taught 2,992,027 students were randomly assigned to receive a variety of behaviorally informed email nudges aimed at improving students’ progress in math. Specifically, we partnered with the nonprofit educational platform Zearn Math to compare the impact of 15 different interventions with a reminder-only megastudy control condition. All 16 conditions entailed weekly emails delivered to teachers over 4-wk in the fall of 2021. The best-performing intervention, which encouraged teachers to log into Zearn Math for an updated report on how their students were doing that week, produced a 5.06% increase in students’ math progress (3.30% after accounting for the winner’s curse). In exploratory analyses, teachers who received any behaviorally informed email nudge (vs. a reminder-only megastudy control) saw their students’ math progress boosted by an average of 1.89% during the 4-wk intervention period; emails referencing personalized data (i.e., classroom-specific statistics) outperformed emails that did not by 2.26%. While small in size, these intervention effects were consistent across school socioeconomic status and school type (public, private, etc.) and, further, persisted in the 8-wk post-intervention period. Collectively, these findings underscore both how difficult it is to change behavior and the need for large-scale, rigorous, empirical research of the sort undertaken in this megastudy.

Linnea Gandhi, Benjamin Manning, Angela Duckworth, Effect Size Magnification: No Variable is as Important as the One You’re Thinking About—While You’re Thinking About It.

Abstract: The goal of psychological science is to discover truths about human nature, and the typical form of empirical insights is a simple statement of the form x relates to y. We suggest that such “one-liners” imply much larger x-y relationships than those we typically study. Given the multitude of factors that compete and interact to influence any human outcome, small effect sizes should not surprise us. And yet they do—as evidenced by the persistent and systematic underpowering of research studies in psychological science. We suggest an explanation. Effect size magnification is the tendency to exaggerate the importance of the variable under investigation because of the momentary neglect of others. Although problematic, this attentional focus serves a purpose akin to that of the eye’s fovea. We see a particular x-y relationship with greater acuity when it is the center of our attention. Debiasing remedies are not straightforward, but we recommend (a) recalibrating expectations about the effect sizes we study, (b) proactively exploring moderators and boundary conditions, and (c) periodically toggling our focus from the x variable we happen to study to the non-x variables we do not.

Katherine L. Milkman, Sean F. Ellis, Dena Gromet, Alex S. Luscher, Rayyan S. Mobarak, Madeline K. Paxson, Ramon A. Silvera Zumaran, Robert Kuan, Ron Berman, Neil A. Lewis Jr. John A. List, Mitesh S. Patel, Christophe Van den Bulte, Kevin G. Volpp, Maryann V. Beauvais, Jonathan K. Bellows, Cheryl A. Marandola, Angela Duckworth (2024), Megastudy Shows That Reminders Boost Vaccination But Adding Free Rides Does Not, Nature. https://doi.org/10.1038/s41586-024-07591-x

Abstract: Encouraging routine COVID-19 vaccinations is likely to be a crucial policy challenge for decades to come. To avert hundreds of thousands of unnecessary hospitalizations and deaths, adoption will need to be higher than it was in the autumn of 2022 or 2023, when less than one-fifth of Americans received booster vaccines. One approach to encouraging vaccination is to eliminate the friction of transportation hurdles. Previous research has shown that friction can hinder follow-through and that individuals who live farther from COVID-19 vaccination sites are less likely to get vaccinated. However, the value of providing free round-trip transportation to vaccination sites is unknown. Here we show that offering people free round-trip Lyft rides to pharmacies has no benefit over and above sending them behaviourally informed text messages reminding them to get vaccinated. We determined this by running a megastudy with millions of CVS Pharmacy patients in the United States testing the effects of (1) free round-trip Lyft rides to CVS Pharmacies for vaccination appointments and (2) seven different sets of behaviourally informed vaccine reminder messages. Our results suggest that offering previously vaccinated individuals free rides to vaccination sites is not a good investment in the United States, contrary to the high expectations of both expert and lay forecasters. Instead, people in the United States should be sent behaviourally informed COVID-19 vaccination reminders, which increased the 30-day COVID-19 booster uptake by 21% (1.05 percentage points) and spilled over to increase 30-day influenza vaccinations by 8% (0.34 percentage points) in our megastudy. More rigorous testing of interventions to promote vaccination is needed to ensure that evidence-based solutions are deployed widely and that ineffective but intuitively appealing tools are discontinued.

Anastasia Buyalskaya, Hung Ho, Katherine L. Milkman, Xiaomen Li, Angela Duckworth, Colin Camerer (2023), What Can Machine Learning Teach Us about Habit Formation? Evidence from Exercise and Hygiene, Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2216115120

Abstract: We apply a machine learning technique to characterize habit formation in two large panel data sets with objective measures of 1) gym attendance (over 12 million observations) and 2) hospital handwashing (over 40 million observations). Our Predicting Context Sensitivity (PCS) approach identifies context variables that best predict behavior for each individual. This approach also creates a time series of overall predictability for each individual. These time series predictability values are used to trace a habit formation curve for each individual, operationalizing the time of habit formation as the asymptotic limit of when behavior becomes highly predictable. Contrary to the popular belief in a “magic number” of days to develop a habit, we find that it typically takes months to form the habit of going to the gym but weeks to develop the habit of handwashing in the hospital. Furthermore, we find that gymgoers who are more predictable are less responsive to an intervention designed to promote more gym attendance, consistent with past experiments showing that habit formation generates insensitivity to reward devaluation.

Angela Duckworth and Katherine L. Milkman (2022), A Guide to Megastudies, Proceedings of the National Academy of Sciences Nexus. https://doi.org/10.1093/pnasnexus/pgac214

Abstract: How can behavioral insights best be leveraged to solve pressing policy challenges? Because research studies are typically designed to test the validity of a particular idea, surprisingly little is known about the relative efficacy of different approaches to changing behavior in any given policy context. We discuss megastudies as a research approach that can surmount this and other obstacles to developing optimal behaviorally informed policy interventions. We define a megastudy as “a massive field experiment in which many different treatments are tested synchronously in one large sample using a common, objectively measured outcome.” We summarize this apples-to-apples approach to research and lay out recommendations, limitations, and promising future directions for scholars who might want to conduct or evaluate megastudies.

Aneesh Rai, Marissa A. Sharif, Edward H. Chang, Katherine L. Milkman, Angela Duckworth (2022), A Field Experiment on Subgoal Framing to Boost Volunteering: The Trade- Off Between Goal Granularity and Flexibility, Journal of Applied Psychology. https://doi.org/10.1037/apl0001040

Abstract: Research suggests that breaking overarching goals into more granular subgoals is beneficial for goal progress. However, making goals more granular often involves reducing the flexibility provided to complete them, and recent work shows that flexibility can also be beneficial for goal pursuit. We examine this trade-off between granularity and flexibility in subgoals in a preregistered, large-scale field experiment (N = 9,108) conducted over several months with volunteers at a national crisis counseling organization. A preregistered vignette pilot study (N = 900) suggests that the subgoal framing tested in the field could benefit goal seekers by bolstering their self-efficacy and goal commitment, and by discouraging procrastination. Our field experiment finds that reframing an overarching goal of 200 hr of volunteering into more granular subgoals (either 4 hr of volunteering every week or 8 hr every 2 weeks) increased hours volunteered by 8% over a 12-week period. Further, increasing subgoal flexibility by breaking an annual 200-hr volunteering goal into a subgoal of volunteering 8 hr every 2 weeks, rather than 4 hr every week, led to more durable benefits.

Katherine L. Milkman, Linnea Gandhi, Sean F. Ellis, Heather N. Graci, Dena Gromet, Rayyan S. Mobarak, Alison Buttenheim, Angela Duckworth, Devin Pope, Ala Stanford, Richard H Thaler, Kevin Volpp (2022), A Citywide Experiment Testing the Impact of Geographically Targeted, High-Pay-Off Vaccine Lotteries, Nature Human Behaviour. https://doi.org/10.1038/s41562-022-01437-0

Abstract: Lotteries have been shown to motivate behaviour change in many settings, but their value as a policy tool is relatively untested. We implemented a pre-registered, citywide experiment to test the effects of three high-pay-off, geographically targeted lotteries designed to motivate adult Philadelphians to get their COVID-19 vaccine. In each drawing, the residents of a randomly selected ‘treatment’ zip code received half the lottery prizes, boosting their chances of winning to 50×–100× those of other Philadelphians. The first treated zip code, which drew considerable media attention, may have experienced a small bump in vaccinations compared with the control zip codes: average weekly vaccinations rose by an estimated 61 per 100,000 people per week (+11%). After pooling the results from all three zip codes treated during our six-week experiment, however, we do not detect evidence of any overall benefits. Furthermore, our 95% confidence interval provides a 9% upper bound on the net benefits of treatment in our study.

Mitesh Patel, Katherine L. Milkman, Linnea Gandhi, Heather N. Graci, Dena Gromet, Hung Ho, Joseph S. Kay, Timothy W. Lee, Jake Rothschild, Modupe Akinola, John Beshears, Jonathan E. Bogard, Alison Buttenheim, Christopher F. Chabris, Gretchen B. Chapman, James J. Choi, Hengchen Dai, Craig R. Fox, Amir Goren, Matthew D. Hilchey, Jillian Hmurovic, Leslie K. John, Dean Karlan, Melanie Kim, David Laibson, Cait Lamberton, Brigitte C. Madrian, M. Meyer, Maria Modanu, Jimin Nam, Todd Rogers, Renante Rondina, Silvia Saccardo, Maheen Shermohammed, Dilip Soman, Jehan Sparks, Caleb Warren, Megan Weber, Ron Berman, Chalanda N. Evans, Seung Hyeong Lee, Christopher K. Snider, Eli Tsukayama, Christophe Van den Bulte, Kevin Volpp, Angela Duckworth (2022), A Randomized Trial of Behavioral Nudges Delivered through Text Messages to Increase Influenza Vaccination Among Patients with an Upcoming Primary Care Visit, American Journal of Health Promotion, 37 (3), pp. 324-332.

Abstract: Purpose: To evaluate if nudges delivered by text message prior to an upcoming primary care visit can increase influenza vaccination rates. Design: Randomized, controlled trial. Setting: Two health systems in the Northeastern US between September 2020 and March 2021. Subjects: 74,811 adults. Interventions: Patients in the 19 intervention arms received 1-2 text messages in the 3 days preceding their appointment that varied in their format, interactivity, and content. Measures: Influenza vaccination. Analysis: Intention-to-treat. Results: Participants had a mean (SD) age of 50.7 (16.2) years; 55.8% (41,771) were female, 70.6% (52,826) were White, and 19.0% (14,222) were Black. Among the interventions, 5 of 19 (26.3%) had a significantly greater vaccination rate than control. On average, the 19 interventions increased vaccination relative to control by 1.8 percentage points or 6.1% (P = .005). The top performing text message described the vaccine to the patient as “reserved for you” and led to a 3.1 percentage point increase (95% CI, 1.3 to 4.9; P < .001) in vaccination relative to control. Three of the top five performing messages described the vaccine as “reserved for you.” None of the interventions performed worse than control. Conclusions: Text messages encouraging vaccination and delivered prior to an upcoming appointment significantly increased influenza vaccination rates and could be a scalable approach to increase vaccination more broadly.