Seminars / Conferences

In collaboration with Alibaba, this study leverages a large-scale field experiment to quantify the impact of a generative AI (gen AI) assistant on worker performance in an e-commerce after-sales service setting, where human agents provide customer support through digital chat. Agents were randomly assigned to either a control or treatment group, with the latter having access to a gen AI assistant that offers two automated functions as text messages: 1) diagnosis of customer order issues in real time and 2) solution suggestions. Agents exhibited varied gen AI usage behavior, choosing to use, modify, or disregard AI suggested messages. We employ two empirical approaches: 1) an intention-to-treat (ITT) analysis to estimate the average treatment effect of gen AI access, and 2) a local average treatment effect (LATE) analysis to estimate the causal impact of gen AI usage. Results show that the gen AI assistant significantly enhanced both service speed and service quality. Interestingly, gen AI automation did not lead agents to reduce effort; rather, it increased their engagement, evidenced by a higher message volume and agent-to-customer message ratio. Analysis by agents’ pretreatment performance reveals that low performers experienced greater improvements in speed and quality, narrowing the performance gap, while high performers saw a decline in service quality, likely because gen AI suggestions fell below their expertise. These findings underscore the potential of gen AI to improve operational efficiency and service quality while highlighting the need for tailored deployment strategies to support workers with varying skill levels.

Large-scale online platforms launch hundreds of randomized experiments (a.k.a. A/B tests) every day to iterate their operations and marketing strategies. The combinations of these treatments are typically not exhaustively tested, which triggers an important question of both academic and practical interest: Without observing the outcomes of all treatment combinations, how does one estimate the causal effect of any treatment combination and identify the optimal treatment combination? We develop a novel framework combining deep learning and doubly robust estimation to estimate the causal effect of any treatment combination for each user on the platform when observing only a small subset of treatment combinations. Our proposed framework (called debiased deep learning, DeDL) exploits Neyman orthogonality and combines interpretable and flexible structural layers in deep learning. We show theoretically that this framework yields efficient, consistent, and asymptotically normal estimators under mild assumptions, thus allowing for identifying the best treatment combination when observing only a few combinations. To empirically validate our method, we collaborated with a large-scale video-sharing platform and implemented our framework for three experiments involving three treatments where each combination of treatments is tested. When observing only a subset of treatment combinations, our DeDL approach significantly outperforms other benchmarks to accurately estimate and infer the average treatment effect of any treatment combination, and to identify the optimal treatment combination.

Our society remains profoundly unequal. This talk discusses how data science and machine learning can be used to combat inequality in health care and public health by presenting several vignettes from policing and other domains.

This study evaluates the impact of generative AI on software developer productivity via randomized controlled trials at Microsoft, Accenture, and an anonymous
Fortune 100 company. These field experiments, run by the companies as part of their ordinary course of business, provided a random subset of developers with access to
an AI-based coding assistant suggesting intelligent code completions. Though each experiment is noisy, when data is combined across three experiments and 4,867 developers, our analysis reveals a 26.08% increase (SE: 10.3%) in completed tasks among developers using the AI tool. Notably, less experienced developers had higher adoption rates and greater productivity gains.

Buoyed by advances in computation and AI, businesses have growing interest in adopting black-box algorithms for decision-making.  Motivated particularly by Reinforcement Learning (RL) at an industry partner, we compare model-based to model-free approaches on several stochastic optimization problems over time.  Here we define “model-based” to construct independent distributions over time and solve dynamic programming, and “model-free” to optimize over historical trajectories in a black-box fashion.

We provide a rigorous theoretical comparison, in a finite-horizon RL setting with offline trajectories that allows perfect hindsight evaluation. We derive surprising horizon-independence results for PAC-learning, which exploit the problem-specific structure of the Inventory Replenishment and Optimal Stopping problems. Perhaps more surprisingly, this requires different approaches for different problems: model-free for Inventory, and model-based for Stopping. Meanwhile, we show that a horizon-independent learning guarantee is impossible for the Pricing problem.

These theoretical findings are consistent with observations in simulations, and explanatory of successes/failures in deploying model-free RL at our industry partner.  The takeaway is that businesses should not adopt a one-size-fits-all verdict on whether to deploy black-box algorithms, as they are relatively advantaged for some problems but not others.

Several basic services, such as energy, clean water and cooking gas, are currently out of reach for millions of people living in poverty. There has been an emergence of private firms that offer these services by, for example, selling solar home systems or clean cooking packages with remote lockout capabilities. These firms deploy a pay-as-you-go (PAYGo) model in which consumers are given the flexibility to manage the amount and frequency of their payments based on their own erratic cash flows. However, because a firm under this model cannot observe how much money the consumers have, they can pay less to the firm and turn their income to other needs.

We employ an optimal contracting approach to investigate the incentives that can mitigate such misuse of payment flexibility. The optimal contract summarizes a consumer’s payment history with a single score, which we call the v-score. The contract allows the consumer to flexibly pay small (resp., large) amount when her disposable income is low (resp., high), and accordingly adjusts the v-score to incentivize payments. The v-score over time rises with relatively high payments and drops with relatively low payments. The v-score also determines the level of technology access granted to the consumer and whether the contract is terminated or continued. The contract proposes an inclusive downpayment scheme that allows consumers to purchase their initial v-score. We discuss how the optimal contract can be implemented in the field and how it can solve some of the practical problems currently faced by the PAYGo firms.

Forthcoming

Estimating the causal effects of an intervention on outcomes is crucial. But often in domains such as healthcare and social services, this critical information about outcomes is documented by unstructured text, e.g. clinical notes in healthcare or case notes in social services. We are motivated by an ongoing collaboration with a nonprofit providing street outreach to homeless clients. Outreach is a common social services intervention, with ambiguous and hard-to-measure outcomes. Outreach workers write case notes after every interaction, recording notable events and client progress beyond final housing placement outcomes. Although experts can succinctly extract relevant information from such unstructured case notes, it is costly or simply infeasible for an entire corpus (millions of notes). Recent advances in large language models (LLMs) enable scalable but potentially inaccurate annotation of unstructured text data. Under a framework of causal inference with missing outcomes, we develop estimators that robustly combine expert annotation vs. noisy imputation. To best leverage a limited annotation budget, we develop a two-stage adaptive algorithm that optimizes the expert annotation probabilities, estimating the ATE with optimal asymptotic variance. Expert labels and LLM annotations can be combined efficiently and responsibly in causal estimation. More generally, our work discusses the optimal choice of validation data for causal inference with non-standard measurement error. We discuss extensions to more complex estimands. This is joint work with Ezinne Nwankwo and Lauri Goldkind.

In recent years, novel data sources (e.g., consumer mobility data) have become increasingly available, alongside new information-sharing frameworks. This shift raises several key questions:  How do new data sources shape firms’ operational decisions? Under what conditions does the availability of such data meaningfully impact operational decisions? What are the optimal ways to share and leverage information? This talk explores these questions through two research vignettes.

The first part examines information sharing in supply chains. Unlike classical models, which assume firms cannot commit to an information-sharing policy, we adopt an information design approach that allows for commitment. We study a two-tier supply chain where a retailer, possessing a private signal about demand, seeks to influence a supplier’s production decisions through strategic information disclosure. We characterize the retailer’s optimal information-sharing policy and highlight settings where selective disclosure strictly outperforms full or no disclosure.

The second part is motivated by the large-scale availability of mobility data and focuses on facility location decisions. We introduce a variant of the facility location problem where each individual has multiple locations, and their cost of using a facility depends on the minimum distance from any of their locations. We establish fundamental complexity results, showing that no polynomial-time algorithm can achieve an approximation ratio better than 2. We then develop an approximation algorithm and establish various performance guarantees. Finally, we validate our approach through experiments on both synthetic data and U.S. Census data from major metropolitan areas.

Together, these studies demonstrate how firms can leverage novel data and structured information-sharing mechanisms to enhance decision-making in supply chains and optimize facility placement.

The first part is based on joint work with Huseyin Gurkan (ESMT) (see https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4378997), and the second part is based on joint work with Yiding Feng (MSR) (see https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4324967).

We discuss a dozen stylized facts about the capabilities of frontier AI models based on OpenAI’s experiences. We discuss various candidates for the “missing capability” which would unlock a greater economic impact. We also discuss how to model AI’s effect in general equilibrium, and how that depends on a characterization of AI’s capabilities.

Experiments have become a major component of data-driven decision making pipeline, allowing organizations to systematically evaluate potential interventions before full implementation. However, in many operational settings—from digital marketplaces to healthcare systems to supply chains—a key challenge is that interventions applied to one unit often affect outcomes for others. When marketplace promotions for one set of products impact the performance of non-promoted products due to shared demand or substitution effects, when treating one patient for a contagious disease affects transmission to others, this “network interference” compromises our ability to accurately measure intervention effects. This talk introduces a framework for analyzing experiments with network interference, motivated by “message passing” techniques from statistical physics. Our approach models how intervention effects propagate through networks over time. We then combine this framework with machine learning methods to estimate the intervention effects without knowledge of the network structure. Unlike traditional approaches that require known network structures or focus only on equilibrium outcomes, our methodology leverages temporal data to extract information from pre-equilibrium dynamics and introduces a distribution-preserving network bootstrapping technique for validation of intervention effect estimates. Through realistic simulated environments ranging from behavioral health promotions to ride-sharing platforms, networks of interacting AI agents, and server load balancing systems, we will demonstrate how these techniques can improve the accuracy of intervention effect estimates where network effects are prevalent. These simulation environments provide known ground truth values while maintaining realistic complexities, enabling systematic comparison of our approach across varying settings.

AI assistance has the potential to improve firm decision-making and economic performance for millions of entrepreneurs across the globe. Existing evidence from field experiments suggests that AI assistance can improve entrepreneurial performance, but that many entrepreneurs struggle to effectively leverage these tools to drive firm growth. In an ongoing field experiment with a large digital platform in Southeast Asia, we present results from the randomized rollout of AI assistants to over one million small business owners across eight countries. Preliminary results highlight the key role that scaling frictions play in shaping both the use of AI by entrepreneurs and its impact on business performance.