Spring 2024
Friday, April 26, 2024
Seminar in JMHH 540
Presenter: Anton Korinek – UVA Darden
Title: Scenarios for the Transition to AGI
Abstract
We analyze how output and wages behave under different scenarios for technological progress that may culminate in Artificial General Intelligence (AGI), defined as the ability of AI systems to perform all tasks that humans can perform. We assume that human work can be decomposed into atomistic tasks that differ in their complexity. Advances in technology make ever more complex tasks amenable to automation. The effects on wages depend on a race between automation and capital accumulation. If the distribution of task complexity exhibits a sufficiently thick infinite tail, then there is always enough work for humans, and wages may rise forever. By contrast, if the complexity of tasks that humans can perform is bounded and full automation is reached, then wages collapse. But declines may occur even before if large-scale automation outpaces capital accumulation and makes labor too abundant. Automating productivity growth may lead to broad-based gains in the returns to all factors. By contrast, irreproducible scarce factors may exacerbate a decline in wages.
Tuesday, April 23, 2024
Seminar in JMHH 540
Presenter: Raghav Singal – Dartmouth University
Title: Online Matching with Heterogeneous Supply and Minimum Allocation Guarantees (joint work with Garud Iyengar)
Abstract
Problem Definition. We study the problem faced by a two-sided platform to match incoming job requests to workers, with the objective of maximizing the average quality of matches while providing minimum allocation guarantees to the workers to prevent them from churning.
Methodology. We propose a model that takes into account the workers’ heterogeneous churning behavior (in addition to their quality and capacity). We provide the platform the ability to control churn by enforcing a minimum requirement on the average amount of work each worker receives per period. In each time period, the jobs arrive sequentially and are matched in an online manner. The number of jobs in each period is random and iid. The platform chooses an online matching policy to maximize the average match quality.
Results. We show policies that are blind to the workers’ quality (e.g., FCFS) or neglect the heterogeneity in their preferences (e.g., Greedy) can perform poorly. We propose an interpretable policy (Tracker). It takes as input the target allocation level of each worker and tracks these levels by allocating an incoming job to the worker with residual capacity who is most under-allocated. These targets are computed by solving a carefully constructed deterministic relaxation of the stochastic optimization problem. We also perform a simulation study calibrated to the data from a labor platform. The proposed policy dominates the two benchmarks in every scenario we tested, with gains of over 10% in many scenarios. Our work influenced the labor platform to change its matching policy, and it observed gains of a similar magnitude.
Managerial Implications. We highlight a fundamental inefficiency of matching policies that neglect worker-level preferences. By accounting for such preferences and using the target-tracking policy we devise, it is possible to increase the quality of matches while increasing workers’ satisfaction.
Friday, April 19, 2024
Seminar in JMHH 540
Presenter: Lee Branstetter – CMU Heinz
Title: Quantifying the Impact of AI on Productivity and Labor Demand: Evidence from U.S. Census Microdata
Abstract
After decades of disappointment, artificial intelligence (AI) has entered a new era of rapidly advancing capabilities that are likely to raise productivity and reshape demand for labor within and across firms and industries. Accurately measuring these effects has been difficult due to a lack of detailed, firm-level data on AI innovation. We address that challenge by using a combination of machine learning algorithms to parse the text of U.S. patent grants and assess the degree to which they are AI-related. This approach indicates that AI-related invention is far more pervasive than previous analyses have suggested. We match our data on AI patenting to U.S. Census microdata collected on the innovating firms. We then perform an event study using these matched data to gauge the impact of these innovations on firm labor demand, growth and wage dispersion. We find that AI-related innovations are positively associated with firm growth as firms with AI-related innovations have both faster employment and revenue growth growth than a comparative set of firms. We also find evidence that AI-related innovations raise output per worker and increase within-firm wage inequality.
Friday, April 12, 2024
Seminar in JMHH 540
Presenter: Jonathan Hersh – Chapman University
Title: Who Benefits When Artificial Intelligence Becomes Explainable? Evidence from a Field Experiment at a Large Bank
Abstract
As AI becomes embedded in firm processes, it is unknown how this will impact workers with complementary or substitutable skills to AI. Explainable AI (XAI) has emerged as a tool for understanding how complex AI models make certain decisions, although there is limited evidence for how they perform in the workplace. In this paper we ask whether making AI decisions more explainable results in greater trust and understanding in the machine learning model, and who benefits from XAI relative to machine learning predictions. To answer these questions, we partnered with a large bank and generated AI predictions for whether one of its loans will be delayed in its final disbursement. We embedded these predictions into a dashboard, surveying 685 workers before and after viewing the tool, where half were randomized to an XAI condition, to determine what impact if any XAI has on AI trust. We find that i) XAI is associated with greater perceived usefulness but less perceived understanding of the machine learning predictions; ii) Certain AI-reluctant groups — in particular senior managers and those with less familiarity with AI — exhibit more reluctance to trust the AI predictions overall; iii) Greater complexity of the prediction context is associated with higher degree of AI trust; and iv) XAI benefits workers with low-ML experience the most, resulting in greater trust and perceived usefulness. Overall we find limited evidence of XAI’s impact on trust overall, but evidence that it is important for individuals with limited ML experience.
Tuesday, April 9, 2024
Seminar in JMHH 540
Presenter: Elisabeth Paulson – Harvard Business School
Title: Improving Refugee Resettlement Outcomes with Optimization
Abstract
Every year, tens of thousands of refugees and asylum seekers are resettled in host countries across the world. In many host countries, newcomers are assigned to a specific locality (e.g., city) upon arrival by a resettlement agency. This assignment decision has a profound long-term impact on integration outcomes. The high-level goal of this line of work is to improve these outcomes through prediction and optimization algorithms. In particular, we will describe a new dynamic assignment algorithm to match refugees and asylum seekers to geographic localities within a host country. This algorithm—currently implemented in a multi-year pilot in Switzerland—achieves near-optimal predicted employment (and improves upon the status quo procedure by about 40%). We will then discuss recent extensions of this method that improve performance of the predict-then-optimize pipeline in the face of non-stationarity, enhance robustness, and improve fairness across demographic groups.
Tuesday, April 2, 2024
Seminar in JMHH 540
Presenter: Maxime Cohen – McGill University
Title: Incentivizing Healthy Food Choices Using Add-on Bundling: A Field Experiment
Abstract
How can retailers incentivize customers to make healthier food choices? Price, convenience, and taste are known to be among the main drivers behind such choices. Unfortunately, healthier food options are often expensive and not adequately promoted. Interestingly, we are observing recent efforts to nudge customers toward healthier food. In this paper, we conducted a field experiment with a global convenience store chain to better understand how different add-on bundle promotions influence healthy food choices. We considered three types of add-on bundles: (i) an unhealthy bundle (when customers purchased a coffee, they could add a pastry for $1), (ii) a healthy bundle (offering a healthy snack, such as fruit, vegetable, or protein, as a coffee add-on for $1), and (iii) a choice bundle (the option of either a pastry or a healthy snack as a coffee add-on for $1). In addition to our field experiment, we conducted an online lab study to strengthen the validity of our results. We found that offering healthy snacks as part of an add-on bundle significantly increased healthy purchases (and decreased unhealthy purchases). Surprisingly, this finding continued to hold for the choice bundle, that is, even when unhealthy snacks were concurrently on promotion. However, we did not observe a long-term stickiness effect, meaning that customers returned to their original (unhealthy) purchase patterns once the healthy or choice bundle was discontinued. Finally, we show that offering an add-on choice bundle is also beneficial for retailers, who can earn higher revenue and profit.
Tuesday, March 26, 2024
Seminar in JMHH 540
Presenter: Steve Chick – INSEAD
Title: Learning Personalized Treatment Strategies with Predictive and Prognostic Covariates in Adaptive Clinical Trials (work with Andres Alban and Spyros Zoumpoulis)
Abstract
Clinical trials are used to evaluate the benefit of new health technologies but can be expensive. They also possess characteristics which add nuances relative to other sequential learning applications. We consider the problem of sequentially allocating patients to arms to learn personalized treatment strategies, i.e., to learn the best treatment as a function of patient covariates. In such settings there may be clinical knowledge of which covariates are predictive (they may interact with the treatment choice) and which are prognostic (they may influence the outcome independent of treatment choice). We extend the expected value of information (EVI)/knowledge gradient framework to develop useful heuristics for a context with predictive and prognostic covariates and a delay in observing outcomes. We also propose and analyse closely related Monte Carlo-based allocation policies to enhance our proposal’s computational efficiency and applicability for adaptive contextual learning. We show that several of our proposed allocation policies are asymptotically optimal in learning treatment strategies. We run simulation experiments motivated by an application for clinical trial design to assess potential treatments of sepsis. We illustrate that the proposed EVI-based allocation policies, with knowledge about which covariates are predictive and prognostic, can improve the rate of inference relative to some existing approaches to adaptive contextual learning.
Friday, March 22nd, 2024
Seminar held in JMHH 540
Presenter: Alex Kasavin – Microsoft
Title: Practical Reflections on Artificial Intelligence and Social Impact
Abstract
Tuesday, February 20, 2024
Seminar in JMHH 540
Presenter: Tarek Abdallah – Kellogg University
Title: Dynamic Pricing in the Large Market Regime
Abstract
Dynamic pricing is a common strategy used by firms to efficiently manage their limited or perishable inventory/capacity. Successfully implementing a dynamic pricing policy requires a firm to continuously update its prices in response to its current levels of inventory and expected demand over the remaining selling horizon. Solving for the optimal dynamic pricing policy is, in general, not possible, and one common approach to solve these problems involves using heuristics that are based on scaling some of the problem parameters. The most widely used regime is the Fluid regime, in which both inventory and market size are scaled proportionally to infinity. In this talk, we propose a new scaling regime that we call the large market regime in which only the market size is scaled while the inventory is kept fixed. Unlike the Fluid regime, which leverages the law of large numbers to solve a deterministic problem, in the large market regime, the firm needs to focus on pricing from the right tail of the distribution. It turns out that the extreme value theory is well suited to this situation and plays a key role in developing our main results. Our main results provide the asymptotics of the optimal expected revenue, pricing policy, and purchasing probability policy in the large market regime. These results imply that asymptotically the optimal pricing policy of the firm is not necessarily a classical run-out rate policy. We introduce the family of generalized run-out rate policies, specific instances of which are shown to be asymptotically optimal. We then proceed to characterize the regret of any asymptotically optimal policy relative to a fluid-derived upper bound. Finally, we conduct several numerical experiments to test the accuracy and performance of our results.
Tuesday, February 13, 2024
Seminar in JMHH 540
Presenter: Chiara Farronato – Harvard Business School
Title: Platform Vertical Integration and Consumer Choice: Evidence from a Field Experiment (joint work with Andrey Fradkin and Alexander MacKay)
Abstract
Many firms, from retailers to investment management companies, offer their own products alongside products sold by competitors. This vertical integration, although common across the economy, is particularly controversial in a digital setting. In this work, we study the effects of Amazon’s vertical integration on consumer choice. Amazon offers products carrying an Amazon brand (such as Amazon Basics), in direct competition with other products. To assess what consumers would choose in the absence of Amazon brands, we run a field experiment using a custom browser extension. The extension allows us to randomize which products users see when searching for products on Amazon.com. Our key preliminary finding is that shoppers substitute towards products that are fairly comparable when Amazon brands are not available. We also find that such substitution is not affected by self-preferencing, because we fail to find any evidence that Amazon prioritizes its products in search results.
Friday, Febuary 2, 2024
Seminar in JMHH 540
Presenter: Jonathan Frankle – Databricks
Title: How to Train an LLM from Scratch
Abstract
In this talk, I will describe the process of training an LLM from scratch, starting from the fundamental design decisions that go into building a model and the cost of doing so, and concluding with the logistics of training it, fine-tuning it, and aligning it with human preferences. Databricks believes in open science, so I will be able to share fine-grained details about how we train industrial-grade LLMs.
Thursday, January 30, 2024
Seminar in JMHH 540
Presenter: Elisa Long – UCLA Anderson
Title: Digital Footprints: Leveraging Smartphone Location Data to Analyze High-Stakes Decision-Making
Abstract
Smartphone geolocation data are increasingly used by social science researchers to study human decision-making and health behaviors. I will summarize three of my recent studies related to nursing home staff networks and COVID; hurricane evacuations; and utilization of reproductive health clinics offering abortion services. I will discuss the benefits and challenges of working with individual device-level smartphone data, including implications for data privacy.
Tuesday, January 23, 2024
Seminar in JMHH 540
Presenter: Amy Ward – Chicago Booth
Title: Learning to Schedule in Multiclass Many Server Queues with Abandonment
Abstract
The multiclass many server queue with abandonment (specifically, the multiclass GI/GI/N+GI queue) is a canonical model for service systems. One key operational question is how to schedule; that is, how to choose the customer that a newly available server will serve. The scheduling question is of fundamental importance because scheduling determines which customer classes have longer wait times (relative to their patience when waiting for service), and, therefore, more abandonments. However, even though there is much work on scheduling in queueing systems, there is comparatively less work on scheduling in queueing systems when the model primitives (that is, distributional and parameter information regarding the inter-arrival, service, and patience times) are unknown and may be learned.
Our objective is to determine a scheduling policy that minimizes regret, which is the difference in expected abandonment cost between a proposed policy, that does not have knowledge of model primitives, and a benchmark policy, that has full knowledge of model primitives. The difficulty is that the state space is very complex, because in order for the system to be Markovian, we must track: (i) the time elapsed since the last arrival for each class; (ii) the amount of time each customer in service has been in service; and (iii) the amount of time each customer in queue has spent waiting. We propose a policy that first learns and then schedules following a Learn-then-Schedule (LTS) algorithm that we develop. We analyze the performance of the LTS policy against the benchmark aμ-rule (that prioritizes classes for service in accordance with their cost of abandonment times service rate). The algorithm is composed of a learning phase, during which empirical estimates of the service rates are formed, and an exploitation phase, during which an empirical aμ-rule based on those estimates is applied. We show that the LTS policy has regret of order logT (where T is the system run-time), which is the smallest order achievable.
Fall 2023
Tuesday, December 12, 2023
Seminar in JMHH 540
Presenter: Rob Bray – Kellogg University
Title: On the endogeneity of U.S. retail prices: Insights from a large-scale field experiment
Abstract
We present experimental evidence that suggest that the observational price variation found in typical supermarket scanner data is insufficient to recover true price elasticities. Our field experiment generated 372,146 random, in-store price changes over the course of 35 weeks, across 412 products at 82 “test” stores of a Midwestern supermarket chain. We compare the demand elasticity estimates derived with these experimental price changes against the elasticity estimates derived with the observational (non-experimental) price changes at 34 “control” stores from the same chain. We identify a large bias in the observational elasticities, ranging between -2 and -0.8 across specifications. We cannot alleviate this bias by controlling for price promotions, instrumenting with the chain price, focusing on a short time window around each price change, focusing on base-price changes, or accounting for longer-term price effects. Our results suggest that all meaningful variation in grocery prices is tainted by endogeneity.
Thursday, November 16, 2023
Seminar in JMHH 540
Presenter: David Holtz – Berkeley Haas
Title: Does AI Spur Entrepreneurial Performance?
Abstract
Access to high quality advice and information is an important determinant of entrepreneurial performance. Generative AI tools, such as large language models (LLMs), open up the potential for low-cost interventions that increase access to advice and information, particularly in settings where advice from human mentors or consultants might be unavailable or prohibitively expensive. In this study, we present results from a 5-month-long RCT in Kenya conducted on a sample of approximately 600 Kenyan small business entrepreneurs. Subjects in the control group were given access to a PDF guide providing advice on business practices, whereas subjects in the treatment group were given access to an AI business mentor Whatsapp chatbot powered by GPT-4. We measure the impact of access to the AI mentor on both management practices and business performance. We find that on average, access to the AI mentor does not improve business outcomes. However, this null result masks substantial heterogeneity; the AI mentor improves performance for entrepreneurs who were high performers in the pre-period but causes performance declines for pre-period low performers. Surprisingly, high and low performers are equally likely to use the AI mentor, suggesting that these performance effects are rooted in differences in the returns to AI usage for high and lower performers in our setting. These findings stand in contrast to recent RCTs showing access to LLMs in more constrained settings narrows the gap between the best and worst performers.
Nicholas Otis (UC Berkeley), Rowan Clarke (HBS), Solene Delecourt (UC Berkeley), David Holtz (UC Berkeley), Rembrand Koning (HBS)
Tuesday, November 14, 2023
Seminar in JMHH 540
Presenter: Alessandro Arlotto – Duke University
Title: Ballot design and relative party order: evidence from North Carolina’s judicial elections
Abstract
In North Carolina, Justices of the State Supreme Court and Judges of the State Court of Appeals are elected through state-wide elections. Political parties are often actively involved in judicial campaigns, despite the fact the law considers judicial elections as non-partisan. In a non-partisan election, the party affiliation of each judicial candidate is generally not listed on the ballot and the order in which candidates appear on the ballot is (partially) randomized. This randomization can result in a party-order of judicial candidates that may be the same or the opposite relative to the party-order in partisan elections, such as the election of the President and the Vice President of the United States. Based on precinct-level election and demographic data, we estimate the (heterogeneous) average treatment effect of flipping the party-order of judicial candidates (relative to the party-order of partisan elections) in the absence of candidates’ party affiliations. We also find that the effect goes away when the party affiliation of judicial candidates is listed on the ballot. (Joint work with Alexandre Belloni, Duke University; Fei Fang, Yale University; and Sasa Pekec, Duke University.)
Tuesday, October 3, 2023
Seminar in JMHH 540
Presenter: Santiago Balseiro – Columbia University
Title: Robust Online Allocation with Dual Mirror Descent
Abstract
Online allocation problems with resource constraints are central problems in revenue management and online advertising. In these problems, requests arrive sequentially during a finite horizon, and for each request, a decision maker needs to choose an action that consumes a certain amount of resources and generates a reward. The objective is to maximize cumulative rewards subject to a constraint on the total consumption of resources. In practice, the distributions of reward and resource consumption are unknown, and there is a need to design data-driven algorithms that can make decisions in uncertain environments.
In this talk, we overview a new algorithm for online allocation problems, called dual mirror descent, that is simple, robust, and flexible. Compared to existing approaches, dual mirror descent is faster as it does not require solving auxiliary optimization problems, is more flexible because it can handle many applications across different sectors with minimal modifications, and is more robust as it enjoys remarkable performance under different environments without knowing which environment it is facing. In particular, dual mirror descent is asymptotically optimal under independent and identically distributed inputs as well as various nonstationary stochastic input models, and it attains an asymptotically optimal fixed competitive ratio when the input is adversarial. We discuss several extensions and applications to network revenue management, online bidding in repeated auctions with budget constraints, personalized assortment optimization with limited inventory, and fair allocation of resources.
The talk is based on the following papers:
https://pubsonline.informs.org/doi/abs/10.1287/opre.2021.2242
https://proceedings.mlr.press/v119/balseiro20a.html
https://proceedings.mlr.press/v139/balseiro21a.html
https://proceedings.mlr.press/v202/balseiro23a.html
https://dl.acm.org/doi/abs/10.1145/3578338.3593559
Tuesday, September 26, 2023
Seminar in JMHH 540
Presenter: Auyon Siddiq – UCLA Anderson
Title: Platform Disintermediation: Information Effects and Pricing Remedies
Abstract
Two-sided platforms typically generate revenue by matching prospective buyers and sellers and extracting commissions from completed transactions. Disintermediation, where sellers transact offline with buyers to bypass commission fees, can undermine the viability of these platforms. While transacting offline allows sellers to avoid commission fees, it also leaves them fully exposed to risky buyers given the absence of the platform’s protections. In this paper, we examine how disintermediation and information quality – specifically, the accuracy of the signal sellers receive about a buyer’s type – jointly impact a platform’s revenue and optimal commission rate. In a setting where transactions occur online-only, an increase in information quality leads sellers to set more efficient prices and complete more transactions, which lifts platform revenue. However, if sellers can transact offline, this effect may be reversed – additional information about buyers can hurt platform revenue by reducing the risk sellers face offline, amplifying disintermediation. Further, while intuition suggests platforms should counter disintermediation by lowering commission rates, in a high-information environment a platform may be better off raising them. Lastly, while charging sellers platform-access fees can hedge against losses from disintermediation, it can fall short of the revenue attained under commission-based pricing when all transactions occur online. Overall, our findings provide insight into the mechanisms through which disintermediation disrupts platform operations and offers prescriptions to platforms seeking to counteract it.
Tuesday, September 21, 2023
Seminar held in JMHH 260
Presenter: Iavor Bojinov – Harvard Business School
Title: Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation
Abstract
Randomized experiments have become the standard method for companies to evaluate the performance of new products or services. In addition to augmenting managers’ decision-making, experimentation mitigates risk by limiting the proportion of customers exposed to innovation. Since many experiments are on customers arriving sequentially, a potential solution is to allow managers to “peek” at the results when new data becomes available and stop the test if the results are statistically significant. Unfortunately, peeking invalidates the statistical guarantees for standard statistical analysis and leads to uncontrolled type-1 error. Our paper provides valid design-based confidence sequences, sequences of confidence intervals with uniform type-1 error guarantees over time for various sequential experiments in an assumption-light manner. In particular, we focus on finite-sample estimands defined on the study participants as a direct measure of the incurred risks by companies. Our proposed confidence sequences are valid for a large class of experiments, including multi-arm bandits, time series, and panel experiments. We further provide a variance reduction technique incorporating modeling assumptions and covariates. Finally, we demonstrate the effectiveness of our proposed approach through a simulation study and three real-world applications from Netflix. Our results show that by using our confidence sequence, harmful experiments could be stopped after only observing a handful of units; for instance, an experiment that Netflix ran on its sign-up page on 30,000 potential customers would have been stopped by our method on the first day before 100 observations.
Tuesday, September 19, 2023
Seminar in JMHH 540
Presenter: Andrew Davis – Cornell University
Title: A Replication Study of Operations Management Experiments in Management Science
Abstract
Over the last two decades, researchers in operations management have increasingly leveraged laboratory experiments to identify key behavioral insights. These experiments inform behavioral theories of operations management, impacting domains including inventory, supply chain management, queuing, forecasting, and sourcing. Yet, until now, the replicability of most behavioral insights from these laboratory experiments has been untested. We remedy this with the first large-scale replication study in operations management. With the input of the wider operations management community, we identify 10 prominent experimental operations management papers published in Management Science, which span a variety of domains, to be the focus of our replication effort. For each paper, we conduct a high-powered replication study of the main results across multiple locations using original materials (when available and suitable). In addition, our study tests replicability in multiple modalities (in-person and online) due to laboratory closures during the COVID-19 pandemic. Our replication study contributes new knowledge about the robustness of several key behavioral theories in operations management and contributes more broadly to efforts in the operations management field to improve research transparency and reliability.