Seminars / Conferences

Seminar Time & Location Information 

Time and Location: 12:00PM – 1:20PM

Jon M Huntsman Hall (JMHH)

3730 Walnut St.

Philadelphia, PA 19104

Suite 540/541 (unless otherwise noted)

To schedule a meeting with a speaker contact brocd@wharton.upenn.edu

Seminars 2023-2024

Fall 2023

Tuesday, September 19, 2023

Seminar in JMHH 540

Presenter: Andrew Davis – Cornell University

Title: A Replication Study of Operations Management Experiments in Management Science

Abstract

Over the last two decades, researchers in operations management have increasingly leveraged laboratory experiments to identify key behavioral insights. These experiments inform behavioral theories of operations management, impacting domains including inventory, supply chain management, queuing, forecasting, and sourcing. Yet, until now, the replicability of most behavioral insights from these laboratory experiments has been untested. We remedy this with the first large-scale replication study in operations management. With the input of the wider operations management community, we identify 10 prominent experimental operations management papers published in Management Science, which span a variety of domains, to be the focus of our replication effort. For each paper, we conduct a high-powered replication study of the main results across multiple locations using original materials (when available and suitable). In addition, our study tests replicability in multiple modalities (in-person and online) due to laboratory closures during the COVID-19 pandemic. Our replication study contributes new knowledge about the robustness of several key behavioral theories in operations management and contributes more broadly to efforts in the operations management field to improve research transparency and reliability.

Tuesday, September 21, 2023

Seminar held in JMHH 260

Presenter: Iavor Bojinov – Harvard Business School

Title: Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation

Abstract

Randomized experiments have become the standard method for companies to evaluate the performance of new products or services. In addition to augmenting managers’ decision-making, experimentation mitigates risk by limiting the proportion of customers exposed to innovation. Since many experiments are on customers arriving sequentially, a potential solution is to allow managers to “peek” at the results when new data becomes available and stop the test if the results are statistically significant. Unfortunately, peeking invalidates the statistical guarantees for standard statistical analysis and leads to uncontrolled type-1 error. Our paper provides valid design-based confidence sequences, sequences of confidence intervals with uniform type-1 error guarantees over time for various sequential experiments in an assumption-light manner. In particular, we focus on finite-sample estimands defined on the study participants as a direct measure of the incurred risks by companies. Our proposed confidence sequences are valid for a large class of experiments, including multi-arm bandits, time series, and panel experiments. We further provide a variance reduction technique incorporating modeling assumptions and covariates. Finally, we demonstrate the effectiveness of our proposed approach through a simulation study and three real-world applications from Netflix. Our results show that by using our confidence sequence, harmful experiments could be stopped after only observing a handful of units; for instance, an experiment that Netflix ran on its sign-up page on 30,000 potential customers would have been stopped by our method on the first day before 100 observations.

Tuesday, September 26, 2023

Seminar in JMHH 540

Presenter: Auyon Siddiq – UCLA Anderson

Title: Platform Disintermediation: Information Effects and Pricing Remedies

Abstract

Two-sided platforms typically generate revenue by matching prospective buyers and sellers and extracting commissions from completed transactions. Disintermediation, where sellers transact offline with buyers to bypass commission fees, can undermine the viability of these platforms. While transacting offline allows sellers to avoid commission fees, it also leaves them fully exposed to risky buyers given the absence of the platform’s protections. In this paper, we examine how disintermediation and information quality – specifically, the accuracy of the signal sellers receive about a buyer’s type – jointly impact a platform’s revenue and optimal commission rate. In a setting where transactions occur online-only, an increase in information quality leads sellers to set more efficient prices and complete more transactions, which lifts platform revenue. However, if sellers can transact offline, this effect may be reversed – additional information about buyers can hurt platform revenue by reducing the risk sellers face offline, amplifying disintermediation. Further, while intuition suggests platforms should counter disintermediation by lowering commission rates, in a high-information environment a platform may be better off raising them. Lastly, while charging sellers platform-access fees can hedge against losses from disintermediation, it can fall short of the revenue attained under commission-based pricing when all transactions occur online. Overall, our findings provide insight into the mechanisms through which disintermediation disrupts platform operations and offers prescriptions to platforms seeking to counteract it.

Tuesday, October 3, 2023

Seminar in JMHH 540

Presenter: Santiago Balseiro – Columbia University

Title: Robust Online Allocation with Dual Mirror Descent

Abstract

Online allocation problems with resource constraints are central problems in revenue management and online advertising. In these problems, requests arrive sequentially during a finite horizon, and for each request, a decision maker needs to choose an action that consumes a certain amount of resources and generates a reward. The objective is to maximize cumulative rewards subject to a constraint on the total consumption of resources. In practice, the distributions of reward and resource consumption are unknown, and there is a need to design data-driven algorithms that can make decisions in uncertain environments.

 

In this talk, we overview a new algorithm for online allocation problems, called dual mirror descent, that is simple, robust, and flexible. Compared to existing approaches, dual mirror descent is faster as it does not require solving auxiliary optimization problems, is more flexible because it can handle many applications across different sectors with minimal modifications, and is more robust as it enjoys remarkable performance under different environments without knowing which environment it is facing. In particular, dual mirror descent is asymptotically optimal under independent and identically distributed inputs as well as various nonstationary stochastic input models, and it attains an asymptotically optimal fixed competitive ratio when the input is adversarial. We discuss several extensions and applications to network revenue management, online bidding in repeated auctions with budget constraints, personalized assortment optimization with limited inventory, and fair allocation of resources.

 

The talk is based on the following papers:

https://pubsonline.informs.org/doi/abs/10.1287/opre.2021.2242

https://proceedings.mlr.press/v119/balseiro20a.html

https://proceedings.mlr.press/v139/balseiro21a.html

https://proceedings.mlr.press/v202/balseiro23a.html

https://dl.acm.org/doi/abs/10.1145/3578338.3593559

Tuesday, November 14, 2023

Seminar in JMHH 540

Presenter: Alessandro Arlotto – Duke University

Title: Ballot design and relative party order: evidence from North Carolina’s judicial elections

Abstract

In North Carolina, Justices of the State Supreme Court and Judges of the State Court of Appeals are elected through state-wide elections. Political parties are often actively involved in judicial campaigns, despite the fact the law considers judicial elections as non-partisan. In a non-partisan election, the party affiliation of each judicial candidate is generally not listed on the ballot and the order in which candidates appear on the ballot is (partially) randomized. This randomization can result in a party-order of judicial candidates that may be the same or the opposite relative to the party-order in partisan elections, such as the election of the President and the Vice President of the United States. Based on precinct-level election and demographic data, we estimate the (heterogeneous) average treatment effect of flipping the party-order of judicial candidates (relative to the party-order of partisan elections) in the absence of candidates’ party affiliations. We also find that the effect goes away when the party affiliation of judicial candidates is listed on the ballot. (Joint work with Alexandre Belloni, Duke University; Fei Fang, Yale University; and Sasa Pekec, Duke University.)

Thursday, November 16, 2023

Seminar in JMHH 540

Presenter: David Holtz – Berkeley Haas

Title: Does AI Spur Entrepreneurial Performance?

Abstract

Access to high quality advice and information is an important determinant of entrepreneurial performance. Generative AI tools, such as large language models (LLMs), open up the potential for low-cost interventions that increase access to advice and information, particularly in settings where advice from human mentors or consultants might be unavailable or prohibitively expensive. In this study, we present results from a 5-month-long RCT in Kenya conducted on a sample of approximately 600 Kenyan small business entrepreneurs. Subjects in the control group were given access to a PDF guide providing advice on business practices, whereas subjects in the treatment group were given access to an AI business mentor Whatsapp chatbot powered by GPT-4. We measure the impact of access to the AI mentor on both management practices and business performance. We find that on average, access to the AI mentor does not improve business outcomes. However, this null result masks substantial heterogeneity; the AI mentor improves performance for entrepreneurs who were high performers in the pre-period but causes performance declines for pre-period low performers. Surprisingly, high and low performers are equally likely to use the AI mentor, suggesting that these performance effects are rooted in differences in the returns to AI usage for high and lower performers in our setting. These findings stand in contrast to recent RCTs showing access to LLMs in more constrained settings narrows the gap between the best and worst performers.

Nicholas Otis (UC Berkeley), Rowan Clarke (HBS), Solene Delecourt (UC Berkeley), David Holtz (UC Berkeley), Rembrand Koning (HBS)

Tuesday, December 12, 2023

Seminar in JMHH 540

Presenter: Rob Bray – Kellogg University

Title: On the endogeneity of U.S. retail prices: Insights from a large-scale field experiment

Abstract

We present experimental evidence that suggest that the observational price variation found in typical supermarket scanner data is insufficient to recover true price elasticities. Our field experiment generated 372,146 random, in-store price changes over the course of 35 weeks, across 412 products at 82 “test” stores of a Midwestern supermarket chain. We compare the demand elasticity estimates derived with these experimental price changes against the elasticity estimates derived with the observational (non-experimental) price changes at 34 “control” stores from the same chain. We identify a large bias in the observational elasticities, ranging between -2 and -0.8 across specifications. We cannot alleviate this bias by controlling for price promotions, instrumenting with the chain price, focusing on a short time window around each price change, focusing on base-price changes, or accounting for longer-term price effects. Our results suggest that all meaningful variation in grocery prices is tainted by endogeneity.

Spring 2024

Tuesday, January 23, 2024

Seminar held in JMHH 540

Presenter: Amy Ward – Chicago Booth

Title: Learning to Schedule in Multiclass Many Server Queues with Abandonment

Abstract

The multiclass many server queue with abandonment (specifically, the multiclass GI/GI/N+GI queue) is a canonical model for service systems. One key operational question is how to schedule; that is, how to choose the customer that a newly available server will serve. The scheduling question is of fundamental importance because scheduling determines which customer classes have longer wait times (relative to their patience when waiting for service), and, therefore, more abandonments. However, even though there is much work on scheduling in queueing systems, there is comparatively less work on scheduling in queueing systems when the model primitives (that is, distributional and parameter information regarding the inter-arrival, service, and patience times) are unknown and may be learned.

Our objective is to determine a scheduling policy that minimizes regret, which is the difference in expected abandonment cost between a proposed policy, that does not have knowledge of model primitives, and a benchmark policy, that has full knowledge of model primitives. The difficulty is that the state space is very complex, because in order for the system to be Markovian, we must track: (i) the time elapsed since the last arrival for each class; (ii) the amount of time each customer in service has been in service; and (iii) the amount of time each customer in queue has spent waiting. We propose a policy that first learns and then schedules following a Learn-then-Schedule (LTS) algorithm that we develop. We analyze the performance of the LTS policy against the benchmark aμ-rule (that prioritizes classes for service in accordance with their cost of abandonment times service rate). The algorithm is composed of a learning phase, during which empirical estimates of the service rates are formed, and an exploitation phase, during which an empirical aμ-rule based on those estimates is applied. We show that the LTS policy has regret of order logT (where T is the system run-time), which is the smallest order achievable.

Tuesday, January 30, 2024

Seminar held in JMHH 540

Presenter: Elisa Long – UCLA Anderson

Title: Digital Footprints: Leveraging Smartphone Location Data to Analyze High-Stakes Decision-Making

Abstract

Smartphone geolocation data are increasingly used by social science researchers to study human decision-making and health behaviors. I will summarize three of my recent studies related to nursing home staff networks and COVID; hurricane evacuations; and utilization of reproductive health clinics offering abortion services. I will discuss the benefits and challenges of working with individual device-level smartphone data, including implications for data privacy.

Friday, Febuary 2, 2024

Seminar held in JMHH 540

Presenter: Jonathan Frankle – Databricks

Title: How to Train an LLM from Scratch

Abstract

In this talk, I will describe the process of training an LLM from scratch, starting from the fundamental design decisions that go into building a model and the cost of doing so, and concluding with the logistics of training it, fine-tuning it, and aligning it with human preferences. Databricks believes in open science, so I will be able to share fine-grained details about how we train industrial-grade LLMs.

Tuesday, February 13, 2024

Seminar held in JMHH 540

Presenter: Chiara Farronato – Harvard Business School

Title: Platform Vertical Integration and Consumer Choice: Evidence from a Field Experiment (joint work with Andrey Fradkin and Alexander MacKay)

Abstract

Many firms, from retailers to investment management companies, offer their own products alongside products sold by competitors. This vertical integration, although common across the economy, is particularly controversial in a digital setting. In this work, we study the effects of Amazon’s vertical integration on consumer choice. Amazon offers products carrying an Amazon brand (such as Amazon Basics), in direct competition with other products. To assess what consumers would choose in the absence of Amazon brands, we run a field experiment using a custom browser extension. The extension allows us to randomize which products users see when searching for products on Amazon.com. Our key preliminary finding is that shoppers substitute towards products that are fairly comparable when Amazon brands are not available. We also find that such substitution is not affected by self-preferencing, because we fail to find any evidence that Amazon prioritizes its products in search results.

Tuesday, February 20, 2024

Seminar held in JMHH 540

Presenter: Tarek Abdallah – Kellogg University

Title: Dynamic Pricing in the Large Market Regime

Abstract

Dynamic pricing is a common strategy used by firms to efficiently manage their limited or perishable inventory/capacity. Successfully implementing a dynamic pricing policy requires a firm to continuously update its prices in response to its current levels of inventory and expected demand over the remaining selling horizon. Solving for the optimal dynamic pricing policy is, in general, not possible, and one common approach to solve these problems involves using heuristics that are based on scaling some of the problem parameters. The most widely used regime is the Fluid regime, in which both inventory and market size are scaled proportionally to infinity. In this talk, we propose a new scaling regime that we call the large market regime in which only the market size is scaled while the inventory is kept fixed. Unlike the Fluid regime, which leverages the law of large numbers to solve a deterministic problem, in the large market regime, the firm needs to focus on pricing from the right tail of the distribution. It turns out that the extreme value theory is well suited to this situation and plays a key role in developing our main results. Our main results provide the asymptotics of the optimal expected revenue, pricing policy, and purchasing probability policy in the large market regime. These results imply that asymptotically the optimal pricing policy of the firm is not necessarily a classical run-out rate policy. We introduce the family of generalized run-out rate policies, specific instances of which are shown to be asymptotically optimal. We then proceed to characterize the regret of any asymptotically optimal policy relative to a fluid-derived upper bound. Finally, we conduct several numerical experiments to test the accuracy and performance of our results.

Friday, March 22nd, 2024

Seminar location TBD

Presenter: Alex Kasavin – Microsoft

Title: Practical Reflections on Artificial Intelligence and Social Impact

Abstract

How can nonprofits and public sector agencies ensure that the benefits of adopting artificial intelligence outweigh the risks to the populations they serve? How can private sector enterprises differentiate themselves through AI without being distracted by it? How can we as individuals maintain our sense of agency, purpose, and focus – letting the waves of innovation propel us forward without drowning us?
Join us for a discussion grounded in nearly a decade of work at the intersection of AI and social impact. The talk will include first-hand accounts of AI initiatives that have succeeded and failed, the factors that contributed to those outcomes, reflections on what we have gained (and lost) in the overwhelming focus on generative AI, and suggestions on how universities and social impact organizations can partner effectively to shape the future of how these technologies evolve. No technical background is necessary, just a sense of curiosity!

Tuesday, March 26, 2024

Seminar held in JMHH 540

Presenter: Steve Chick – INSEAD

Title: Learning Personalized Treatment Strategies with Predictive and Prognostic Covariates in Adaptive Clinical Trials (work with Andres Alban and Spyros Zoumpoulis)

Abstract

Clinical trials are used to evaluate the benefit of new health technologies but can be expensive. They also possess characteristics which add nuances relative to other sequential learning applications. We consider the problem of sequentially allocating patients to arms to learn personalized treatment strategies, i.e., to learn the best treatment as a function of patient covariates. In such settings there may be clinical knowledge of which covariates are predictive (they may interact with the treatment choice) and which are prognostic (they may influence the outcome independent of treatment choice). We extend the expected value of information (EVI)/knowledge gradient framework to develop useful heuristics for a context with predictive and prognostic covariates and a delay in observing outcomes. We also propose and analyse closely related Monte Carlo-based allocation policies to enhance our proposal’s computational efficiency and applicability for adaptive contextual learning. We show that several of our proposed allocation policies are asymptotically optimal in learning treatment strategies. We run simulation experiments motivated by an application for clinical trial design to assess potential treatments of sepsis. We illustrate that the proposed EVI-based allocation policies, with knowledge about which covariates are predictive and prognostic, can improve the rate of inference relative to some existing approaches to adaptive contextual learning.

Friday, March 29, 2024

Seminar held in JMHH 540

Presenter: Angela Fan – Meta

Abstract

Forthcoming

Tuesday, April 2, 2024

Seminar held in JMHH 540

Presenter: Maxime Cohen – McGill University

Abstract

Forthcoming

Tuesday, April 9, 2024

Seminar held in JMHH 540

Presenter: Elisabeth Paulson – Harvard Business School

Title: Improving Refugee Resettlement Outcomes with Optimization

Abstract

Every year, tens of thousands of refugees and asylum seekers are resettled in host countries across the world. In many host countries, newcomers are assigned to a specific locality (e.g., city) upon arrival by a resettlement agency. This assignment decision has a profound long-term impact on integration outcomes. The high-level goal of this line of work is to improve these outcomes through prediction and optimization algorithms. In particular, we will describe a new dynamic assignment algorithm to match refugees and asylum seekers to geographic localities within a host country. This algorithm—currently implemented in a multi-year pilot in Switzerland—achieves near-optimal predicted employment (and improves upon the status quo procedure by about 40%). We will then discuss recent extensions of this method that improve performance of the predict-then-optimize pipeline in the face of non-stationarity, enhance robustness, and improve fairness across demographic groups.

Friday, April 12, 2024

Seminar held in JMHH 540

Presenter: Jonathan Hersh – Chapman University

Abstract

Forthcoming

Friday, April 19, 2024

Seminar held in JMHH 540

Presenter: Lee Branstetter – CMU Heinz

Abstract

Forthcoming

Tuesday, April 23, 2024

Seminar held in JMHH 540

Presenter: Raghav Singal – Dartmouth University

Title: Online Matching with Heterogeneous Supply and Minimum Allocation Guarantees (joint work with Garud Iyengar)

Abstract

Problem Definition. We study the problem faced by a two-sided platform to match incoming job requests to workers, with the objective of maximizing the average quality of matches while providing minimum allocation guarantees to the workers to prevent them from churning.

Methodology. We propose a model that takes into account the workers’ heterogeneous churning behavior (in addition to their quality and capacity). We provide the platform the ability to control churn by enforcing a minimum requirement on the average amount of work each worker receives per period. In each time period, the jobs arrive sequentially and are matched in an online manner. The number of jobs in each period is random and iid. The platform chooses an online matching policy to maximize the average match quality.

Results. We show policies that are blind to the workers’ quality (e.g., FCFS) or neglect the heterogeneity in their preferences (e.g., Greedy) can perform poorly. We propose an interpretable policy (Tracker). It takes as input the target allocation level of each worker and tracks these levels by allocating an incoming job to the worker with residual capacity who is most under-allocated. These targets are computed by solving a carefully constructed deterministic relaxation of the stochastic optimization problem. We also perform a simulation study calibrated to the data from a labor platform. The proposed policy dominates the two benchmarks in every scenario we tested, with gains of over 10% in many scenarios. Our work influenced the labor platform to change its matching policy, and it observed gains of a similar magnitude.

Managerial Implications. We highlight a fundamental inefficiency of matching policies that neglect worker-level preferences. By accounting for such preferences and using the target-tracking policy we devise, it is possible to increase the quality of matches while increasing workers’ satisfaction.

Friday, April 26, 2024

Seminar held in JMHH 540

Presenter: Anton Korinek – UVA Darden

Abstract

Conferences

October 12-13, 2023

Information

Since 2006, the Workshop for Empirical Research in Operations Management brings together a community of scholars with a passion for empirical research in Operations. The purpose of the Workshop is to exchange research ideas, share experiences in the publication process, discuss methodological issues, and grow together as a group of colleagues with a common research interest.