Lyle Ungar
- Associate Professor of Computer and Information Science, Professor of Chemical and Biomolecular Engineering
- Professor of Electrical and Systems Engineering, Professor of Operations, Information and Decisions, Professor of Genomics and Computational Biology
Contact Information
- Primary Email:
ungar@wharton.upenn.edu - Office Phone:
(215) 898-7449
- office Address:
200 South 33rd Street
504 Levine Hall
Philadelphia, PA 19104
Research Interests: information economics, statistical relational learning text mining active learning market-based methods for distributed scheduling and optimization of systems including human and computer agents, and gene and protein expression. clustering and collaborative filtering feature selection genomics, and regulatory network modeling information extraction in biological texts, consumer purchases, machine learning and data mining, on data such as scientific papers, proteomics, shopbots and pricebots auctions and mechanism designs computational biolology and (someday) computer go.
Links: Personal Website
Research
Katherine L. Milkman, Linnea Gandhi, Mitesh Patel, Heather N. Graci, Dena Gromet, Hung Ho, Joseph S. Kay, Timothy W. Lee, Jake Rothschild, Jonathan E. Bogard, Ilana Brody, Christopher F. Chabris, Edward Chang, Gretchen B. Chapman, Jennifer E. Dannals, Noah J. Goldstein, Amir Goren, Hal E. Hershfield, Alexander Hirsch, Jillian Hmurovic, Samantha Horn, Dean Karlan, Ariella Kristal, Cait Lamberton, M. Meyer, Allison H. Oakes, Maurice Schweitzer, Maheen Shermohammed, Joachim H. Talloen, Caleb Warren, Ashley Whillans, Kuldeep N. Yadav, Julian J. Zlatev, Ron Berman, Chalanda N. Evans, Rahul Ladhania, Jens Ludwig, Nina Mazar, Sendhil Mullainathan, Christopher K. Snider, Jann Spiess, Eli Tsukayama, Lyle Ungar, Christophe Van den Bulte, Kevin Volpp, Angela Duckworth (2022), A 680,000-Person Megastudy of Nudges to Encourage Vaccination in Pharmacies, Proceedings of the National Academy of Sciences, 119 (6). 10.1073/pnas.211512611
Abstract: Encouraging vaccination is a pressing policy problem. To assess whether text-based reminders can encourage pharmacy vaccination and what kinds of messages work best, we conducted a megastudy. We randomly assigned 689,693 Walmart pharmacy patients to receive one of 22 different text reminders using a variety of different behavioral science principles to nudge flu vaccination or to a business-as-usual control condition that received no messages. We found that the reminder texts that we tested increased pharmacy vaccination rates by an average of 2.0 percentage points, or 6.8%, over a 3-mo follow-up period. The most effective messages reminded patients that a flu shot was waiting for them and delivered reminders on multiple days. The top performing intervention included two texts delivered 3 d apart and communicated to patients that a vaccine was “waiting for you.” Neither experts nor lay people anticipated that this would be the best-performing treatment, underscoring the value of simultaneously testing many different nudges in a highly powered megastudy.
Katherine L. Milkman, Dena Gromet, Hung Ho, Joseph S. Kay, Timothy W. Lee, Predrag Pandiloski, Yeji Park, Aneesh Rai, Max Bazerman, John Beshears, Lauri Bonacorsi, Colin Camerer, Edward Chang, Gretchen B. Chapman, Robert Cialdini, Hengchen Dai, Lauren Eskreis-Winkler, Ayelet Fishbach, James J. Gross, Samantha Horn, Alexa Hubbard, Steven J. Jones, Dean Karlan, Tim Kautz, Erika Kirgios, Joowon Klusowski, Ariella Kristal, Rahul Ladhania, George Loewenstein, Jens Ludwig, Barbara Mellers, Sendhil Mullainathan, Silvia Saccardo, Jann Spiess, Gaurav Suri, Joachim H. Talloen, Jamie Taxer, Yaacov Trope, Lyle Ungar, Kevin Volpp, Ashley Whillans, Jonathan Zinman, Angela Duckworth (2021), Megastudies Improve the Impact of Applied Behavioural Science, , 600 (), pp. 478-483.
Abstract: Policy-makers are increasingly turning to behavioural science for insights about how to improve citizens’ decisions and outcomes. Typically, different scientists test different intervention ideas in different samples using different outcomes over different time intervals. The lack of comparability of such individual investigations limits their potential to inform policy. Here, to address this limitation and accelerate the pace of discovery, we introduce the megastudy—a massive field experiment in which the effects of many different interventions are compared in the same population on the same objectively measured outcome for the same duration. In a megastudy targeting physical exercise among 61,293 members of an American fitness chain, 30 scientists from 15 different US universities worked in small independent teams to design a total of 54 different four-week digital programmes (or interventions) encouraging exercise. We show that 45% of these interventions significantly increased weekly gym visits by 9% to 27%; the top-performing intervention offered microrewards for returning to the gym after a missed workout. Only 8% of interventions induced behaviour change that was significant and measurable after the four-week intervention. Conditioning on the 45% of interventions that increased exercise during the intervention, we detected carry-over effects that were proportionally similar to those measured in previous research. Forecasts by impartial judges failed to predict which interventions would be most effective, underscoring the value of testing many ideas at once and, therefore, the potential for megastudies to improve the evidentiary value of behavioural science.
Pavel Atanasov, Lyle Ungar, Barbara Mellers, Philip Tetlock (2020), Small steps to accuracy: Incremental belief updaters are better forecasters, Organizational Behavior and Human Decision Processes.
Gilmer Valdes, Albert J. Chang, Yannet Interian, Kenton Owens, Shane T. Jensen, Lyle Ungar, Adam Cunha, Timothy D. Solberg, I-Chow Hsu (2018), Salvage HDR Brachytherapy: Multiple Hypothesis Testing Versus Machine Learning Analysis, International Journal of Radiation Oncology - Biology - Physics, 101 (3), pp. 694-703.
Barbara Mellers, Eric Stone, Terry Murray, Angela Minster, Nick Rohrbaugh, Michael Bishop, Eva Chen, Joshua Baker, Yuan Hou, Michael Horowitz, Lyle Ungar, Philip Tetlock (2015), Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions, Perspectives on Psychological Science.
Barbara Mellers, Eric Stone, Pavel Atanasov, Nick Rohrbaugh, S. E. Metz, Lyle Ungar, Michael Bishop, Michael Horowitz (2015), The Psychology of Intelligence Analysis: Drivers of Prediction Accuracy in World Politics, Journal of Experimental Psychology: Applied.
Jonathan Baron, Barbara Mellers, Philip Tetlock, Eric Stone, Lyle Ungar (2015), Two Reasons to Make Aggregated Probability Forecasts More Extreme, Decision Analysis.
Barbara Mellers, Lyle Ungar, Jonathan Baron, Jaime Ramos, Burcu Gurcay, Katrina Fincher, Sydney Scott, Don Moore, Pavel Atanasov, Samuel Swift, Terry Murray, Eric Stone, Philip Tetlock (2014), Psychological Strategies for Winning a Geopolitical Forecasting Tournament, Psychological Science.
Barbara Mellers, Lyle Ungar, Jonathan Baron, Jaime Ramos, Burcu Gurcay, Katrina Fincher, Sydney Scott, Don Moore, Pavel Atanasov, Samuel Swift, Terry Murray, Eric Stone, Philip Tetlock (2014), Psychological Strategies for Winning Geopolitical Forecasting Tournaments, Psychological Science.
Shay Cohen, Karl Stratos, Michael Collins, Dean P. Foster, Lyle Ungar (2013), Experiments with Spectral Learning of Latent-Variable PCFGs, NAACL-HLT 2013.
Teaching
Current Courses (Spring 2024)
CIS6200 - Adv Top In Mach Learning
This course covers a variety of advanced topics in machine learning, such as the following: statistical learning theory (statistical consistency properties of surrogate loss minimizing algorithms); approximate inference in probabilistic graphical models (variational inference methods and sampling-based inference methods); structured prediction (algorithms and theory for supervised learning problems involving complex/structured labels); and online learning in complex/structured domains. The precise topics covered may vary from year to year based on student interest and developments in the field.
CIS6200001
CIS8990 - Phd Independent Study
For doctoral students studying a specific advanced subject area in computer and information science. The Independent Study may involve coursework, presentations, and formally gradable work comparable to that in a CIS 5000 or 6000 level course. The Independent Study may also be used by doctoral students to explore research options with faculty, prior to determining a thesis topic. Students should discuss with the faculty supervisor the scope of the Independent Study, expectations, work involved, etc. The Independent Study should not be used for ongoing research towards a thesis, for which the CIS 9990 designation should be used.
CIS8990042
CIS9990 - Thesis/diss Research
For students pursuing advanced research to fulfill PhD dissertation requirements.
CIS9990042
All Courses
AMCS5999 - Independent Study
Independent Study allows students to pursue academic interests not available in regularly offered courses. Students must consult with their academic advisor to formulate a project directly related to the student’s research interests. All independent study courses are subject to the approval of the AMCS Graduate Group Chair.
AMCS9999 - Ind Study & Research
Study under the direction of a faculty member.
CIS0099 - Ugrad Resrch/Ind Study
An opportunity for the student to become closely associated with a professor (1) in a research effort to develop research skills and techniques and/or (2) to develop a program of independent in-depth study in a subject area in which the professor and student have a common interest. The challenge of the task undertaken must be consistent with the student's academic level. To register for this course, the student must submit a detailed proposal, signed by the independent study supervisor, to the SEAS Office of Academic Programs (111 Towne) no later than the end of the "add" period. Prerequisite: A maximum of 2 c.u. of CIS 0099 may be applied toward the B.A.S. or B.S.E. degree requirements.
CIS1400 - Intro Cognitive Science
How do minds work? This course surveys a wide range of answers to this question from disciplines ranging from philosophy to neuroscience. The course devotes special attention to the use of simple computational and mathematical models. Topics include perception, learning, memory, decision making, emotion and consciousness. The course shows how the different views from the parent disciplines interact and identifies some common themes among the theories that have been proposed. The course pays particular attention to the distinctive role of computation in such theories and provides an introduction to some of the main directions of current research in the field. It is a requirement for the BA in Cognitive Science, the BAS in Computer and Cognitive Science, and the minor in Cognitive Science, and it is recommended for students taking the dual degree in Computer and Cognitive Science.
CIS5200 - Machine Learning
This course covers the foundations of statistical machine learning. The focus is on probabilistic and statistical methods for prediction and clustering in high dimensions. Topics covered include linear and logistic regression, SVMs, PCA and dimensionality reduction, EM and HMMs, and deep learning. Elementary probability, calculus, and linear algebra. Basic programming experience.
CIS5220 - Deep Learning/Data Sci
Deep learning techniques now touch on data systems of all varieties. Sometimes, deep learning is a product; sometimes, deep learning optimizes a pipeline; sometimes, deep learning provides critical insights; sometimes, deep learning sheds light on neuroscience or vice versa. The purpose of this course is to deconstruct the hype by teaching deep learning theories, models, skills, and applications that are useful for applications.
CIS5450 - Big Data Analytics
In the new era of big data, we are increasingly faced with the challenges of processing vast volumes of data. Given the limits of individual machines (compute power, memory, bandwidth), increasingly the solution is to process the data in parallel on many machines. This course focuses on the fundamentals of scaling computation to handle common data analytics tasks. You will learn about basic tasks in collecting, wrangling, and structuring data; programming models for performing certain kinds of computation in a scalable way across many compute nodes; common approaches to converting algorithms to such programming models; standard toolkits for data analysis consisting of a wide variety of primitives; and popular distributed frameworks for analytics tasks such as filtering, graph analysis, clustering, and classification. Recommended: broad familiarity with probability and statistics, as well as programming in Python. Additional background in statistics, data analysis (e.g., in Matlab or R), and machine learning is helpful (example : ESE 5420).
CIS5970 - Master's Thesis Research
For students working on an advanced research leading to the completion of a Master's thesis.
CIS5990 - Master's Indep Study
For master's students studying a specific advanced subject area in computer and information science. Involves coursework and class presentations. A CIS 5990 course unit will invariably include formally gradable work comparable to that in a CIS 500-level course. Students should discuss with the faculty supervisor the scope of the Independent Study, expectations, work involved, etc.
CIS6200 - Adv Top in Mach Learning
This course covers a variety of advanced topics in machine learning, such as the following: statistical learning theory (statistical consistency properties of surrogate loss minimizing algorithms); approximate inference in probabilistic graphical models (variational inference methods and sampling-based inference methods); structured prediction (algorithms and theory for supervised learning problems involving complex/structured labels); and online learning in complex/structured domains. The precise topics covered may vary from year to year based on student interest and developments in the field.
CIS7000 - Cis-Topics
One time course offerings of special interest. Equivalent to a CIS 5XX level course.
CIS8950 - Teaching Practicum
Enrollment for students participating in Teaching Practicum.
CIS8990 - PhD Independent Study
For doctoral students studying a specific advanced subject area in computer and information science. The Independent Study may involve coursework, presentations, and formally gradable work comparable to that in a CIS 5000 or 6000 level course. The Independent Study may also be used by doctoral students to explore research options with faculty, prior to determining a thesis topic. Students should discuss with the faculty supervisor the scope of the Independent Study, expectations, work involved, etc. The Independent Study should not be used for ongoing research towards a thesis, for which the CIS 9990 designation should be used.
CIS9950 - Dissertation
For Ph.D. candidates working exclusively on their dissertation research, having completed enrollment for a total of ten semesters (fall and spring). There is no credit or grade for CIS 9950.
CIS9990 - Thesis/Diss Research
For students pursuing advanced research to fulfill PhD dissertation requirements.
COGS1001 - Intro Cognitive Science
How do minds work? This course surveys a wide range of answers to this question from disciplines ranging from philosophy to neuroscience. The course devotes special attention to the use of simple computational and mathematical models. Topics include perception, learning, memory, decision making, emotion and consciousness. The course shows how the different views from the parent disciplines interact and identifies some common themes among the theories that have been proposed. The course pays particular attention to the distinctive role of computation in such theories and provides an introduction to some of the main directions of current research in the field. It is a requirement for the BA in Cognitive Science, the BAS in Computer and Cognitive Science, and the minor in Cognitive Science, and it is recommended for students taking the dual degree in Computer and Cognitive Science.
COGS3998 - Senior Thesis
This course is a directed study intended for cognitive science majors who have been admitted to the cognitive science honors program. Upon admission into the program, students may register for this course under the direction of their thesis supervisor.
COGS3999 - Independent Study
Departmental permission required
EAS8970 - Master's CPT
Master's Student Curricular Practical Training (CPT) credit for Summer terms. Graduate students in Engineering who meet the USCIS eligibility criteria may apply for academic credit for the purposes of F-1 curricular practical training (CPT). In order to be eligible for CPT, students must have already completed one academic year (September to May) of course work, full-time at Penn, but have not completed all of their degree requirements. https://grad.seas.upenn.edu/student-handbook/academic-options/curricular-practical-training/
EAS8980 - PhD CPT
PhD Student Curricular Practical Training (CPT) credit. Graduate students in Engineering who meet the USCIS eligibility criteria may apply for academic credit for the purposes of F-1 curricular practical training (CPT). In order to be eligible for CPT, students must have already completed one academic year (September to May) of course work, full-time at Penn, but have not completed all of their degree requirements. https://grad.seas.upenn.edu/student-handbook/academic-options/curricular-practical-training/
ESE8990 - PhD Independent Study
For students who are studying a specific advanced subject area in electrical engineering. Students must submit a proposal outlining and detailing the study area, along with the faculty supervisor's consent, to the graduate group chair for approval. A maximum of 1 c.u. of ESE 8990 may be applied toward the MSE degree requirements. A maximum of 2 c.u.'s of ESE 8990 may be applied toward the Ph.D. degree requirements.
ESE9990 - Thesis/Diss Res
For students working on an advanced research program leading to the completion of master's thesis or Ph.D. dissertation requirements.
GCB9950 - Dissertation
Ph.D. students enroll in this course after passing their candidacy exam. They work on their dissertation full-time under the guidance of their dissertation supervisor and other members of their dissertation committee.
LING1005 - Intro Cognitive Science
How do minds work? This course surveys a wide range of answers to this question from disciplines ranging from philosophy to neuroscience. The course devotes special attention to the use of simple computational and mathematical models. Topics include perception, learning, memory, decision making, emotion and consciousness. The course shows how the different views from the parent disciplines interact and identifies some common themes among the theories that have been proposed. The course pays particular attention to the distinctive role of computation in such theories and provides an introduction to some of the main directions of current research in the field. It is a requirement for the BA in Cognitive Science, the BAS in Computer and Cognitive Science, and the minor in Cognitive Science, and it is recommended for students taking the dual degree in Computer and Cognitive Science.
PHIL1840 - Intro Cognitive Science
How do minds work? This course surveys a wide range of answers to this question from disciplines ranging from philosophy to neuroscience. The course devotes special attention to the use of simple computational and mathematical models. Topics include perception, learning, memory, decision making, emotion and consciousness. The course shows how the different views from the parent disciplines interact and identifies some common themes among the theories that have been proposed. The course pays particular attention to the distinctive role of computation in such theories and provides an introduction to some of the main directions of current research in the field. It is a requirement for the BA in Cognitive Science, the BAS in Computer and Cognitive Science, and the minor in Cognitive Science, and it is recommended for students taking the dual degree in Computer and Cognitive Science.
PSYC1333 - Intro Cognitive Science
How do minds work? This course surveys a wide range of answers to this question from disciplines ranging from philosophy to neuroscience. The course devotes special attention to the use of simple computational and mathematical models. Topics include perception, learning, memory, decision making, emotion and consciousness. The course shows how the different views from the parent disciplines interact and identifies some common themes among the theories that have been proposed. The course pays particular attention to the distinctive role of computation in such theories and provides an introduction to some of the main directions of current research in the field. It is a requirement for the BA in Cognitive Science, the BAS in Computer and Cognitive Science, and the minor in Cognitive Science, and it is recommended for students taking the dual degree in Computer and Cognitive Science.
PSYC4998 - Mentored Research
Mentored research involving data collection. Students do independent empirical work under the supervision of a faculty member, leading to a written paper. Normally taken in the junior or senior year.
PSYC4999 - Honors Mentored Research
The Honors Program has been developed to recognize excellence in psychology among Penn undergraduates and to enhance skills related to psychological research. The 4998 credit signifies an Honors Independent Study, completed as part of the Honors Program. The honors program involves: (a) completing a year-long empirical research project in your senior year under the supervision of a faculty member (for a letter grade). This earns 2 cu's. (b) completing a second term of statistics (for a letter grade) before graduation. (c) participating in the year-long Senior Honors seminar (for a letter grade). This seminar is designed especially for Psychology Honors majors; this receives a total of 1 cu. (d) participating in the Undergraduate Psychology Research Fair in the Spring semester, at which honors students present a poster and give a 15-minute talk about their research. (e) a total of 15 cu's in psychology is required. Students will be selected to be part of the Honors Program in the Spring of their junior year (see application process online)
STAT9950 - Dissertation
In the News
Knowledge at Wharton
- Everything You Have Read About Contact Tracing Apps Is Wrong, Knowledge at Wharton - 6/30/2020
Activity
Latest Research
In the News
Everything You Have Read About Contact Tracing Apps Is WrongContact tracing is critical to reopening society. But are the apps for that any good? Penn's Lyle Ungar answers that question and more in this opinion piece. …Read More
Knowledge at Wharton - 6/30/2020