Schedule
Fall 2026 · Tuesday & Thursday, 2:00–3:20 pm · POS 153. Assignments are submitted on Canvas.
This page is the source for session descriptions, readings, and slides. Click any topic in the table to jump to its details.
At a glance
| Week | Date | Topic | Project milestone |
|---|---|---|---|
| 1 | Tue Aug 25 | Class overview | |
| 1 | Thu Aug 27 | Why ML systems can fail in practice | Team creation (Aug 28) |
| 2 | Tue Sep 1 | Scoping ML systems | Project idea (Aug 31) |
| 2 | Thu Sep 3 | Scoping - Applied to Class Projects | |
| 3 | Tue Sep 8 | Analytical formulation and baselines | |
| 3 | Thu Sep 10 | Data exploration | Proposal & scope (Sep 14) |
| 4 | Tue Sep 15 | Project pitches | 3-min pitch due (2pm) |
| 4 | Thu Sep 17 | Model selection (evaluation) | |
| 5 | Tue Sep 22 | Model performance metrics | |
| 5 | Thu Sep 24 | Feature engineering | |
| 6 | Tue Sep 29 | ML modeling & hyperparameter tuning | |
| 6 | Thu Oct 1 | ML pipelines | Baseline(s) due |
| 7 | Tue Oct 6 | Project update presentations | Update presentation due (2pm) |
| 7 | Thu Oct 8 | Project update presentations | |
| 8 | Oct 13 / 15 | No class — Fall Break | |
| 9 | Tue Oct 20 | ML ethics | Initial ML solution due |
| 9 | Thu Oct 22 | Building systems with foundation models | |
| 10 | Tue Oct 27 | Building systems with AI agents | |
| 10 | Thu Oct 29 | Evaluation in the field | |
| 11 | Tue Nov 3 | No class — Democracy Day | |
| 11 | Thu Nov 5 | Evaluating GenAI systems | Evaluation due |
| 12 | Tue Nov 10 | Causality | |
| 12 | Thu Nov 12 | Distribution shift and robustness | |
| 13 | Tue Nov 17 | Uncertainty quantification | |
| 13 | Thu Nov 19 | Interpretability | Iteration 2 due |
| 14 | Tue Nov 24 | Fairness | |
| 14 | Thu Nov 26 | No class — Thanksgiving Break | |
| 15 | Tue Dec 1 | Project presentations | Project presentation due |
| 15 | Thu Dec 3 | Project presentations | |
| — | Tue Dec 8 | Project writeup & demo due |
Module 1 · Building ML Systems
Building end-to-end ML systems for real problems — scoping, formulation, validation, metrics, features, and pipelines.
Class overview
Tue Aug 25. Overview of the course
Why ML systems can fail in practice
Thu Aug 27. We’ll discuss real-world failure modes of ML systems, moving beyond model accuracy to system-level issues including data, deployment, governance, incentives, etc. The goal here is to encourage critical thinking about preventing failures and to motivate the topics to be covered during the rest of the semester.
GenAI in practice: Are there LLM-specific failure modes?
Case Studies for Breakout Groups: Case Studies
Project: team creation due Aug 28.
Scoping ML systems
Tue Sep 1. We’ll discuss scoping, problem definition, and understanding and balancing organizational goals. Before we start doing technical ML work, a decision needs to be made about whether a given problem can and should be addressed with machine learning. What outcomes is the system supposed to achieve? How is success measured? What actions and decisions does the system enable? What ethical discussions need to happen?
GenAI in practice: Does scoping change for GenAI systems?
Required Readings: Data Science Project Scoping Guide
Optional Readings: Fine-grained dengue forecasting using telephone triage services (Rehman et al., Sci. Adv. 2016); Deconstructing Statistical Questions (Hand, J. Royal Stat. Soc. A, 1994);
Project: submit project ideas (to get feedback) due Aug 31 on canvas.
Scoping ML systems (applied to class projects)
Thu Sep 3. Students will apply scoping methodology presented in the previous session to their class project ideas, critique, and iterate.
[Slides]https://docs.google.com/presentation/d/1nhrvkptOIaiNPCiGaF0bstGIVwqyRpYHtHpdUj_foIY/)
Analytical formulation and baselines
Tue Sep 8. Turning the scope into an ML problem. What design decisions need to be made? Defining the label, the available features, and the baseline we need to compare against — which should reflect the status quo or easy-to-implement approaches, rarely “random.”
GenAI in practice: an LLM zero/few-shot prompt is an example of an easy-to-implement baseline.
Required:
Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations (Obermeyer et al., Science 2019);
Measuring the predictability of life outcomes with a scientific mass collaboration (Salganik et al PNAS 2022)
Optional:
Problem Formulation and Fairness (Passi & Barocas, FAT* 2019).
Always Start with a Stupid Model, No Exceptions (Ameisen);
Create a Common-Sense Baseline First (Ramakrishnan);
Data Science for Business (Provost & Fawcett, ch. 2).
Data (Acquisition, Stoerage, Linkage, and exploration) For ML
Thu Sep 10. Data exploration in the context of ML systems. How does data exploration support making modeling decisions? What phases of an ML project require doing data exploration?
GenAI in practice: LLMs for wrangling, parsing, and extracting structure from messy/unstructured data.
Slides Optional: Can Foundation Models Wrangle Your Data? (Narayan et al., VLDB 2023). Project: project proposal and scope due Mon Sep 14.
Project pitches
Tue Sep 15. 3-minute team pitches: the problem, why it matters, who the decision-maker is, and what data exists.
Project: 3-minute pitch due in class (2:00pm).
Model selection (evaluation)
Thu Sep 17. Validation strategy: how it relates to how you want the model to generalize during deployment/test time, and why k-fold often fails in practice.
Slides: TBD Required: Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure (Roberts et al., Ecography 2017). Optional: Amazing Things Come From Having Many Good Models; Time Series Nested Cross-Validation (Cochrane); The Secrets of Machine Learning (Rudin & Carlson, 2019); Big Data and Social Science (Foster, Ghani et al., §7.7); Evaluating and Comparing Classifiers (Stapor, CORES 2017).
Model performance metrics
Tue Sep 22. How do we back into the metric we care about? Metrics that match the decision and the deployment setting: precision@k / top-k under capacity constraints, calibration, and why we may need multiple metrics.
GenAI in practice: What metrics do we need for GenAI systems?
Slides: TBD Required: The Misuse of AUC: What High-Impact Risk Assessment Gets Wrong (2023). Optional: Transductive Optimization of Top-k Precision (Liu et al., IJCAI 2016).
Feature engineering
Thu Sep 24. Tabular data still requires serious feature engineering to express domain knowledge through features. How do we create such features? What are common templates? How do we handle outliers and missingness without leaking?
GenAI in practice: LLMs for feature extraction, labeling / weak supervision, and embeddings.
Slides: TBD Optional: Missing Data Conundrum (Akinfaderin); Feature Engineering for Machine Learning (Zhang & Casari, ch. 2); Missing-data imputation (Gelman). Optional (data infrastructure): Netflix’s Trillions Scale Real-time Data Infrastructure; Real-time Data Infrastructure at Uber.
ML modeling in practice & hyperparameter tuning
Tue Sep 29. Where to start on a real problem: what models to explore, what is a reasonable hyperparameter grid, why optuna is not always the right answer, common pitfalls, and how to avoid them.
Slides: TBD Required: Three Pitfalls to Avoid in Machine Learning (Riley, Nature 2019); Top 10 ways your Machine Learning models may have leakage (Ghani et al., DSSG blog).
ML pipelines
Thu Oct 1. Modular, config-driven pipelines vs. monolithic notebooks.
GenAI in practice: RAG and agent systems are pipelines — prompts, retrieval, and tools are components you can test and swap.
Slides: TBD Project: Implement Baseline(s) due tonight (11:59pm).
Project update presentations
Tue Oct 6 & Thu Oct 8. Progress, formulation, baselines, and early results; structured peer feedback.
Project: update presentation due in class (2:00pm).
Fall Break
Tue Oct 13 & Thu Oct 15. No class.
Module 2 · Beyond the Basic Model and Accuracy
Additional topics beyond building an accuracy-focused model: the modern ML systems you’ll build (foundation models and agents) and how to evaluate them, as well as considerations around ethics, causality, field evaluation, robustness, uncertainty, interpretability, and fairness — applied to the project.
ML ethics
Tue Oct 20. What ethical issues show up in the context of ML systems, across the entire lifecycle? How do we embed these discussions throughout the lifecycle, and what can we do at each stage that allows us to manage and reduce the downstream risks?
Slides: TBD Required: Ethics and Data Science (Loukides, Mason & Patil, O’Reilly 2018 — short) Optional: Princeton AI Ethics Case Study 6: Public Sector Data Analysis. Project: Initial ML Solution due tonight (11:59pm).
Building systems with foundation models
Thu Oct 22. The foundation-model paradigm: build from scratch vs. use off-the-shelf vs. fine-tune vs. prompt; retrieval and grounding (RAG).
Slides: TBD
Building systems with AI agents
Tue Oct 27. When multi-step / agentic designs (tool use, planning) help, best practices, tools, and the reliability of agentic systems.
Slides: TBD
Evaluation in the field
Thu Oct 29. Does the system actually improve outcomes? RCTs and quasi-experimental designs.
GenAI in practice: the offline→online gap — a system that looks good on an eval set can still fail on the real decision it supports.
Slides: TBD Required: A Guide to Running Randomized Controlled Trials (IGL). Recommended case: Breaking the Cycle of Incarceration With Targeted Mental Health Outreach (Rodolfa, Salomon, … Ghani, 2025). Optional: The Econometrics of Randomized Experiments (Athey & Imbens).
Evaluating GenAI systems
Thu Nov 5. Evaluating generative systems where there may be no ground truth to compare against: task-specific eval sets, human evaluation, LLM-as-judge, etc.
Slides: TBD
Causality
Tue Nov 10. The intersection between ML and causal inference. The tools of causal inference and how to use them.
Slides: TBD Required: The Seven Tools of Causal Inference, with Reflections on Machine Learning (Pearl, CACM 2019). Optional: Elements of Causal Inference (Peters et al., MIT Press — ch. 1–2).
Distribution shift and robustness
Thu Nov 12. Why models degrade over time and off-distribution; detecting and handling shift.
GenAI in practice: prompt injection (incl. indirect), jailbreaks, and quiet drift as underlying models change.
Slides: TBD
Uncertainty quantification
Tue Nov 17. Distinguishing types of uncertainty; estimating, interpreting, and communicating them to decision-makers.
GenAI in practice: calibration and abstention.
Slides: TBD
Interpretability
Thu Nov 19. Global vs. local explanation; matching the explanation to the user and use case.
GenAI in practice: tracing and grounding an answer to its sources, rather than feature-importance-style explanation.
Slides: TBD Required: Explainable Machine Learning for Public Policy: Use Cases, Gaps, and Research Directions (Amarasinghe et al., 2020); Benchmarking and Survey of Explanation Methods for Black Box Models (Bodria et al., 2021). Project: Iteration 2 due Thu Nov 19 — add one component from Module 2.
Fairness
Tue Nov 24. Definitions of fairness and their incompatibilities; where bias enters the pipeline; equity auditing.
GenAI in practice: bias evaluations for LLMs and how the generative case differs from the predictive one.
Slides: TBD Required: Fairness Definitions Explained (Verma & Rubin); revisit Obermeyer et al. (2019). Optional: A Theory of Justice (Rawls, 1971 — ch. 1, pp. 1–19); Racial Equity in Algorithmic Criminal Justice (Huq, Duke Law Journal 2018); Is Algorithmic Affirmative Action Legal? (Bent, Georgetown Law Journal 2019); Does Mitigating ML’s Impact Disparity Require Treatment Disparity? (Lipton et al., NeurIPS 2018); Equality of Opportunity (Roemer & Trannoy, 2013).
Project presentations
Tue Dec 1 & Thu Dec 3. The full arc: problem, approach, evaluation, honest limitations, and recommendations.
Project: Project Presentation due Tue Dec 1; Project Writeup and Demo due Tue Dec 8.