These papers, books, and essays serve as a shared basis of knowledge for the MIT Probabilistic Computing Project.
Non-Technical Background
- Artificial Intelligence — The Revolution Hasn’t Happened Yet by Michael I. Jordan
- The Structure of Scientific Revolutions by Thomas Kuhn
- The Innovator’s Dilemma: The Revolutionary Book That Will Change the Way You Do Business by Clayton Christensen
- Epigrams on Programming by Alan Perlis (Turing award page, Foreword to SICP)
- The Effective Executive: The Definitive Guide to Getting the Right Things Done by Peter F. Drucker
- Technology and Courage by Ivan Sutherland
- Mathematical Writing by Don Knuth
- You and Your Research by Richard Hamming
- Whirlwind I Computer
- Worse Is Better by Richard Gabriel
- Who Inspired the Leamer-Rosenthal Prizes? Part II by Ed Leamer
- The Laws of Medicine: Field Notes From an Uncertain Science by Siddhartha Mukherjee (Interview)
- The War of Art: Winning the Inner Creative Battle by Steven Pressfield
- Study of Telemedicine Finds Misdiagnoses of Skin Problems by Melinda Beck
- Many Analysts, One Dataset: Making Transparent How Variations in Analytical Choices Affect Results by Silberzahn et al.
- What Is It Like To Be A Bat? by Thomas Nagel
- When U.S. air force discovered the flaw of averages by Todd Rose
- The origins of SageMath – creating a viable open source alternative to Magma, Maple, Mathematica, and Matlab by William Stein
- Politics and the English Language by George Orwell
- The Sciences of the Artificial by Herbert Simon
- Building Robust Systems: An Essay by Gerald Jay Sussman
- Statistics: What is the question? by Leek and Peng
Technical Background
- Mathematics: Its Content, Methods and Meaning: Chapter 11. The Theory of Probability by A. N. Kolmogorov
- Structure and Interpretation of Computer Programs by Harold Abelson, Gerald Jay Sussman, and Julie Sussman
- Classifier Technology and the Illusion of Progress by David J. Hand
- Machine Learning: The High-Interest Credit Card of Technical Debt by Sculley et al.
- On Computable Numbers and the Entscheidungsproblem by Alan Turing (Skim)
- Noncomputable Conditional Distributions by Ackerman, Freer, and Roy (Skim)
- Natively Probabilistic Computation; PhD Dissertation by Vikash Mansinghka
- Design of LISP-Based Processors by Steele and Sussman
- Lisp: A Language for Stratified Design by Hal Abelson and Gerlad Sussman
- Steps Toward Artificial Intelligence by Marvin Minsky
- On Chomsky and the Two Cultures of Statistical Learning by Peter Norvig
- Statistical Modeling: The Two Cultures by Leo Breiman
- Critique of Paper by “Deep Learning Conspiracy” (Nature 521 p 436) by Jürgen Schmidhuber
- Artificial Intelligence A Modern Approach (Introduction p. 3-31) by Russell and Norvig
- Confessions of a Used Programming Language Salesman by Erik Meijer
Probabilistic Programming
- VentureScript Tutorial
- BayesDB Documentation
- Picture: an imperative probabilistic programming language for scene perception by Kulkarni, Kohli, Tenenbaum, and Mansinghka
- Bayesian Statistics Without Tears: A Sampling-Resampling Perspective by Smith and Gelfand
- When are probabilistic programs probably computationally tractable? Freer, Mansinghka, and Roy
- Bayesian Optimization as a Probabilistic Meta-Program; MEng thesis by Ben Zinberg
Probabilistic Programming (Optional, Outdated)
- Venture: a general purpose probabilistic programming platform with efficient stochastic inference by Mansinghka, Selsam, and Perov
- A stochastic programming perspective on nonparametric Bayes by Roy, Mansinghka, Goodman and Tenenbaum
Foundations of Probabilistic Modeling (Required)
- Bayesianism and Causality, or, Why I am Only a Half-Bayesian by Judea Pearl
- Why I Am Not a Bayesian by Clark Glymour
- Probability Theory: The Logic of Science (Chapters 1-3) by E. T. Jaynes
- Information Theory, Inference, and Learning Algorithms (Chapter 28, Model Comparison and Occam’s Razor) by David MacKay
Standard Library of Modeling Techniques
- Information Theory, Inference, and Learning Algorithms (Chapter 45, Gaussian Processes) by David MacKay
- Unifying Rational Models of Categorization via the Hierarchical Dirichlet Process by Griffiths, Canini, Sanborn, and Navarro
- Discovering latent classes in relational data by Kemp, Griffiths, and Tenenbaum
- Topics in Semantic Representation by Griffiths and Steyvers
- Finding Scientific Topics by Griffiths and Steyvers
- Continuous Time Dynamic Topic Models by Wang, Blei, and Heckerman
- Practical Bayesian Optimization of Machine Learning Algorithms by Snoek, Larochelle, and Adams
- A Unifying Review of Linear Gaussian Models by Roweis and Ghahramani
Standard Library of Inference Techniques
- Composable Probabilistic Inference with Blaise by Keith A. Bonawitz
- An Introduction to MCMC for Machine Learning by Andrieu, De Freitas, Doucet, and Jordan
- The Markov Chain Monte Carlo Revolution by Persi Diaconis
- Convergence of Sequential Monte Carlo-based Sampling Methods by Huggins and Roy
Useful Examples (Optional)
- MrBayes: Bayesian Inference of Phylogenetic Trees by Huelsenbeck and Ronquist [pdf] [webpage]
- FastSLAM: FastSLAM: A Factored Solution to the Simultaneous Localization and Mapping Problem
- First few chapters of Bayesian Spectrum Analysis and Parameter Estimation by Bretthorst
- Recovering Intrinsic Images from a Single Image by Tappen, Freeman, and Adelson
- Bayesian Inference for PCFGs via Markov chain Monte Carlo by Johnson, Griffiths and Goldwater
- Global Seismic Monitoring as Probabilistic Inference by Arora, Russell, Kidwell, and Sudderth
Automatic Model Discovery (Optional)
- CrossCat: A Fully Bayesian Nonparametric Method for Analyzing Heterogeneous, High Dimensional Data by Mansinghka et. al.
- Structure Discovery in Nonparametric Regression through Compositional Kernel Search by Duvenaud, Lloyd, Grosse, Tenenbaum, Ghahramani
- Exploiting compositionality to explore a large space of model structures by Grosse et al
- Model Selection in Compositional Spaces; PhD Thesis by Roger Grosse
- The discovery of structural form by Kemp and Tenenbaum
- Structured priors for structure learning by Mansinghka, Kemp, Tenenbaum, and Griffiths
- Information Theory, Inference, and Learning Algorithms (Chapter 28, Model Comparison and Occam’s Razor) by David MacKay
Reference Material on Approximate Inference (Optional)
- Probabilistic Inference using Markov chain Monte Carlo Methods by Neal
- A Tutorial on Particle Filtering and Smoothing: Fifteen years later by Doucet and Johansen
- Hamiltonian Monte Carlo tutorial by Neal
- Exact and approximate inference via systematic stochastic search by Mansinghka et al.
- Factor graphs and the sum-product algorithm by Kschischang, Frey, and Loeliger
- An Architecture for Parallel Topic Models by Smola and Narayanamurthy
- Efficient BackProp by LeCun, Bottou, Orr and Muller
- Black Box Variational Inference by Ranganath, Gerrish and Blei