Reading List

These papers, books, and essays serve as a shared basis of knowledge for the MIT Probabilistic Computing Project.

Non-Technical Background

  • Artificial Intelligence — The Revolution Hasn’t Happened Yet by Michael I. Jordan [essay]
  • The Structure of Scientific Revolutions by Thomas Kuhn [Amazon]
  • The Innovator’s Dilemma: The Revolutionary Book That Will Change the Way You Do Business by Clayton Christensen [Amazon]
  • Epigrams on Programming by Alan Perlis [html] [Turing award page][Foreword to SICP]
  • The Effective Executive: The Definitive Guide to Getting the Right Things Done by Peter F. Drucker [Amazon]
  • Technology and Courage by Ivan Sutherland [pdf]
  • Mathematical Writing by Don Knuth [pdf]
  • You and Your Research by Richard Hamming [html]
  • Whirlwind I Computer [Wikipedia]
  • Worse Is Better by Richard Gabriel [html]
  • Who Inspired the Leamer-Rosenthal Prizes? Part II by Ed Leamer [html]
  • The Laws of Medicine: Field Notes From an Uncertain Science by Siddhartha Mukherjee [Interview][Amazon]
  • The War of Art: Winning the Inner Creative Battle by Steven Pressfield [Amazon]
  • Study of Telemedicine Finds Misdiagnoses of Skin Problems by Melinda Beck [html]
  • Many Analysts, One Dataset: Making Transparent How Variations in Analytical Choices Affect Results by Silberzahn et al. [pdf]
  • What Is It Like To Be A Bat? by Thomas Nagel [pdf]
  • When U.S. air force discovered the flaw of averages [html]
  • The origins of SageMath – creating a viable open source alternative to Magma, Maple, Mathematica, and Matlab [pdf]
  • Politics and the English Language by George Orwell [html]
  • The Sciences of the Artificial by Herbert Simon [Amazon]
  • Building Robust Systems an essay by Gerald Jay Sussman [pdf]

Technical Background

  • Mathematics: Its Content, Methods and Meaning: Chapter 11. The Theory of Probability by A. N. Kolmogorov [Amazon]
  • Structure and Interpretation of Computer Programs by Harold Abelson, Gerald Jay Sussman, and Julie Sussman [Amazon] [pdf]
  • Classifier Technology and the Illusion of Progress by David J. Hand [pdf]
  • Machine Learning: The High-Interest Credit Card of Technical Debt by Sculley et al. [pdf]
  • (Skim) On Computable Numbers and the Entscheidungsproblem by Alan Turing [pdf]
  • (Skim) Noncomputable Conditional Distributions by Ackerman, Freer, and Roy [pdf]
  • Natively Probabilistic Computation; PhD Dissertation by Vikash Mansinghka [pdf]
  • Design of LISP-Based Processors by Steele and Sussman [pdf]
  • Lisp: A Language for Stratified Design by Abelson and Sussman [pdf]
  • Steps Toward Artificial Intelligence by Minsky [pdf]
  • On Chomsky and the Two Cultures of Statistical Learning by Peter Norvig [html]
  • Statistical Modeling: The Two Cultures by Leo Breiman [pdf]
  • Critique of Paper by “Deep Learning Conspiracy” (Nature 521 p 436) by Jürgen Schmidhuber [html]
  • Artificial Intelligence A Modern Approach (Introduction p. 3-31) by Russell and Norvig [pdf]
  • Confessions of a Used Programming Language Salesman by Erik Meijer [pdf]

Probabilistic Programming

  • VentureScript Tutorial [html]
  • BayesDB Documentation [html]
  • Picture: an imperative probabilistic programming language for scene perception by Kulkarni, Kohli, Tenenbaum, and Mansinghka, [webpage]43
  • Bayesian Statistics Without Tears: A Sampling-Resampling Perspective by Smith and Gelfand. [pdf]
  • When are probabilistic programs probably computationally tractable? Freer, Mansinghka, and Roy [pdf]
  • Bayesian Optimization as a Probabilistic Meta-Program; MEng thesis by Ben Zinberg [pdf]

Probabilistic Programming (Optional, Outdated)

  • Venture: a general purpose probabilistic programming platform with efficient stochastic inference. Mansinghka, Selsam, and Perov [pdf]
  • A stochastic programming perspective on nonparametric Bayes by Roy, Mansinghka, Goodman and Tenenbaum [pdf]

Foundations of Probabilistic Modeling (Required)

  • Bayesianism and Causality, or, Why I am Only a Half-Bayesian by Judea Pearl [pdf]
  • Why I Am Not a Bayesian by Clark Glymour [pdf]
  • Probability Theory: The Logic of Science (Chapters 1-3) by E. T. Jaynes [pdf] [Amazon]
  • Information Theory, Inference, and Learning Algorithms (Chapter 28, Model Comparison and Occam’s Razor) by David MacKay [webpage]

Standard Library of Modeling Techniques

  • Information Theory, Inference, and Learning Algorithms (Chapter 45, Gaussian Processes) by David MacKay [webpage]
  • Unifying Rational Models of Categorization via the Hierarchical Dirichlet Process by Griffiths, Canini, Sanborn, and Navarro [pdf]
  • Discovering latent classes in relational data by Kemp, Griffiths, and Tenenbaum [pdf]
  • Topics in Semantic Representation by Griffiths and Steyvers [pdf]
  • Finding Scientific Topics by Griffiths and Steyvers [pdf]
  • Continuous Time Dynamic Topic Models by Wang, Blei, and Heckerman [pdf]
  • Practical Bayesian Optimization of Machine Learning Algorithms by Snoek, Larochelle, and Adams [webpage]
  • A Unifying Review of Linear Gaussian Models by Roweis and Ghahramani [pdf]

Standard Library of Inference Techniques

  • Composable Probabilistic Inference with Blaise by Keith A. Bonawitz. [pdf]
  • An Introduction to MCMC for Machine Learning by Andrieu, De Freitas, Doucet, and Jordan [pdf]
  • The Markov Chain Monte Carlo Revolution by Persi Diaconis [pdf]
  • Convergence of Sequential Monte Carlo-based Sampling Methods by Huggins and Roy [pdf]

Useful Examples (Optional)

  • MrBayes: Bayesian Inference of Phylogenetic Trees by Huelsenbeck and Ronquist [pdf] [webpage]
  • FastSLAM: FastSLAM: A Factored Solution to the Simultaneous Localization and Mapping Problem [pdf]
  • First few chapters of Bayesian Spectrum Analysis and Parameter Estimation by Bretthorst [pdf]
  • Recovering Intrinsic Images from a Single Image by Tappen, Freeman, and Adelson [pdf]
  • Bayesian Inference for PCFGs via Markov chain Monte Carlo by Johnson, Griffiths and Goldwater [pdf]
  • Global Seismic Monitoring as Probabilistic Inference by Arora, Russell, Kidwell, and Sudderth [pdf]

Automatic Model Discovery (Optional)

  • CrossCat: A Fully Bayesian Nonparametric Method for Analyzing Heterogeneous, High Dimensional Data by Mansinghka et. al.
  • Structure Discovery in Nonparametric Regression through Compositional Kernel Search by Duvenaud, Lloyd, Grosse, Tenenbaum, Ghahramani [pdf]
  • Exploiting compositionality to explore a large space of model structures by Grosse et al [pdf]
  • Model Selection in Compositional Spaces; PhD Thesis by Roger Grosse [pdf]
  • The discovery of structural form by Kemp and Tenenbaum [pdf]
  • Structured priors for structure learning by Mansinghka, Kemp, Tenenbaum, and Griffiths [pdf]
  • Information Theory, Inference, and Learning Algorithms (Chapter 28, Model Comparison and Occam’s Razor) by David MacKay [webpage]

Reference Material on Approximate Inference (Optional)

  • Probabilistic Inference using Markov chain Monte Carlo Methods by Neal [pdf]
  • Hamiltonian Monte Carlo tutorial by Neal [pdf]
  • Exact and approximate inference via systematic stochastic search by Mansinghka et al. [pdf]
  • Factor graphs and the sum-product algorithm by Kschischang, Frey, and Loeliger [pdf]
  • An Architecture for Parallel Topic Models by Smola and Narayanamurthy [pdf]
  • LeCun, Bottou, Orr and Muller: Efficient BackProp, in Orr, and Muller (Eds), Neural Networks: Tricks of the trade [pdf]
  • Black Box Variational Inference by Ranganath, Gerrish and Blei [pdf]

Selecting Actions (Optional)

  • Towards common-sense reasoning via conditional simulation: legacies of Turing in Artificial Intelligence by Freer, Roy, Tenenbaum [pdf]
  • Rational Metareasoning and Compilation for Optimizing Decisions Under Bounded Resources by Eric Horvitz [pdf]