(Probabilistic) graphical models

Printer-friendly version

Placeholder for my notes on probabilistic graphical models.
This should be broken out into sub-categories.

Directed graphs

a.k.a PGMs, Bayesian networks, Directed graphical models.

These can even be causal graphical models, and when we can infer those we are
extracting Science (ONO) from observational data.

No refs here, I’m still reeling from a whole semester of them.

Oooh! look! Software! bnlearn

Unidirected Graphical Models

a.k.a Markov random fields, Markov random networks.

I would like to know about Poisson random fields, Markov random fields,
Bernoulli (or is it Boolean?) random fields,
esp for discrete multivariate sequences.
Gibbs and Boltzman dist inference.

  • Wasserman’s nice explanation: Estimating Undirected Graphs Under Weak Assumptions
  • Besag, J. (1974). Spatial Interaction and the Statistical Analysis of Lattice Systems. Journal of the Royal Statistical Society. Series B (Methodological), 36(2), 192–236. Online.
  • Besag, J. (1975). Statistical Analysis of Non-Lattice Data. Journal of the Royal Statistical Society. Series D (The Statistician), 24(3), 179–195. DOI. Online.
  • Blake, A., & Kohli, P. (2011). Introduction to Markov Random Fields. In Markov Random Fields for Vision and Image Processing. MIT Press. Online.
  • Clifford, P. (1990). Markov random fields in statistics. In G. R. Grimmett & D. J. A. Welsh (Eds.), Disorder in Physical Systems: A Volume in Honour of John Hammersley. Oxford England : New York: Oxford University Press.
  • Della Pietra, S., Della Pietra, V., & Lafferty, J. (1995). Inducing Features of Random Fields. arXiv:cmp-lg/9506014. Online.
  • Fridman, A. (2003). Mixed Markov models. Proceedings of the National Academy of Sciences, 100(14), 8092–8096. DOI. Online.
  • Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., & Saul, L. K.(1999). An Introduction to Variational Methods for Graphical Models. Machine Learning, 37(2), 183–233. DOI. Online.
  • Krämer, N., Schäfer, J., & Boulesteix, A.-L. (2009). Regularized estimation of large-scale gene association networks using graphical Gaussian models. BMC Bioinformatics, 10(1), 384. DOI. Online.
  • Meinshausen, N., & Bühlmann, P. (2006). High-dimensional graphs and variable selection with the lasso. The Annals of Statistics, 34(3), 1436–1462. DOI. Online.
  • Montanari, A. (2011). Lecture Notes for Stat 375 Inference in Graphical Models. Online.
  • Murphy, K. P.(2012). Undirected graphical models (Markov random fields). In Machine Learning: A Probabilistic Perspective (1 edition.). Cambridge, MA: The MIT Press. Online.
  • Rabbat, M. G., Figueiredo, M., & Nowak, R. (2006). Inferring network structure from co-occurrences. In Advances in Neural Information Processing Systems (pp. 1105–1112). MIT Press. Online.
  • Ravikumar, P., Wainwright, M. J., & Lafferty, J. D.(2010). High-dimensional Ising model selection using ℓ1-regularized logistic regression. The Annals of Statistics, 38(3), 1287–1319. DOI. Online.
  • Richardson, M., & Domingos, P. (2006). Markov logic networks. Machine Learning, 62(1-2), 107–136. Online.
  • Wang, C., Komodakis, N., & Paragios, N. (2013). Markov Random Field modeling, inference & learning in computer vision & image understanding: A survey. Computer Vision and Image Understanding, 117(11), 1610–1627. DOI. Online.
  • Wasserman, L., Kolar, M., & Rinaldo, A. (2013). Estimating Undirected Graphs Under Weak Assumptions. arXiv:1309.6933 [cs, Math, Stat]. Online.

Chain graphs

Partially directed random fields.
The classic chain graph of the 80s allows you to have cycling sets of mutually influencing varaibles, connected by directed acycling influence.
It looks a hell of a lot like the CPDAGs of Maathuis et al, and I’m curious what the relationship is.

  • Maathuis, M. H., Kalisch, M., & Bühlmann, P. (2009). Estimating high-dimensional intervention effects from observational data. The Annals of Statistics, 37(6A), 3133–3164. DOI. Online.
  • Bouckaert, R. R., & Studený, M. (1995). Chain graphs: Semantics and expressiveness. In C. Froidevaux & J. Kohlas (Eds.), Symbolic and Quantitative Approaches to Reasoning and Uncertainty (pp. 69–76). Springer Berlin Heidelberg. Online.
  • Buntine, W. L.(1996). A guide to the literature on learning probabilistic networks from data. IEEE Transactions on Knowledge and Data Engineering, 8(2), 195–210. DOI.
  • Buntine, W. L.(2013). Chain Graphs for Learning. arXiv:1302.4933 [cs]. Online.
  • Dawid, A. P., & Lauritzen, S. L.(1993). Hyper Markov Laws in the Statistical Analysis of Decomposable Graphical Models. The Annals of Statistics, 21(3), 1272–1317. DOI. Online.
  • Drton, M. (2009). Discrete chain graph models. Bernoulli, 15(3), 736–753. DOI. Online.
  • Lauritzen, S. L.(1992). Propagation of Probabilities, Means, and Variances in Mixed Graphical Association Models. Journal of the American Statistical Association, 87(420), 1098–1108. DOI. Online.
  • Lauritzen, S. L., & Richardson, T. S.(2002). Chain graph models and their causal interpretations. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64(3), 321–348. DOI. Online.
  • Lauritzen, S. L., & Wermuth, N. (1989). Graphical Models for Associations between Variables, some of which are Qualitative and some Quantitative. The Annals of Statistics, 17(1), 31–57. DOI. Online.
  • Studený, M. (1997a). A recovery algorithm for chain graphs. International Journal of Approximate Reasoning, 17(2–3), 265–293. DOI. Online.
  • Studený, M. (1997b). On Recovery Algorithm for Chain Graphs.
  • Studený, M., & Bouckaert, R. R.(1998). On chain graph models for description of conditional independence structures. The Annals of Statistics, 26(4), 1434–1495. DOI. Online.
  • Wermuth, N., & Lauritzen, S. L.(1990). On Substantive Research Hypotheses, Conditional Independence Graphs and Graphical Chain Models. Journal of the Royal Statistical Society. Series B (Methodological), 52(1), 21–50. Online.
  • Xing, E. P., Jordan, M. I., & Russell, S. (2012). A Generalized Mean Field Algorithm for Variational Inference in Exponential Families. arXiv:1212.2512 [cs, Stat]. Online.

Conditional Random Fields

Er… Markov random fields… conditonal…. on something else?

  • Altun, Y., Smola, A. J., & Hofmann, T. (2004). Exponential Families for Conditional Random Fields. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence (pp. 2–9). Arlington, Virginia, United States: AUAI Press. Online.
  • McCallum, A. (2012). Efficiently Inducing Features of Conditional Random Fields. arXiv:1212.2504 [cs, Stat]. Online.

Factor Graphs

A unifying formalism for the directed and undirected, above.
How does that work then?

  • data extractio system Deepdive uses factor graphs
  • Kschischang, F. R., Frey, B. J., & Loeliger, H.-A. (2001). Factor graphs and the sum-product algorithm. IEEE Transactions on Information Theory, 47(2), 498–519. DOI.
  • Loeliger, H.-A. (2004). An introduction to factor graphs. IEEE Signal Processing Magazine, 21(1), 28–41. DOI. Online.
  • Sutton, C., & McCallum, A. (2010). An Introduction to Conditional Random Fields. arXiv:1011.4088. Online.


  • E.Z. graphical regression with (so-called) BayesDB.

See original: The Living Thing / Notebooks (Probabilistic) graphical models