The Mechanics of Bayesian Inference

Bayes’ Theorem

We can decompose Bayes’ Theorem into three principal terms: \[\begin{eqnarray*} p(\theta|x) & \qquad& \text{posterior}\\ p(x|\theta) & \qquad& \text{likelihood}\\ p(\theta) & \qquad& \text{prior} \end{eqnarray*}\]

\[\mbox{Posterior}= \mbox{Likelihood}\frac{\mbox{Proposition prior probability}}{\mbox{Evidence prior probability}}\]

library(Rgraphviz)
DAG.matrix = matrix(c(0, 1, 0, 1, 0, 0, 0), ncol = 6, nrow = 6)
rownames(DAG.matrix) = c("A", "B", "C", "X", "Y", "Z")
colnames(DAG.matrix) = c("A", "B", "C", "X", "Y", "Z")
DAG.matrix
  A B C X Y Z
A 0 0 0 0 1 0
B 1 0 0 0 0 1
C 0 1 0 0 0 0
X 1 0 1 0 0 0
Y 0 1 0 1 0 0
Z 0 0 1 0 1 0
am.graph = new("graphAM", adjMat = DAG.matrix, edgemode = "directed")

\[P(A,B,C,X,Y,Z) = P(A|B)*P(B)\]

Joint Density

  • Joint Density = Conditional Density*Marginal Density \[P(A,B) = P(A|B)*P(B)\]

Bayesian Net: Independency

  • Each variable is conditionally independent of all its non-descendants in the graph given the value of all its parents.

\[P(x_{1}, x_{2},...x_{N} )\prod_{i=1}^N P(x_{i}|parent(x_{i})) \]

  • Three Types of Independency Strucutre:

D-Separation

If a causal relationship between variables X and Y is said to be d-separated for a set of given variables A, then X and Y are conditionally independent given the set A. The meaning of d-separated is analogue to directional-unconnected.

Rule 1: Unconditional separation

  • sourced from http://bayes.cs.ucla.edu/BOOK-2K/d-sep.html
  • A and B are d-connected if there is an unblocked path between them
  • “path”: refer to any consecutive sequence of edges,
  • “unblocked path”: refer to a path without traversing a pair of arrows that collide “head-to-head”; otherwise it is called “collider.”
library(Rgraphviz)
nodes <- c("a", "b", "c", "d", "e", "f", "g", "h", "m", "n", "o", "p", "q", "r",
    "s", "t", "z")

# edges
edgeList <- list(a = list(edges = c("d")), b = list(edges = c("n")), c = list(edges = c("b")),
    d = list(edges = c("g")), e = list(edges = c("n")), f = list(edges = c("c")),
    g = list(edges = c("s")), h = list(edges = c("e")), m = list(edges = c("z")),
    n = list(edges = c("b", "r")), o = list(edges = c("h")), p = list(edges = c("m")),
    q = list(edges = c("n", "r")), r = list(edges = c("f")), s = list(edges = c("z")),
    t = list(edges = c("o")), z = list(edges = c("t")))

graph01 <- new("graphNEL", nodes = nodes, edgeL = edgeList, edgemode = "directed")

d-connected:

  • a->d->g->s is unblocked, hence a and s are d-connected.
  • a->d->g is unblocked, hence a and g are d-connected.
  • a->d is unblocked, hence a and d are d-connected.

Not d-connected: blocked by a collider

  • a->d->g->s<-z<-m<-p is blocked by z (z is a collider), hence a and p are not d-connected
  • therefore, a and m are not d-connected
  • a and z are not d-connected
  • d and p are not d-connected
  • d and m are not d-connected
  • d and z are not d-connected
  • g and z are not d-connected

Conditionally d-separated

  • blocked by a collider and/or a conditional variable
  • assume circle represents marginal density while square represents conditional density
  • a->d->g->s is conditionally blocked (conditionally d-separated) by g
  • h->o->t is conditionally blocked conditionally d-separated by o
  • m->z->s: z a collider and conditionally d-separated
  • t->z->s: z a collider and conditionally d-separated
  • t->z->m: z a collider and conditionally d-separated
  • let V be the set {g,z} of conditional distributed, then a->d->g->s->z<-m<-p is d-separated by V