Gaussian Models

Jake

06/12/2022

Graphical Models with Continuous Variables

  • Everything from discrete probabilities holds for continuous distributions

  • However cannot use tables, have to use probability distribution functions

Gaussian Bayesian Networks

  • All variables are continuous and modelled by Gaussian Densities
    • Often good approximation for many real-world distributions
  • Root nodes use univariate distributions
  • Conditional Probability Distributions (CPDs) are commonly represented with linear Gaussian models
    • Variance is independent of parents

  • Matrix notation:

\[ P(X|u)=\mathcal{N}(\beta_0+\beta^\top u,\sigma^2_X)\]

  • Learn parameters \(\mu,\sigma^2\) from data
    • Once we have parameters, we can sample from distribution
  • Multivariate Gaussian Distribution:

  • Can identify independence assumptions directly from the Gaussian distribution parameters
    • X_i, X_j are independent iff \(\Sigma_{i,j}=0\)
    • \(X_i\perp X_j|X-{X_i,X_j}\) iff \(\Sigma^{-1}_{i,j}=0\)

Linear to Gaussian

  • Linear Gaussian Network defines a joint multivariate Gaussian Dist:
    • Y is linear Gaussian with parents \(X_1,...,X_k\)
    • \(P(Y|x)=N(\beta_0+\beta^\top x,\sigma^2)\)
    • \(X_1,..,X_k\) are jointly Gaussian with \(N(\mu,\Sigma\)
  • Then Distribution \(Y\) is normal, \(P(Y)=N(\mu_Y,\sigma^2_Y)\)
    • Using parents:
      • \(\mu_y = \beta_0+\beta^\top\mu\)
      • \(\sigma_y^2 = \sigma^2+\beta^\top\Sigma\beta\)
  • Joint dist over \(\{X,Y\}\) is normal with \(Cov(X_i,Y)=\Sigma^k_{k=1}B_k\Sigma_{i,j}\)

Gaussian to Linear

  • A joint multivariate Gauss distribution defines a linear Gaussian network
    • Given a set of variables \(\{X,Y\}\) in the form of a joint normal distribution
    • \(p(Y|X)=N(\beta_0+\beta^\top X,\sigma^2)\)
  • With parameters

Inference Operations

  • Need to use canonical form to simplify inference operations over normal distribution
    • Allows inference in closed form
  • If factors have two different scopes, we extend scopes to make them match by adding 0 entries to \(K\) and \(h\)

Canonical Form

  • Canonical form \(C(X;K,h,g)\) is defined as:

\[ C(X;K,h,g) = \exp\left(-\frac{1}{2}X^\top KX+h^\top X+g\right)\]

  • With Variables:

Join

  • Canonical join operation:

\[ C(K_1,h_1,g_1)*C(K_2,h_2,g_2)=C(K_1+K_2,h_1+h_2,g_1+g_2)\]

Marginalisation

  • Consider marginalising \(Y\) for the canonical form, resulting in \(C'\):

Reduction

  • Consider setting evidence \(Y=Y\), resulting in \(C'\):

Linear Model

  • For linear Gaussian model \(Y\sim N(\beta_0+\beta^\top X,\sigma^2)\):

Variable Elimination

  • Use canonical form operations and CPDs with evidence set.