Posterior updating using Dirichlet Process priors

1 Posterior distribution

If \(y_i\mid G \stackrel{\text{iid}}{\sim}G\), for \(i=1,\ldots,n\), with \(G\sim\textsf{DP}(\alpha,G_0)\), then the posterior distribution of \(G\) is a \(\textsf{DP}(\alpha^*,G_0^*)\) with \[ \alpha^* = \alpha + n \qquad\text{and}\qquad G_0^*(\cdot) = \frac{\alpha}{\alpha + n}G_0(\cdot) + \frac{n}{\alpha + n}G_n(\cdot) \] where \(G_n(\cdot)\) is the empirical distribution function of the data. Recall that this function is a step function that jumps up by \(1/n\) at each one of the \(n\) data points \(y_1,\ldots,y_n\). Its value at any specified value is the fraction of observations that are less than or equal to the specified value, i.e., \[ G_n(\cdot) = \frac{1}{n}\sum_{i=1}^n 1_{[y_i,\infty)}(\cdot)\,, \] where \(1_A\) is the indicator function of a set \(A\).

This result shows that the DP is a conjugate prior.

2 Posterior mean

The posterior mean estimate for \(G(x)\) is given by \[ \textsf{E}(G(x)\mid\mathbf{y}) = \frac{\alpha}{\alpha + n}G_0(x) + \frac{n}{\alpha + n}G_n(x)\,, \] where \(\mathbf{y}=(y_1,\ldots,y_n)\).

• For small \(\alpha\) relative to \(n\), little weight is placed on the prior guess \(G_0\).
• For large \(\alpha\) relative to \(n\), little weight is placed on the data \(y_1,\ldots,y_n\).
• \(\alpha\) quantifies the degree of belief in the initial guess \(G_0\) (e.g, \(\alpha = 1\) indicates strength of belief in \(G_0\) worth one prior observation).
• \(\alpha\to 0\) does not correspond to a noninformative DP prior since \(\alpha\) controls the the smoothness as well as the variance of the DP prior.

3 Example

Generate \(n=10\) observations \(\mathbf{y}=(y_1,\ldots,y_n)\) from a Cauchy distribution with location parameter \(\mu = 0\) and scale parameter \(\gamma = 1\). Assume that \(y_i\mid G \stackrel{\text{iid}}{\sim}G\), for \(i=1,\ldots,n\), with \(G\sim\textsf{DP}(5,\textsf{N}(0,1))\). Compute the empirical distribution function of the data \(G_n(\cdot)\) and the posterior mean \(\textsf{E}(G(\cdot)\mid\mathbf{y})\).

# random sample from a Cauchy distribution
n <- 10
set.seed(1)
y <- rcauchy(n = n, location = 0, scale = 1)
# prior
alpha <- 5
G0 <- function(x) pnorm(x)
# empirical distribution function 
Gn <- ecdf(x = y)
# posterior mean
Gp <- function(x) (alpha*G0(x) + n*Gn(x))/(alpha + n)
# evaluation and plotting
x <- seq(from = -5, to = 5, len = 1000)
par(mfrow = c(1,1), mar = c(3,3,1.4,1.4), mgp = c(1.75,0.75,0))
curve(expr = pcauchy(x, location = 0, scale = 1), from = -5, to = 5, n = 1000, col = 1, ylab = "Distribution", ylim = c(0,1))
curve(expr = G0(x), n = 1000, add = TRUE, col = "gray")
lines(x = x, y = Gn(x), type = "l", lty = 2, col = 3)
lines(x = x, y = Gp(x), type = "l", lty = 3, col = 4)
legend("topleft", legend = c("Truth","G0","Gn","E(G | y)"), col = c(1,"gray",3,4), lty = c(1,1,3,4), bty = "n")

4 References

• Gershman, S. J., & Blei, D. M. (2012). A tutorial on Bayesian nonparametric models. Journal of Mathematical Psychology, 56(1), 1-12.
• Ghosal, S., & van der Vaart, A. W. (2017). Fundamentals of nonparametric Bayesian inference (Vol. 44). Cambridge University Press.
• Hanson, T. E., Branscum, A. J., & Johnson, W. O. (2005). Bayesian nonparametric modeling and data analysis: An introduction. Handbook of Statistics, 25, 245-278.
• Jara, A. (2017). Theory and computations for the Dirichlet process and related models: An overview. International Journal of Approximate Reasoning, 81, 128-146.
• Jara, A., Hanson, T. E., Quintana, F. A., Müller, P., & Rosner, G. L. (2011). DPpackage: Bayesian semi-and nonparametric modeling in R. Journal of Statistical Software, 40(5), 1.
• MacEachern, S. N. (2016). Nonparametric Bayesian methods: A gentle introduction and overview. Communications for Statistical Applications and Methods, 23(6), 445-466.
• Müller, P., & Mitra, R. (2013). Bayesian nonparametric inference–why and how. Bayesian Analysis (Online), 8(2).
• Müller, P., Quintana, F. A., Jara, A., & Hanson, T. (2015). Bayesian nonparametric data analysis (Vol. 1). New York: Springer.
• Müeller, P., Quintana, F. A., & Page, G. (2018). Nonparametric Bayesian inference in applications. Statistical Methods & Applications, 27, 175-206.
• Rodriguez, A., & Müller, P. (2013). Nonparametric Bayesian inference. In NSF-CBMS Regional Conference Series in Probability and Statistics (Vol. 9). Institute of Mathematical Statistics.