Dirichlet Distribution

The Dirichlet distribution is a multivariate distribution whose components all take values on (0,1) and which sum to one.

The Dirichlet distribution is frequently used to describe uncertainty about the probabilities of a Multinomial distribution. As such, the Dirichlet distribution is to the Multinomial distribution what the Beta distribution is to the Binomial distribution.

The Dirichlet distribution is parameterized by the vector α, which has the same number of elements (k) as our multinomial parameter θ.

The Dirichlet distribution is a generalization of the beta distribution into multiple dimensions

We have a model • This model encodes beliefs about the parameters as well (Prior) • We observe data (Likelyhood) • We update our beliefs about the parameters according to the observed data (Posterior)

Beta • Note that the Beta distribution is a specific case of a Dirichlet • The Dirichlet distribution is conjugate to the multinomial distribution • This means that identifying the posterior will again be easy

library(Compositional)
library(MCMCpack)
## Warning: package 'MCMCpack' was built under R version 3.3.3
## Loading required package: coda
## Warning: package 'coda' was built under R version 3.3.3
## Loading required package: MASS
## ##
## ## Markov Chain Monte Carlo Package (MCMCpack)
## ## Copyright (C) 2003-2017 Andrew D. Martin, Kevin M. Quinn, and Jong Hee Park
## ##
## ## Support provided by the U.S. National Science Foundation
## ## (Grants SES-0350646 and SES-0350613)
## ##
draw = 10
alpha = c(1,2,3)
dimension = 3
x = rdirichlet(10, c(1,2,3))
x
##             [,1]      [,2]      [,3]
##  [1,] 0.14603686 0.6005238 0.2534393
##  [2,] 0.29981294 0.1421750 0.5580121
##  [3,] 0.02693696 0.5467726 0.4262905
##  [4,] 0.09378367 0.4023002 0.5039161
##  [5,] 0.15564760 0.5801769 0.2641755
##  [6,] 0.14038460 0.2249018 0.6347136
##  [7,] 0.05332627 0.4821364 0.4645373
##  [8,] 0.07198878 0.5245364 0.4034748
##  [9,] 0.21998003 0.1285864 0.6514336
## [10,] 0.20652214 0.2984746 0.4950032
# Find densities at random points.
ddirichlet(x, c(1,2,3))
##  [1] 2.314353 2.656205 5.961688 6.129401 2.429387 5.436255 6.242556
##  [8] 5.123417 3.274059 4.388082

Dirichlet Distribution Plots simulate Topic Modeling

We like to plot 15 random draws of Dirichlet distribution with α=1 and dimension n=10. This image is from Prof. David Blei’s Topic Modeling tutorial at KDD 2011.

alpha = Dirichlet Parameter x = Per-document topic proposions

## Warning: package 'ggplot2' was built under R version 3.3.3

The higher value of αi, the greater “weight” of Xi and the greater amount of the total “mass” is assigned to it (recall that in total it must be x1+⋯+xk=1). If all αi are equal, the distribution is symmetric. If αi<1, it can be thought as anti-weight that pushes away xi toward extremes, while when it is high, it attracts xi toward some central value (central in the sense that all points are concentrated around it, not in the sense that it is symmetrically central). If α1=⋯=αk=1, then the points are uniformly distributed.

Visulization of Trivariate Dirichlet Distributions

This can be seen on the plots below, where you can see trivariate Dirichlet distributions (unfortunately we can produce reasonable plots only up to three dimensions) parameterized by (a) α1=α2=α3=1, (b) α1=α2=α3=10, (c) α1=1,α2=10,α3=5, (d) α1=α2=α3=0.2.

density <- ddirichlet(c(.1,.2,.7), c(1,1,1))
density
## [1] 2
density <- ddirichlet(c(.5,.5,.5), c(1,1,1))
density
## [1] 0
density <- ddirichlet(c(.1,.1,.8), c(1,1,1))
density
## [1] 2
density <- ddirichlet(c(.1,.1,.9), c(1,1,1))
density
## [1] 0
density <- ddirichlet(c(.5,.5,.5), c(10,10,10))
density
## [1] 0
draws <- rdirichlet(200, c(.1,.1,.1) )
bivt.contour(draws)

draws <- rdirichlet(200, c(1,1,1) )
bivt.contour(draws)

draws <- rdirichlet(200, c(10,10,10) )
bivt.contour(draws)

draws <- rdirichlet(200, c(100,100,100) )
bivt.contour(draws)

draws <- rdirichlet(200, c(1000,1000,1000) )
bivt.contour(draws)

Visulization of Iris Dataset

This famous (Fisher’s or Anderson’s) iris data set gives the measurements in centimeters of the variables sepal length and width and petal length and width, respectively, for 50 flowers from each of 3 species of iris. The species are Iris setosa, versicolor, and virginica.

library(Compositional)
x <- as.matrix( iris[, 1:3] )
x <- x / rowSums(x)
head(iris)
##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 1          5.1         3.5          1.4         0.2  setosa
## 2          4.9         3.0          1.4         0.2  setosa
## 3          4.7         3.2          1.3         0.2  setosa
## 4          4.6         3.1          1.5         0.2  setosa
## 5          5.0         3.6          1.4         0.2  setosa
## 6          5.4         3.9          1.7         0.4  setosa
head(x)
##      Sepal.Length Sepal.Width Petal.Length
## [1,]    0.5100000   0.3500000    0.1400000
## [2,]    0.5268817   0.3225806    0.1505376
## [3,]    0.5108696   0.3478261    0.1413043
## [4,]    0.5000000   0.3369565    0.1630435
## [5,]    0.5000000   0.3600000    0.1400000
## [6,]    0.4909091   0.3545455    0.1545455
bivt.contour(x)