Suppose we have a variable \(X\) with probability density function of \(f(x)\) (continuous) or probability mass function (discrete). Then \(F(x)\) is the probability distribution where \(F(x) = P(X<x)\).
The Mixture model takes the form: \[F(x) = \sum_{k=1}^{K}w_kG_k(x),\] where \(w_k\) are the weights and \(G_k(x)\) are the components such that \(\sum_{k=1}^{K}w_k=1\) and \(G_1,...,G_K\) are cumulative distribution functions \((cdf)\). Similarly, the mixture model version of the probability density function can be written as: \[f(x) = \sum_{k=1}^{K}w_kg_k(x)\]
Example: Location mixture of normals \[f(x) = \sum_{k=1}^{K}w_k\frac{1}{\sqrt{2\pi}\sigma}\exp\left[-\frac{1}{2\sigma^2}(x-\mu_k)^2\right]\] However, we can write a more flexible version (different variances) referred to as a location-scale mixture of normals: \[f(x) = \sum_{k=1}^{K}w_k\frac{1}{\sqrt{2\pi}\sigma_k}\exp\left[-\frac{1}{2\sigma_k^2}(x-\mu_k)^2\right]\]
\(f(x)\) can also be made up of different density functions (or kernals). For example, we can construct a combination of a normal and an exponential density: \[f(x) =w_1\frac{1}{\sqrt{2\pi}\sigma}\exp\left[-\frac{1}{2\sigma^2}(x-\mu)^2\right]+(1-w_1)\frac{1}{\lambda}\exp\left[-\frac{x}{\lambda}\right]\] Another example is zero-inflated data where the density at zero is very high (e.g., bird nests with or without eggs). \[f(x) =w_1\frac{e^{-\lambda}\lambda^x}{x!}+(1-w_1)\delta_0(x)\] where \(\delta_0(x)\) represents the point mass density function at \(x=0\).
If all the \(G_k(x)\)’s are from the same family of distributon functions, we can write \[F(x) = \sum_{k=1}^{K}w_kG(x|\theta_k)\] where \(\theta_k\) are the parameters of the family (normal \(\mu\) and \(\sigma^2\)).
What is the expected value of \(X\) when \[f(x) = \sum_{k=1}^{K}w_kg_k(x)\] \[E[f(x)]=\int_{-\infty}^{\infty}x\cdot f(x)\space dx=\int_{-\infty}^{\infty}x \cdot \left[\sum_{k=1}^{K}w_kg_k(x)\right]\space dx=\sum_{k=1}^{K}w_k\int_{-\infty}^{\infty}x\cdot g_k(x)\space dx\] \[=\sum_{k=1}^{K}w_k\cdot E[g_k(x)]\] So, the expected value of the mixture is simply the weighted sum of the expected values of the components in the mixture.
Now, let’s look at the variance:
\[Var[f(x)]=E[f(x^2)]-[E(f(x)]^2,\] where \[E[f(x^2)]=\int_{-\infty}^{\infty}x^2\cdot f(x)\space dx=\int_{-\infty}^{\infty}x^2 \cdot \left[\sum_{k=1}^{K}w_kg_k(x)\right]\space dx\] \[=\sum_{k=1}^{K}w_k\int_{-\infty}^{\infty}x^2\cdot g_k(x)\space dx=\sum_{k=1}^{K}w_k\cdot E[g_k(x^2)]\] \[=\sum_{k=1}^{K}w_k\left[Var[g_k(x)] +E[g_k(x)]^2\right]\] Therefore, \[Var[f(x)]=\sum_{k=1}^{K}w_k\left[Var[g_k(x)] +E[g_k(x)]^2\right]-\left[\sum_{k=1}^{K}w_k\cdot E[g_k(x)]\right]^2\] Unfortunately, there is no easy way to simply this expression for the variance. However, in the case of \(E[g_k(x)]=0 \space \forall \space K,\) \[Var[f(x)]=\sum_{k=1}^{K}w_k\cdot Var[g_k(x)]\]
A mixture of Poisson distributions of the form \[f(k,\lambda)=P(X=k)=\frac{\lambda^ke^{-\lambda}}{k!}\] where \(E[f(x)]=\lambda\)
\[f(x)=0.3\frac{2^xe^{-2}}{x!}+0.45\frac{3^xe^{-3}}{x!}+0.25\frac{\frac{1}{2}^xe^{-\frac{1}{2}}}{x!}\]
\[E[f(x)]=(0.3)(2)+(0.45)(3)+(0.25)(0.5)\]
(0.3)*(2)+(0.45)*(3)+(0.25)*(0.5)## [1] 2.075\(Var[f(x)]=\lambda\) for Poisson distributions. Therefore, \[Var[f(x)]=\sum_{k=1}^{K}w_k\left[Var[g_k(x)] +E[g_k(x)]^2\right]-\left[\sum_{k=1}^{K}w_k\cdot E[g_k(x)]\right]^2\] \[Var[f(x)]=\sum_{k=1}^{3}w_k\left[\lambda_k+\lambda_k^2 \right]-\left[\sum_{k=1}^{3}w_k \lambda_k\right]^2\]
(0.3)*(2+2^2)+(0.45)*(3+3^2)+(0.25)*(0.5+0.5^2)-((0.3)*(2)+(0.45)*(3)+(0.25)*(0.5))^2## [1] 3.081875