Problems TODO: 8.62, 8.63, 8.64, 8.65, 8.66

8.62

Show that the gamma distribution is a conjugate prior for the exponential distribution. Suppose that the waiting time in a queue is modeled as an exponential random variable with unknown parameter \(\lambda\), and that the average time to serve a random sample of 20 customers is 5.1 minutes. A gamma distribution is used as a prior. Consider two cases: (1) the mean of the gamma is 0.5 and the standard deviation is 1, and (2) the mean is 10 and the standard deviation is 20. Plot the two posterior distributions and compare them. Find the two posterior means and compare them. Explain the differences.

Solution

First we show that the gamma distribution is a conjugate prior for the exponential distribution. By a definition in the textbook we have that: If the prior distribution belongs to a family \(G\) and conditional on the parameters of \(G\), the data have a distribution \(H\) then \(G\) is said to be conjugate to \(H\) if the posterior is in the family \(G\).

So let \(\theta \sim Gamma(\alpha, \beta)\) such that we have a prior: \[f_\Theta(\theta) = \frac{\beta^\alpha\theta^{\alpha - 1}e^{-\beta\theta}}{\Gamma(\alpha)}\] And now we assume that the data given the parameter \(\theta\) have an exponential distribution, we can write this as \(X_i|\theta \sim exp(\theta)\): \[f_{X_i|\Theta}(x_i|\theta) = \theta e^{-\theta x_i}\] Now the joint distribution of the n i.i.d samples is given by: \[f(X|\theta) = \theta^n e^{-\theta \sum x_i}\]

We now use the following to get the posterior: \[\text{Posterior} \propto \text{Likelihood} \times \text{Prior}\] \[\Rightarrow f_{\Theta | X}(\theta | x) \propto f_{x|\Theta}(x|\theta)\times f_\Theta(\theta)\] \[f_{\Theta | X}(\theta | x) \propto \theta^n e^{-\theta \sum x_i} \times \frac{\beta^\alpha\theta^{\alpha - 1}e^{-\beta\theta}}{\Gamma(\alpha)}\] Now since we are deriving based on a proportion we can eliminate any constants, since they too only add proportion (I don’t know if that made sense, it did in my head), so we can write: \[f_{\Theta | X}(\theta | x) \propto \theta^n e^{-\theta \sum x_i} \times \theta^{\alpha - 1}e^{-\beta\theta}\] \[ = \theta^{n + \alpha - 1}e^{-\theta(\sum x_i + \beta)}\] \[\therefore \theta|X \sim Gamma(n+\alpha, \sum x_i + \beta)\] and so a conjugate.

Now we consider the queue being modeled as an exponential. First we formulate the general form of the posterior then consider each of the two cases. Building upon Example A Section 8.6 we have that: \[f_{\Lambda | X}(\lambda | x) \propto \lambda^{\sum x_i + \alpha - 1}e^{-(n+\beta)\lambda}\] where we derive the parameters for the gamma as follows: \[\beta = \mu_1/(\mu_2 - \mu_1^2)\;\; \text{and} \;\; \alpha = \beta\mu_1\]

Case I: Mean of the gamma is .5 and \(\sigma = 1\). Then we formulate \(\mu_2 = \mu_1^2 + \sigma^2 = .5^2 + 1 = 1.25\), then we have that: \[\beta = .5/(1.25 - .25) = .5 \;\; \text{and} \;\; \alpha = .5\times .5 = .25\] It follows then that the posterior gamma has parameters: \[\alpha' = \sum x_i + \alpha = 102 + .5 = 102.5\] \[\beta' = 20 + .5 = 20.5\] So posterior mean results in: \[\mu_{post} = 102.5/20.5 = 5\]

Case II: Here we consider gamma mean to be 10 and standard deviation 20. Analogous to the above we can formulate the parameters of the priors as follows: \[\beta = 10/(500 - 100) = .025 \;\; \text{and} \;\; \alpha = .025 \times 10 = .25\] \[\Rightarrow \alpha'' = 102 + .025 = 102.025 \;\; \text{and} \;\; \beta'' = 20 + .25 = 20.25\] And so posterior mean is: \[\mu_{post} = 102.25/20.025 = 5.106\]

Plots:

Now we plot these two posterior distributions.

As we can see the gamma for the first case is a bit more condensed, moreover the posterior for the second case appears to be more centered about the true value.

8.63

Suppose that 100 items are sampled from a manufacturing process and 3 are found to be defective. A beta prior is used for the unknown proportion \(\theta\) of defective items. Consider two cases: (1) \(a = b = 1\), and (2) \(a = 0.5\) and \(b = 5\). Plot the two posterior distributions and compare them. Find the two posterior means and compare them. Explain the differences.

Solution

The experiment taking place is one that can be modeled as a Binomial so we can write the posterior: \[f_{\Theta|X}(\theta|x) \propto f_{X|\Theta}(x|\theta)f_\Theta(\theta)\] since Beta is conjugate Binomial. \[ = \binom{n}{x}\theta^x(1 - \theta)^{n-x}\frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)}\theta^{\alpha - 1}(1 - \theta)^{\beta - 1}\] \[\propto \theta^x(1 - \theta)^{n-x}\theta^{\alpha - 1}(1 - \theta)^{\beta - 1}\] \[ = \theta^{x + \alpha -1}(1 - \theta)^{n - x + \beta -1}\]

From the information given we have that the likelihood is given by: \[f_{X|\Theta}(x|\theta) \propto \theta^3(1 - \theta)^{97} \ni \] \[f_{\Theta|X}(\theta|x) \propto \theta^{3 + \alpha -1}(1 - \theta)^{100 - x + \beta -1}\]

So we have a Beta posterior \(\theta|X \sim Beta(3 + a, 97 + b)\)

Case I: \(a = b = 1\)

\(\theta|X \sim Beta(4, 98)\) and so we have the posterior mean is given by: \[\mu_{post} = 4/(4 + 98) = 0.03921569\]

Case II: \(a = .5, b = 5\)

\(\theta|X \sim Beta(3.5, 102)\) and so we have the posterior mean is given by: \[\mu_{post} = 3.5/(3.5 + 102) = 0.03317536\]

8.64

This is a continuation of the previous problem. Let \(X = 0\) or \(1\) according to whether an item is defective. For each choice of the prior, what is the marginal distribution of \(X\) before the sample is taken? What are the marginal distributions after the sample is taken? (Hint: for the second question, use the posterior distribution of \(θ\).)

Solution

Before the data is taken.

In order to find the marginal of \(X\) we simply have to “integrate out” the parameter at hand \(\theta\) from the joint distribution of \(\Theta\) and \(X|\Theta\). \[f_X(x) = \int f_{X|\Theta}(x|\theta)f_\Theta(\theta)d\theta\] \[ = \int \binom{n}{x}\theta^x(1 - \theta)^{n - x}\frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)}\theta^{\alpha - 1}(1 - \theta)^{\beta - 1}d\theta\] \[\binom{n}{x}\frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)} \int \theta^x(1 - \theta)^{n - x}\theta^{\alpha - 1}(1 - \theta)^{\beta - 1}d\theta\] \[ = \binom{n}{x}\frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)} \int \theta^{x + \alpha -1}(1 - \theta)^{n - x + \beta -1}d\theta\]

Once a sample has been taken we can use the posterior as follows: \[f_{\Theta|X}(\theta|x) = \frac{f_{X|\Theta}(x|\theta)f_\Theta(\theta)}{f_X(x)}\] \[ = \frac{1}{f_{\Theta|X}(\theta|x)} = \frac{f_X(x)}{f_{X|\Theta}(x|\theta)f_\Theta(\theta)}\] \[\Rightarrow f_X(x) = \frac{f_{X|\Theta}(x|\theta)f_\Theta(\theta)}{f_{\Theta|X}(\theta|x)}\] Now we can simply substitute in the known distributions

\[f_X(x) = \frac{\binom{n}{x}\frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)} \theta^{x + \alpha -1}(1 - \theta)^{n - x + \beta -1}}{\frac{\Gamma(3 + \alpha + 100 -3 +\beta)}{\Gamma(3+\alpha)\Gamma(100 - 3 +\beta)} \theta^{3 + \alpha -1}(1 - \theta)^{n - 3 + \beta -1}}\]

Case I: \(a=b=1\)

8.65

Suppose that a random sample of size 20 is taken from a normal distribution with unknown mean and known variance equal to 1, and the mean is found to be \(\bar{x} = 10\). A normal distribution was used as the prior for the mean, and it was found that the posterior mean was 15 and the posterior standard deviation was 0.1. What were the mean and standard deviation of the prior?

Solution

As was derived in both class and the textbook we can find the mean and standard deviation of the prior by solving the following respectively:

\[\mu_{post} = 15 = \frac{\bar{X}\sigma_0^2 + \mu_0(\sigma^2/n)}{\sigma_0^2 + (\sigma^2/n)}\] \[ = \frac{10\sigma_0^2 + \mu_0(1/20)}{\sigma_0^2 + (1/20)}\] and \[\sigma_{post}^2 = \frac{\sigma^2(\sigma_0^2/n)}{\sigma_0^2 + (\sigma^2/n)}\] \[ 0.1^2 = \frac{sigma_0^2/20}{\sigma_0^2 + (1/20)} \Rightarrow \sigma_0^2 = 1/80\] Now plugging this back in to \(\mu_{post}\) we solve to get \[\mu_0 = 16.25\]

8.66

Let the unknown probability that a basketball player makes a shot successfully be \(\theta\). Suppose your prior on \(\theta\) is uniform on [0, 1] and that she then makes two shots in a row. Assume that the outcomes of the two shots are independent.

  1. What is the posterior density of \(\theta\)
  2. What would you estimate the probability that she makes a third shot to be?

Solution

  1. The likelihood is given by the binomial as follows: \[f_{X|\Theta}(x|\theta) = \binom{2}{2}\theta^2(1 - \theta)^0 = \theta^2\] The prior was chosen to be the uniform on \([0,1]\) so this is: \[f_\Theta(\theta) = 1\] All in all we have that the posterior is given by: \[f_{\Theta|X}(\theta|x) = \frac{\theta^2}{\int_0^1 \theta^2} = 3\theta^2 \propto Beta(3,1)\]

  2. To estimate the probability that she makes a third shot we are interested in: \[P(\theta|2) \]