7.6 - the F distribution

Motivation: two normal samples

  • Let \(Y_1,Y_2,...,Y_n\) be an i.i.d. sample from a \(N(\mu_Y,\sigma^2_Y)\) population, and \(X_1,X_2, ...,X_m\) an i.i.d. sample from a \(N(\mu_X, \sigma^2_X)\) population
  • Consider the ratio of the sample variances scaled by their population variances: \(\frac{S^2_X/\sigma^2_X}{S^2_Y/\sigma^2_Y}\)
  • This ratio follows an F distribution

The \(F\) pdf

A random variable \(W\) is said to follow an \(F\) distribution with \(p\) numerator and \(q\) denominator degrees-of-freedom, and we say \(W\sim F_{p,q}\), if it has pdf given by:

\[f_W(w) = \frac{\Gamma\left(\frac{p+q}{2}\right)}{\Gamma{\left(\frac{p}{2}\right)}\Gamma{\left(\frac{q}{2}\right)}}\left(\frac{p}{q}\right)^{p/2}w^{p/2-1}\left(1 + \frac{p}{q}w\right)^{-\left(\frac{p+q}{2}\right)}; w > 0\]

Plots of \(F\) distributions

Various F distributions

The \(F\) as a ratio of chi-squares

  • The \(F\) distribution arises as the distribution of the ratio of two scaled independent chi-squares
  • Specifically, if…
    • \(U\sim \chi^2_{p}\)
    • \(V\sim \chi^2_q\)
    • \(U \perp\!\!\!\perp V\)
    • \(W = \frac{U/p}{V/q}\)
  • …then \(W \sim F_{p,q}\)
  • To prove:
    • \(2\rightarrow 2\) transformation of \((U,V) \rightarrow (W,X)\) for conveniently chosen \(X\)
    • Integrate out \(X\) to find marginal of \(W\)

Joint of \((U,V)\) and inverses

  • Step 1: find joint of \((U,V)\). If \(U\sim \chi^2_{p}\) , \(V\sim \chi^2_q\), and \(U \perp\!\!\!\perp V\), then:

\[\scriptsize f_{U,V}(u,v) = \frac{1}{2^{p/2}\Gamma(p/2)} u^{p/2-1} e^{-u/2} \cdot \frac{1}{2^{q/2}\Gamma(q/2)} v^{q/2-1} e^{-v/2}, \qquad u>0,\; v>0.\]

  • Step 2: define transformation and inverses

\[\scriptsize \begin{align}W &= \frac{U/p}{V/q}\\ X &= V \end{align}\] Note that \(W >0, X>0\) is joint support

\[ \scriptsize \begin{align} U &=\frac{p}{q} W X\\ V &= X\end{align}\]

  • Step 3: Jacobian

\[\scriptsize J = \begin{bmatrix} \dfrac{p}{q} x & \dfrac{p}{q} w \\[6pt] 0 & 1 \end{bmatrix}\Rightarrow \left| det(J) \right| = \frac{p}{q} x.\]

Joint of \((W,X)\)

  • Step 4: plug and chug

\[f_{W,X}(w,x) = f_{U,V}\!\left( \frac{p}{q}wx,\; x \right) \left| J \right|\]

\[= \frac{1}{2^{p/2}\Gamma(p/2)} \left( \frac{p}{q}wx \right)^{p/2-1} e^{-\frac{pwx}{2q}} \cdot \frac{1}{2^{q/2}\Gamma(q/2)} x^{q/2-1} e^{-x/2} \cdot \frac{p}{q} x\]

\[=\frac{1}{2^{(p+q)/2}\Gamma(p/2)\Gamma(q/2)} \left( \frac{p}{q} \right)^{p/2} w^{p/2-1} x^{(p+q)/2-1} \exp\!\left[-\frac{x}{2}\!\left(1+\frac{p}{q}w\right)\right], w>0,\; x>0.\]

Marginal of \(W\)

  • Step 5: integrate out \(X\)

\[f_W(w) = \int_0^\infty \frac{1}{2^{(p+q)/2}\Gamma(p/2)\Gamma(q/2)} \left( \frac{p}{q} \right)^{p/2} w^{p/2-1} x^{(p+q)/2-1} \exp\!\left[-\frac{x}{2}\!\left(1+\frac{p}{q}w\right)\right]\, dx\]

\[\stackrel{show!}{=}\frac{\Gamma\left(\frac{p+q}{2}\right)}{\Gamma{\left(\frac{p}{2}\right)}\Gamma{\left(\frac{q}{2}\right)}}\left(\frac{p}{q}\right)^{p/2}w^{p/2-1}\left(1 + \frac{p}{q}w\right)^{-\left(\frac{p+q}{2}\right)}; w > 0\]

Ratio of scaled normal sample variances

  • Let \(Y_1,Y_2,...,Y_n\) be an i.i.d. sample from a \(N(\mu_Y,\sigma^2_Y)\) population, and \(X_1,X_2, ...,X_m\) an i.i.d. sample from a \(N(\mu_X, \sigma^2_X)\) population
  • Then \(\frac{S^2_X/\sigma^2_X}{S^2_Y/\sigma^2_Y} \sim F_{m-1,n-1}\)
  • Proof: practice!

\(F\) distribution in R

  • df(x, df1, df2): evaluate \(f_W(x)\)
  • pf(x, df1, df2): evaluate cumulative probabilities
  • qf(p, df1, df2): find quantiles

Application: testing equality of population variances

  • Let \(Y_1,Y_2,...,Y_n\) be an i.i.d. sample from a \(N(\mu_Y,\sigma^2_Y)\) population, and \(X_1,X_2, ...,X_m\) an i.i.d. sample from a \(N(\mu_X, \sigma^2_X)\) population
  • Consider testing:

\[H_0: \sigma^2_X = \sigma^2_Y\] \[H_a: \sigma^2_X \ne \sigma^2_Y\]

  • Under \(H_0\), \(\sigma^2_X = \sigma^2_Y \Rightarrow \frac{S^2_X/\sigma^2_X}{S^2_Y/\sigma^2_Y}=\frac{S^2_X}{S^2_Y}\)
  • Reject \(H_0\) if \(\frac{S^2_X}{S^2_Y}\) is either “too large” or “too small”, as determined by \(F_{m-1, n-1}\) distribution

Application: one-way ANOVA

  • In one-way ANOVA, we have data \(Y_{i1}, Y_{i2},...,Y_{ir_i}\)
    • \(i \in \{1,2,...,k\}\)
    • \(r_i\) replications in group \(i\)
    • \(N = \sum_{i=1}^k r_i\) observations in total
  • We assume that \(Y_{i1}, Y_{i2},...,Y_{ir_i} \stackrel{i.i.d.}{\sim}N(\mu_i, \sigma^2)\): potentially different means, common variance
one way anova visual
  • Visual above: \(k=3\), \(r_1=r_2=r_3 =100\), \(N=300\)

ANOVA test statistic

  • In one-way ANOVA we want to test:

\[H_0: \mu_1 = \mu_2 = ... = \mu_k\] \[H_a: \mbox{at least one mean differs from the others}\]

  • To test, we form the test statistic:

\[ \small F = \frac{\sum_{i=1}^k r_i (\bar Y_{i\cdot}-\bar{Y}_{\cdot\cdot})^2/(k-1)}{\sum_{i=1}^k\sum_{j=1}^{r_i}(Y_{ij}-\bar{Y_{i\cdot}})^2/(N-k)} = \frac{SSTreatment/(k-1)}{SSError/(N-k)} =\frac{(SSTotal - SSError)/(k-1)}{SSError/(N-k)} \]

  • Reject \(H_0\) when \(F\) is “large”. How to determine “large”? Need a sampling distribution!

Distribution of F ingredients

Under \(H_0\), all \(Y_{ij}\) come from one common parent population: \(N(\mu,\sigma^2)\). Thus:

\[\frac{SSTotal}{\sigma^2} = \frac{\sum_{i=1}^k\sum_{j=1}^{r_i}(Y_{ij}-\bar{Y_{\cdot\cdot}})^2}{\sigma^2} = \frac{(N-1)S_p^2}{\sigma^2} \sim \chi^2_{N-1}\]

\[\frac{SSError}{\sigma^2} = \frac{\sum_{i=1}^k\sum_{j=1}^{r_i}(Y_{ij}-\bar{Y_{i\cdot}})^2}{\sigma^2} = \underbrace{\frac{\sum_{i=1}^k(r_i -1)S^2_i}{\sigma^2}}_{\mbox{function of }S_i^2}= \sum_{i=1}^k\underbrace{\frac{(r_i -1)S^2_i}{\sigma^2}}_{\sim \chi^2_{r_i-1}} \sim \chi^2_{N-k}\]

\[\frac{SSTreatment}{\sigma^2} =\underbrace{\frac{\sum_{i=1}^kr_i(\bar{Y_{i\cdot}}-\bar Y_{\cdot \cdot})^2}{\sigma^2}}_{\mbox{function of }\bar Y_i} \Rightarrow SSTreatment \perp \!\!\!\perp SSError\]

\[\underbrace{\frac{SSTotal}{\sigma^2}}_{\sim \chi^2_{N-1}}=\underbrace{\frac{SSTreatment}{\sigma^2}+\underbrace{\frac{SSError}{\sigma^2}}_{\sim \chi^2_{N-k}}}_{independent} \Rightarrow \frac{SSTreatment}{\sigma^2}\sim \chi^2_{N-1-(N-k)}\equiv \chi^2_{k-1} \mbox{(see 7.4)}\]

\[\Rightarrow F = \frac{SSTreatment/(k-1)}{SSError/(N-k)} ``\equiv"\frac{\chi^2_{k-1}/(k-1)}{\chi^2_{N-k}/(N-k)}\sim F_{k-1,N-k}\]

Note required assumptions for this to be the right sampling distribution for determining “large”! Normality of data and common population variance.