Calculation of Linear Models

• Collinearity

In this section we’re going to explain the concept of collinearity and how understanding this can help us avoid designing experiments where ultimately we won’t be able to estimate the effects we’re interested in.

Let’s motivate this with a system of equations:

The following system of equations \[ \left.\begin{align}a+c&=1\\b-c&=1\\a+b&=2\end{align}\right\}\#\] has more than one solution: \(a=1-c\), \(b=1+c\).

Two examples: \(a=1,b=1,c=0\) and \(a=0,b=2,c=1\).

Using linear algebra, we can see this immediately because one of the three columns can be written as a linear combination of the first two.

\(\#\) can be written as follows: \[\begin{pmatrix}1&0&1\\0&1&-1\\ 1&1&0\end{pmatrix}\begin{pmatrix}a\\b\\c\end{pmatrix}=\begin{pmatrix}1\\1\\2\end{pmatrix}\] Note that the third column is a linear combination of the first two:

\[\begin{pmatrix}1\\0\\1\end{pmatrix}-\begin{pmatrix}0\\1\\1\end{pmatrix}=\begin{pmatrix}1\\-1\\0\end{pmatrix}\] So we can say that the third column is collinear to the other two. What this implies is that we can rewrite the system of equations using that fact and simplify it into two multiplications: \[\begin{align}\begin{pmatrix}1&0&1\\0&1&-1\\ 1&1&0\end{pmatrix}\begin{pmatrix}a\\b\\c\end{pmatrix}&=a\begin{pmatrix}1\\0\\1\end{pmatrix}+b\begin{pmatrix}0\\1\\1\end{pmatrix}+c\begin{pmatrix}1-0\\0-1\\1-1\end{pmatrix}\\ &=(a+c)\begin{pmatrix}1\\0\\1\end{pmatrix}+(b-c)\begin{pmatrix}0\\1\\1\end{pmatrix}\end{align}\] the third column does not add a constraint.

Note that an infinite number of values of \(a\) and \(c\) give same \(a+c\).

• Collinearity and Least Squares

In data analysis the way this comes up is when we try to fit a model that has this problem, that one of the columns of our design matrix is collinear to the others.

A simple example:

Consider a design matrix \(X\) with two collinear columns:

\[X=(1 \ \ X_1 \ \ X_{2} \ \ X_{3}) \ \ \text{with, say} \ \ X_{3}=-X_{2}\] we can rewrite the residuals like this \[\begin{align}&Y-\{1\beta_{0}+X_{1}\beta_{1}+X_{2}\beta_{2}+X_{3}\beta_{3}\}\\ &=Y-\{1\beta_{0}+X_{1}\beta_{1}+X_{2}\beta_{2}-X_{2}\beta_{3}\} \\ &=Y-\{1\beta_{0}+X_{1}\beta_{1}+X_{2}(\beta_{2}-\beta_{3})\}\end{align}\] If \(\hat{\beta_{1}},\hat{\beta_{2}},\hat{\beta_{3}}\) is a solution, then \(\hat{\beta_{1}},\hat{\beta_{2}}+1,\hat{\beta_{3}}+1\) is also a solution.

So it does not have a unique solution and, we can’t estimate these parameters.

• Example

Imagine that you’re interested in studying the effect of four drugs \(A,B,C \ \ \text{and} \ \ D\) and you’re going to treat mice with these drugs. You assign two mice to each group. After starting the experiment by giving \(A\) and \(B\) to female mice, we realize there might be sex effect. We decide to give \(C\) and \(D\) to males with hopes of estimating this sex effect as well.

Now we write down the design matrix without an intercept for explaining purposes:

\[\begin{pmatrix}Sex & A & B & C & D\\0 & 1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 & 0 \\0 & 0 & 1 & 0 & 0 \\0 & 0 & 1 & 0 & 0 \\1 & 0 & 0 & 1 & 0 \\1 & 0 & 0 & 1 & 0 \\1 & 0 & 0 & 0 & 1 \\ 1 & 0 & 0 & 0 & 1\end{pmatrix}\] We can see that the “Sex” column is equal to the C-column plus the D-column: \[\begin{pmatrix}Sex\\0\\0\\0\\0\\1\\1\\1\\1\end{pmatrix}=\begin{pmatrix}C\\0\\0\\0\\0\\1\\1\\0\\0\end{pmatrix}+\begin{pmatrix}D\\0\\0\\0\\0\\0\\0\\1\\1\end{pmatrix}\] Therefore it is collinear and we can’t obtain estimates. We can see this in R very quickly.

• \(\underline{\text{Definition}}\) The rank of a matrix columns is the number of columns that are independent to all the others.

If the rank is smaller than the number of columns the LSE are not unique.

set.seed(1)
Sex<-c(0,0,0,0,1,1,1,1)
A<-c(1,1,0,0,0,0,0,0)
B<-c(0,0,1,1,0,0,0,0)
C<-c(0,0,0,0,1,1,0,0)
D<-c(0,0,0,0,0,0,1,1)
X<-model.matrix(~0+Sex+A+B+C+D)
cat("number of columns=",ncol(X),"rank=",qr(X)$rank,"\n")

number of columns= 5 rank= 4