1.2 - linear transformations

Linear combinations and dot products

• A linear combination of a vector is simply a weighted sum of its coordinates

• Linear combinations are often represented by dot products

• The dot product of two \(p-\)dimensional vectors is a single number

• E.g., consider the dot product of \(u = (3,1)\) with weighting vector \(w = (-1, 4)\):

Example: Kwik Trip

• At Kwik Trip, a corn dog costs $1.99; a milkshake costs $3.80, and a 9.9 oz bag of takis costs $5.29.
  
• You go to Kwik Trip and buy 2 corn dogs, a milkshake, and 5 bags of takis.
• How can you represent \(C\), the total cost, as the dot product of two vectors? How would you define these vectors? What is \(C\)?

\[u = ?\] \[w = ?\] \[C = <u,w> = ?\]

Example: linear regression

• In linear regression, we often represent fitted values as the dot product between and \(x\) vector and a \(\hat\beta\) vector.
• Suppose you’ve fit a linear regression model of house price on the size (in square meters) and number of bedrooms.
  
• You have the following output:

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept)   55957.2    22896.1   2.443   0.0187 *  
size           989.4       195.6   5.056  2.04e-06 ***
bedrooms     10291.9      3688.2   2.791   0.0076 ** 

• GOAL: predict the price of a home with 200 sq m and 3 bedrooms.
• This is the dot product \(<x,\hat\beta>\) with:

\[x = ?\] \[\hat\beta = ?\]

Geometric interpretation

• The dot product of two \(p\)-dimensional vectors is a single number

• The sign of this number is related to the angle between the vectors:

Geometric interpretation

It turns out:

\[ <u,v> = ||u||_2 ||v||_2 cos(\theta) \]

where \(\theta\) is the angle formed by \(u\) and \(v\):

Note also: orthogonal (i.e., perpendicular) vectors have dot product = 0!

Matrix multiplication: a refresher

Consider the \(2 \times 2\) matrix \(A = \begin{pmatrix} 3 & 2 \\ 1 & 2 \end{pmatrix}\) and 2-dimensional vector \(u = (1,2)\).

Then:

\[Au = \begin{pmatrix} 3 & 2 \\ 1 & 2 \end{pmatrix}\begin{pmatrix} 1 \\ 2 \end{pmatrix}\]

Matrix dimensions practice

Consider the matrices below \(A\):

\[A = \begin{pmatrix} 2 & 1 & 0\\ 1 & 3 & 2\\ 2 & 1 & 4 \end{pmatrix}, B = \begin{pmatrix} 3 & 1 & 4 \\ 2 & 3 & 1 \end{pmatrix}, C = \begin{pmatrix} 2 & 4 & 0 \\ 1 & 3 & 2 \\ 0 & 2 &1\\ 3 & 1 &4\\ \end{pmatrix}\]

Which of the following matrix products are defined? If defined, what is the dimension of the product? Pick one of the defined products and find it.

Order matters!

• Reconsider the \(2\times 2\) matrix \(A = \begin{pmatrix} 3 & 2 \\ 1 & 2 \end{pmatrix}\) and vector \(u = (1,2)\).
• To multiply \(Au\), \(u\) must be expressed as a \(2 \times 1\) matrix: \[\begin{pmatrix} 1 \\ 2 \end{pmatrix}\]
• Result of \(Au\) is \(2\times 1\) (\(ncol(A) \times nrow(u)\))

Linear transformation: a visual

Identity matrix

• The identity matrix \(I\) is the matrix equivalent of “1”.
• Square \(p\times p\) matrix with 1’s on the diagonal and 0’s elsewhere:

\[ I = \begin{pmatrix} 1 & 0 & \dots & 0 \\ 0 & 1 & \dots& 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \dots & 1 \end{pmatrix} \]

• For any \(n \times p\) matrix \(X\), \(XI = X\).

Matrix inverses: going backward

Finding arbitrary transformation matrix

ggplot code

library(ggplot2)
library(patchwork) #for adding plots side-by-side

bluedf <- data.frame(bluecoords)
greendf <- data.frame(greencoords)


# Create base
base <- ggplot() +
  scale_x_continuous(breaks=seq(-5,5,by=1),limits=c(-5,5)) +
    scale_y_continuous(breaks=seq(-5,5,by=1),limits=c(-5,5)) + 
  geom_vline(aes(xintercept = 0)) +  geom_hline(aes(yintercept = 0)) +  
  theme_minimal(base_size = 14) + 
  theme(panel.grid.minor = element_blank()) + 
  labs(x='', y='')

# Add points
bluepoints <- base +  
  geom_point(data = bluedf, aes(x = X1, y = X2), size = 3,col='cornflowerblue') 
greenpoints <- base + 
  geom_point(data = df2, aes(x = X1, y = X2), size = 3,col='forestgreen')
p1 + p2

Transformations don’t have to be \(p\rightarrow p\)

• In the examples we’ve considered so far, we’ve transformed 2-dimensional space to 2-dimensional space.
• We don’t have to go \(p\rightarrow p\): we can go \(p\rightarrow m\) with \(m<p\), or \(p\rightarrow q\) with \(q > p\).
  
• In this class, we will most frequently encounter dimension reduction: that is. transformations into lower dimensional space (\(p \rightarrow m\) with \(m < p\)).
• There are 2 ways to “transform down”:
  • Use a lower dimensional transformation matrix;
  • Use a rank deficient transformation matrix

Lower dimension \(A\)

• The easiest way to transform down is to use a lower-dimension transformation matrix.

• Consider 3-dimensional row vector \(u = (-1, 3, 2)\) and \(3 \times 2\) transformation matrix \(A = \begin{pmatrix}3 & 2 \\ 1 & 1 \\ -1 & 4\end{pmatrix}\)

\[ u A = (-1, 3, 2)\begin{pmatrix}3 & 2 \\ 1 & 1 \\ -1 & 4\end{pmatrix}\]

Rank deficient \(A\)

• In the previous example it was obvious that using a \(3\times 2\) matrix \(A\) would transform 3-dimensional \(u\) into 2 dimensions
• But downward transformations can be more subtle!
• Consider now:

\[A = \begin{pmatrix} 1 & 4 & 2 \\ 2 & 5 & 1 \\ 3 & 6 & 0 \end{pmatrix}\]

Plotting \(U\) and \(UA\)

Determinants and rank deficiency

• A rank deficient transformation matrix has redundancy in rows or columns
• In other words, one or more of the columns is a linear combination of the others (equivalent for rows)
• A rank deficient, \(p\times p\) matrix \(A\) has determinant = 0

A

     [,1] [,2] [,3]
[1,]    1    4    2
[2,]    2    5    1
[3,]    3    6    0

det(A)

[1] 0

• Determinant = scalar-valued function of a square matrix

\[det \begin{pmatrix} a & b \\ c & d \end{pmatrix} =ad-bc\] - …increasingly more complex for higher \(p\)! (use R in general)

Rank deficiency in 2 dimensions

Consider the \(n=50\), \(p=2\)-dimensional vectors in the cars data set:

Rank deficiency in 2 dimensions

• Let’s apply the \(2 \times 2\) transformation matrix \(A = \begin{pmatrix} 1 &0\\ 0&0\end{pmatrix}\)
• This collapses the data set down to just its \(x\) coordinates:

A <- matrix(c(1,0,
              0,0
              ), 
            nrow = 2, ncol = 2,
            byrow=TRUE
            )
det(A)

[1] 0

cars_transformed <- data.frame(as.matrix(cars) %*% A)