Activity 1.3 - covariance, correlations, and eigendecompositions

For each of the following, fully justify your answers. While I encourage you to show your math work using the “math mode” syntax, I will also accept screenshots of matrix work. To embed screenshots, enter Visual mode on your editor and simply paste the screenshot where you want it to go.

Question 1

Consider the covariance matrix from a data set of temperature and precipitation of 41 US cities:

       temp precip
temp     52     33
precip   33    139

A)

If an eigendecomposition of this matrix is done, which of these could be the pair of eigenvalues?

  1. 161 and 20
  2. 100 and 80
  3. 150 and 41
  4. 190 and 5

$$(52 + 139) = 191$$

\[150 + 41 = 191 -> iii\]

B)

What is the correlation between temperature and precipitation?

\[33 / (\sqrt52 * \sqrt 139) = .388\]

C)

Which of these could be the pair of eigenvalues from decomposing the correlation matrix?

  1. 1.39 and 0.61
  2. 1.61 and 0.53
  3. 1.12 and 0.91
  4. 1.82 and 0.43

\[1.39 + .61 = 2 -> answer = i\]

D)

In R, create the correlation matrix \(R\) and use eigen() to perform the eigendecomposition. Verify your answer to part C is correct.

R <- matrix(c(1, 0.388, 0.388, 1), nrow = 2, ncol = 2)
rownames(R) <- colnames(R) <- c("temp", "precip")
R
        temp precip
temp   1.000  0.388
precip 0.388  1.000
eig <- eigen(R)
eig$values
[1] 1.388 0.612

Question 2

A)

Consider the following matrix:

\[M = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}\]

For each of the \(v_i\) below, determine which are eigenvectors of \(M\) by checking whether \(Mv_i = \lambda v_i\) for some \(\lambda\).

  • \(v_1 = \begin{pmatrix} 2 \\ 2 \end{pmatrix}\)
  • \(v_2 = \begin{pmatrix} 1 \\ 2 \end{pmatrix}\)
  • \(v_3 = \begin{pmatrix} -3 \\ 3 \end{pmatrix}\)
  • \(v_4 = \begin{pmatrix} 2 \\ 0 \end{pmatrix}\)
  • \(v_5 = \begin{pmatrix} 3 \\ 3 \end{pmatrix}\)
  • \(v_6 = \begin{pmatrix} 1 \\ 1 \end{pmatrix}\)
  • \(v_7 = \begin{pmatrix} -1 \\ 1 \end{pmatrix}\)
  • \(v_8 = \begin{pmatrix} -1/3 \\ 1/3 \end{pmatrix}\)

\[Mv_1 = \begin{pmatrix} 2 \\ 2 \end{pmatrix} = 3v1 \] YES \[Mv_2 = \begin{pmatrix} 4 \\ 5 \end{pmatrix} \]

NO \[Mv_3 = \begin{pmatrix} -3 \\ 3 \end{pmatrix} = 1v3 \]

YES \[Mv_4 = \begin{pmatrix} 4 \\ 2 \end{pmatrix}\]

NO \[Mv_5 = \begin{pmatrix} 9 \\ 9 \end{pmatrix} = 3v5\]

YES \[Mv_6 = \begin{pmatrix} 1 \\ 1 \end{pmatrix} = v6\]

YES \[Mv_7 = \begin{pmatrix} -1 \\ 1 \end{pmatrix} = v7\]

YES \[Mv_8 = \begin{pmatrix} -1/3 \\ 1/3 \end{pmatrix} = v8\]

YES

For those that are eigenvectors, can you start to get a feel for what the eigenvalues might be by noticing common themes?

Denominator / numerator abs value = 1

B)

Find the eigenvalues of \(M\) by solving the characteristic equation:

\[ det(M - \lambda I) = 0\]

  1. Set up the matrix \(M-\lambda I\)
    \[\begin{pmatrix} 2 - \lambda & 1 \\ 1 & 2 - \lambda \end{pmatrix}\]

  2. Find the determinant of \(M-\lambda I\)

    \[(2−λ)(2−λ)−(1)(1) = (2−λ)^2−1\]

  3. For which 2 values of \(\lambda\) does the determinant = 0?

    \[(λ−1)(λ−3)=0 \] \[ λ=1orλ=3\]

Question 3

Consider the matrix

\[A = \begin{pmatrix} 3 & 2 \\ 2 & 3 \end{pmatrix}\]

The eigenvalues of this matrix are \(\lambda_1 = 5\) and \(\lambda_2 = 1\). Find the eigenbases. Are they orthogonal? Why/why not?

\(\lambda_1 = 5\) -> \[\begin{pmatrix} -2 & 2 \\ 2 & -2 \end{pmatrix} -> \begin{pmatrix} 1 \\ 1 \end{pmatrix}\]

\(\lambda_2 = 1\)

\[\begin{pmatrix} 2 & 2 \\ 2 & 2 \end{pmatrix} -> \begin{pmatrix} 1 \\ -1 \end{pmatrix}\]

dot product = 0 therefore orthogonal

Question 4

Consider the matrix

\[A = \begin{pmatrix} 1 & 2 \\ 2 & 1 \end{pmatrix}\] Suppose I tell you:

  • \(\begin{pmatrix} 2\\2 \end{pmatrix}\) is an eigenvector for \(\lambda_1\)
  • \(\begin{pmatrix} -1\\1 \end{pmatrix}\) is an eigenvector for \(\lambda_2\)

A)

Find \(\lambda_1\), \(\lambda_2\), and the eigenbases without using eigen().

\[\begin{pmatrix} 1 & 2 \\ 2 & 1 \end{pmatrix} * \begin{pmatrix} 2 \\ 2 \end{pmatrix} = \begin{pmatrix} 6 \\ 6 \end{pmatrix} -> \lambda_1= 3\]

\[\begin{pmatrix} 1 & 2 \\ 2 & 1 \end{pmatrix} * \begin{pmatrix} -1 \\ 1 \end{pmatrix} = \begin{pmatrix} 1 \\ -1 \end{pmatrix} -> \lambda_1= -1\]

B)

Use R code to create \(A\). Verify your result from A) using eigen().

A <- matrix(c(1, 2, 2, 1), nrow = 2, ncol = 2)

eig <- eigen(A)
eig$values
[1]  3 -1

Question 5

Consider the plot below:

Which of these are most likely pairs of eigenvectors corresponding to unique eigenvalues of \(\Sigma\), the covariance matrix?

A) (1,1) and (0,1)

B) (-1,1) and (1, -1)

C) (1,0) and (0,1)

D) (-1, 1) and (-1, -1)

Question 6

Consider the 6 data sets below:

Match the covariance matrix eigendecompositions with the plot it represents.

> eigen(Sigma1)
eigen() decomposition
$values
[1] 1.9 0.1

$vectors
           [,1]       [,2]
[1,] -0.7071068 -0.7071068
[2,]  0.7071068 -0.7071068

> eigen(Sigma2)
eigen() decomposition
$values
[1] 1.9 0.1

$vectors
          [,1]       [,2]
[1,] 0.7071068 -0.7071068
[2,] 0.7071068  0.7071068

> eigen(Sigma3)
eigen() decomposition
$values
[1] 6 2

$vectors
          [,1]       [,2]
[1,] 0.7071068 -0.7071068
[2,] 0.7071068  0.7071068
> eigen(Sigma4)
eigen() decomposition
$values
[1] 7.6 0.4

$vectors
          [,1]       [,2]
[1,] 0.7071068 -0.7071068
[2,] 0.7071068  0.7071068

> eigen(Sigma5)
eigen() decomposition
$values
[1] 1.5 0.5

$vectors
           [,1]       [,2]
[1,] -0.7071068 -0.7071068
[2,]  0.7071068 -0.7071068

> eigen(Sigma6)
eigen() decomposition
$values
[1] 1.5 0.5

$vectors
          [,1]       [,2]
[1,] 0.7071068 -0.7071068
[2,] 0.7071068  0.7071068

A -> Sigma2

B - > Sigma6

C -> Sigma3

D -> Sigma4

E -> Sigma5

F -> Sigma1

Question 7

For this problem, use the cars data set that comes installed with R.

A)

Create a mean-centered (but not scaled!) version of the data: call it cars_centered. Plot the data with ggplot.

library(ggplot2)

cars_centered <- as.data.frame(scale(cars, center = TRUE, scale = FALSE))

ggplot(cars_centered, aes(x = speed, y = dist)) +
  geom_point() +
  geom_hline(yintercept = 0) +
  geom_vline(xintercept = 0) +
  labs(title = "Mean-Centered Cars Data", x = "speed (centered)", y = "dist (centered)") +
  theme_minimal()

B)

Find the covariance matrix \(\Sigma_{centered}\) and the correlation matrix \(R_{centered}\) of the mean-centered data set, using cov() and cor(). Verify that \(\Sigma_{centered} = \frac{1}{n-1}X_C^T X_C\).

Sigma_centered <- cov(cars_centered)

R_centered <- cor(cars_centered)

Xc <- as.matrix(cars_centered)
n <- nrow(Xc)

Sigma_manual <- (1 / (n - 1)) * t(Xc) %*% Xc
Sigma_manual
          speed     dist
speed  27.95918 109.9469
dist  109.94694 664.0608

C)

Perform an eigendecomposition of \(\Sigma_{centered}\). Verify that the eigenvalues and the marginal variances sum to the same thing.

eig_centered <- eigen(Sigma_centered)
eig_centered$values
[1] 682.528426   9.491574
eig_centered$vectors
          [,1]       [,2]
[1,] 0.1656479 -0.9861850
[2,] 0.9861850  0.1656479
sum_eigenvalues <- sum(eig_centered$values)
sum_eigenvalues
[1] 692.02
sum_variances <- sum(diag(Sigma_centered))
sum_variances
[1] 692.02

D)

Use the eigenvectors to rotate the centered data. Create a plot of the rotated data.

Xc <- as.matrix(cars_centered)
rotated <- Xc %*% eig_centered$vectors

cars_rotated <- as.data.frame(rotated)
colnames(cars_rotated) <- c("PC1", "PC2")

ggplot(cars_rotated, aes(x = PC1, y = PC2)) +
  geom_point() +
  geom_hline(yintercept = 0) +
  geom_vline(xintercept = 0) +
  coord_fixed() +
  labs(title = "Rotated", x = "PC1", y = "PC2") +
  theme_minimal()

E)

Find the marginal variances of the rotated data, and verify that they equal the eigenvalues.

var_rotated <- apply(cars_rotated, 2, var)
var_rotated
       PC1        PC2 
682.528426   9.491574 
eig_centered$values
[1] 682.528426   9.491574

F)

Now scale the data: call it cars_scaled. Find the covariance of this data \(\Sigma_{scaled}\) and verify that it equals \(R_{centered}\).

cars_scaled <- as.data.frame(scale(cars, center = TRUE, scale = TRUE))

Sigma_scaled <- cov(cars_scaled)
Sigma_scaled
          speed      dist
speed 1.0000000 0.8068949
dist  0.8068949 1.0000000
R_centered
          speed      dist
speed 1.0000000 0.8068949
dist  0.8068949 1.0000000

G)

Rotate the scaled data using the appropriate eigenvectors, and plot the rotated data.

eig_scaled <- eigen(Sigma_scaled)

Xs <- as.matrix(cars_scaled)
rotated_scaled <- Xs %*% eig_scaled$vectors

cars_scaled_rotated <- as.data.frame(rotated_scaled)
colnames(cars_scaled_rotated) <- c("PC1", "PC2")

ggplot(cars_scaled_rotated, aes(x = PC1, y = PC2)) +
  geom_point() +
  geom_hline(yintercept = 0) +
  geom_vline(xintercept = 0) +
  coord_fixed() +
  labs(title = "Rotated", x = "PC1", y = "PC2") +
  theme_minimal()

H)

Verify that the marginal variances of the rotated data equal the eigenvalues of \(R_{centered}\) aka \(\Sigma_{scaled}\). Also verify that they sum to 2 (the diagonals of \(R_{centered}\) aka \(\Sigma_{scaled}\)).

var_rotated_scaled <- apply(cars_scaled_rotated, 2, var)
eig_scaled$values
[1] 1.8068949 0.1931051
sum(eig_scaled$values)
[1] 2
sum(diag(R_centered))
[1] 2