For each of the following, fully justify your answers. While I encourage you to show your math work using the “math mode” syntax, I will also accept screenshots of matrix work. To embed screenshots, enter Visual mode on your editor and simply paste the screenshot where you want it to go.
Consider the covariance matrix from a data set of temperature and precipitation of 41 US cities:
temp precip
temp 52 33
precip 33 139If an eigendecomposition of this matrix is done, which of these could be the pair of eigenvalues?
Answer: i. 150 and 41
What is the correlation between temperature and precipitation?
\[\rho = \frac{\text{Cov}}{\text{SD}_1 \cdot \text{SD}_2} = \frac{33}{\sqrt{52} \cdot \sqrt{139}} = \frac{33}{\sqrt{52 \times 139}} = \frac{33}{\sqrt{7228}} \approx \frac{33}{85.0176} \approx 0.3882\]
Which of these could be the pair of eigenvalues from decomposing the correlation matrix?
answer: 1.39 and 0.61
In R, create the correlation matrix \(R\) and use eigen() to perform the eigendecomposition. Verify your answer to part C is correct.
# Define the covariance matrix
Sigma <- matrix(c(52, 33, 33, 139), nrow = 2, byrow = TRUE)
# Convert covariance to correlation matrix
R <- cov2cor(Sigma)
print(R) [,1] [,2]
[1,] 1.0000000 0.3881547
[2,] 0.3881547 1.0000000# Perform eigendecomposition
decomp_R <- eigen(R)
decomp_R$values[1] 1.3881547 0.6118453Consider the following matrix:
\[M = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}\]
For each of the \(v_i\) below, determine which are eigenvectors of \(M\) by checking whether \(Mv_i = \lambda v_i\) for some \(\lambda\).
\[Mv_1 = \begin{pmatrix} 2(2) + 1(2) \\ 1(2) + 2(2) \end{pmatrix} = \begin{pmatrix} 6 \\ 6 \end{pmatrix} = 3 \begin{pmatrix} 2 \\ 2 \end{pmatrix}\]Eigenvector (with \(\lambda = 3\)). \[Mv_2 = \begin{pmatrix} 2(1) + 1(2) \\ 1(1) + 2(2) \end{pmatrix} = \begin{pmatrix} 4 \\ 5 \end{pmatrix} \neq \lambda \begin{pmatrix} 1 \\ 2 \end{pmatrix}\]Not an eigenvector. \[Mv_3 = \begin{pmatrix} 2(-3) + 1(3) \\ 1(-3) + 2(3) \end{pmatrix} = \begin{pmatrix} -3 \\ 3 \end{pmatrix} = 1 \begin{pmatrix} -3 \\ 3 \end{pmatrix}\]Eigenvector (with \(\lambda = 1\)) \[Mv_4 = \begin{pmatrix} 2(2) + 1(0) \\ 1(2) + 2(0) \end{pmatrix} = \begin{pmatrix} 4 \\ 2 \end{pmatrix} \neq \lambda \begin{pmatrix} 2 \\ 0 \end{pmatrix}\]Not an eigenvector. \[Mv_5 = \begin{pmatrix} 2(3) + 1(3) \\ 1(3) + 2(3) \end{pmatrix} = \begin{pmatrix} 9 \\ 9 \end{pmatrix} = 3 \begin{pmatrix} 3 \\ 3 \end{pmatrix}\]Eigenvector(with \(\lambda = 3\) \[Mv_6 = \begin{pmatrix} 2(1) + 1(1) \\ 1(1) + 2(1) \end{pmatrix} = \begin{pmatrix} 3 \\ 3 \end{pmatrix} = 3 \begin{pmatrix} 1 \\ 1 \end{pmatrix}\]Eigenvector (with \(\lambda = 3\)) \[Mv_7 = \begin{pmatrix} 2(-1) + 1(1) \\ 1(-1) + 2(1) \end{pmatrix} = \begin{pmatrix} -1 \\ 1 \end{pmatrix} = 1 \begin{pmatrix} -1 \\ 1 \end{pmatrix}\]Eigenvector (with \(\lambda = 1\)). \[Mv_8 = \begin{pmatrix} 2(-1/3) + 1(1/3) \\ 1(-1/3) + 2(1/3) \end{pmatrix} = \begin{pmatrix} -1/3 \\ 1/3 \end{pmatrix} = 1 \begin{pmatrix} -1/3 \\ 1/3 \end{pmatrix}\]Eigenvector (with \(\lambda = 1\)).
For those that are eigenvectors, can you start to get a feel for what the eigenvalues might be by noticing common themes?
Any vector where both numbers are identical produces an eigenvalue of \(\lambda = 3\).Any vector where the numbers are opposites produces an eigenvalue of \(\lambda = 1\).
Find the eigenvalues of \(M\) by solving the characteristic equation:
\[ det(M - \lambda I) = 0\]
Set up the matrix \(M-\lambda I\)
\[M - \lambda I = \begin{pmatrix} 2 - \lambda & 1 \\ 1 & 2 - \lambda \end{pmatrix}\]
Find the determinant of \(M-\lambda I\) \[\det(M - \lambda I) = (2 - \lambda)(2 - \lambda) - (1)(1) = (4 - 4\lambda + \lambda^2) - 1 = \lambda^2 - 4\lambda + 3\]
For which 2 values of \(\lambda\) does the determinant = 0? \[\lambda^2 - 4\lambda + 3 = (\lambda - 3)(\lambda - 1) = 0\]\[\lambda_1 = 3, \quad \lambda_2 = 1\]
Consider the matrix
\[A = \begin{pmatrix} 3 & 2 \\ 2 & 3 \end{pmatrix}\]
The eigenvalues of this matrix are \(\lambda_1 = 5\) and \(\lambda_2 = 1\). Find the eigenbases. Are they orthogonal? Why/why not?
\[\text{Eigenbase } 1 = \begin{pmatrix} 1/\sqrt{2} \\ 1/\sqrt{2} \end{pmatrix} \approx \begin{pmatrix} 0.7071 \\ 0.7071 \end{pmatrix}\]
\[\text{Eigenbase } 2 = \begin{pmatrix} -1/\sqrt{2} \\ 1/\sqrt{2} \end{pmatrix} \approx \begin{pmatrix} -0.7071 \\ 0.7071 \end{pmatrix}\]
yes they are orthogonal.
\[\begin{pmatrix} 1/\sqrt{2} \\ 1/\sqrt{2} \end{pmatrix} \cdot \begin{pmatrix} -1/\sqrt{2} \\ 1/\sqrt{2} \end{pmatrix} = \left(\frac{1}{\sqrt{2}}\right)\left(-\frac{1}{\sqrt{2}}\right) + \left(\frac{1}{\sqrt{2}}\right)\left(\frac{1}{\sqrt{2}}\right) = -\frac{1}{2} + \frac{1}{2} = 0\]
Consider the matrix
\[A = \begin{pmatrix} 1 & 2 \\ 2 & 1 \end{pmatrix}\] Suppose I tell you:
Find \(\lambda_1\), \(\lambda_2\), and the eigenbases without using eigen().
\[A v_1 = \begin{pmatrix} 1 & 2 \\ 2 & 1 \end{pmatrix} \begin{pmatrix} 2 \\ 2 \end{pmatrix} = \begin{pmatrix} 1(2) + 2(2) \\ 2(2) + 1(2) \end{pmatrix} = \begin{pmatrix} 6 \\ 6 \end{pmatrix} = 3 \begin{pmatrix} 2 \\ 2 \end{pmatrix}\]Therefore, \(\lambda_1 = 3\).Its normalized eigenbase is \(\begin{pmatrix} 1/\sqrt{2} \\ 1/\sqrt{2} \end{pmatrix}\).
\[A v_2 = \begin{pmatrix} 1 & 2 \\ 2 & 1 \end{pmatrix} \begin{pmatrix} -1 \\ 1 \end{pmatrix} = \begin{pmatrix} 1(-1) + 2(1) \\ 2(-1) + 1(1) \end{pmatrix} = \begin{pmatrix} 1 \\ -1 \end{pmatrix} = -1 \begin{pmatrix} -1 \\ 1 \end{pmatrix}\]Therefore, \(\lambda_2 = -1\).Its normalized eigenbase is \(\begin{pmatrix} -1/\sqrt{2} \\ 1/\sqrt{2} \end{pmatrix}\).
Use R code to create \(A\). Verify your result from A) using eigen().
A <- matrix(c(1, 2, 2, 1), nrow = 2, byrow = TRUE)
decomp <- eigen(A)
decomp$values[1] 3 -1decomp$vectors [,1] [,2]
[1,] 0.7071068 -0.7071068
[2,] 0.7071068 0.7071068Consider the plot below:
Which of these are most likely pairs of eigenvectors corresponding to unique eigenvalues of \(\Sigma\), the covariance matrix?
A) (1,1) and (0,1)
B) (-1,1) and (1, -1)
C) (1,0) and (0,1)
D) (-1, 1) and (-1, -1)
Answer: D) (-1,1) and (-1,-1)
Consider the 6 data sets below:
Match the covariance matrix eigendecompositions with the plot it represents.
> eigen(Sigma1)
eigen() decomposition
$values
[1] 1.9 0.1
$vectors
[,1] [,2]
[1,] -0.7071068 -0.7071068
[2,] 0.7071068 -0.7071068
> eigen(Sigma2)
eigen() decomposition
$values
[1] 1.9 0.1
$vectors
[,1] [,2]
[1,] 0.7071068 -0.7071068
[2,] 0.7071068 0.7071068
> eigen(Sigma3)
eigen() decomposition
$values
[1] 6 2
$vectors
[,1] [,2]
[1,] 0.7071068 -0.7071068
[2,] 0.7071068 0.7071068> eigen(Sigma4)
eigen() decomposition
$values
[1] 7.6 0.4
$vectors
[,1] [,2]
[1,] 0.7071068 -0.7071068
[2,] 0.7071068 0.7071068
> eigen(Sigma5)
eigen() decomposition
$values
[1] 1.5 0.5
$vectors
[,1] [,2]
[1,] -0.7071068 -0.7071068
[2,] 0.7071068 -0.7071068
> eigen(Sigma6)
eigen() decomposition
$values
[1] 1.5 0.5
$vectors
[,1] [,2]
[1,] 0.7071068 -0.7071068
[2,] 0.7071068 0.7071068Plot A: Sigma2 Plot B: Sigma6 Plot C: Sigma3 Plot D: Sigma4 Plot E: Sigma5 Plot F: Sigma1
For this problem, use the cars data set that comes installed with R.
Create a mean-centered (but not scaled!) version of the data: call it cars_centered. Plot the data with ggplot.
Find the covariance matrix \(\Sigma_{centered}\) and the correlation matrix \(R_{centered}\) of the mean-centered data set, using cov() and cor(). Verify that \(\Sigma_{centered} = \frac{1}{n-1}X_C^T X_C\).
Perform an eigendecomposition of \(\Sigma_{centered}\). Verify that the eigenvalues and the marginal variances sum to the same thing.
Use the eigenvectors to rotate the centered data. Create a plot of the rotated data.
Find the marginal variances of the rotated data, and verify that they equal the eigenvalues.
Now scale the data: call it cars_scaled. Find the covariance of this data \(\Sigma_{scaled}\) and verify that it equals \(R_{centered}\).
Rotate the scaled data using the appropriate eigenvectors, and plot the rotated data.
Verify that the marginal variances of the rotated data equal the eigenvalues of \(R_{centered}\) aka \(\Sigma_{scaled}\). Also verify that they sum to 2 (the diagonals of \(R_{centered}\) aka \(\Sigma_{scaled}\)).
library(ggplot2)
data(cars)
n <- nrow(cars)
# -----------------------------------------------------------
# Part A: Mean-center data and plot
# -----------------------------------------------------------
cars_centered <- as.data.frame(scale(cars, center = TRUE, scale = FALSE))
ggplot(cars_centered, aes(x = speed, y = dist)) +
geom_point(color = "steelblue", size = 2) +
geom_hline(yintercept = 0, linetype = "dashed") +
geom_vline(xintercept = 0, linetype = "dashed") +
theme_minimal() +
labs(title = "Part A: Mean-Centered Cars Data", x = "Centered Speed", y = "Centered Dist")
# -----------------------------------------------------------
# Part B: Covariance & Correlation matrices + Formula check
# -----------------------------------------------------------
Sigma_centered <- cov(cars_centered)
R_centered <- cor(cars_centered)
Xc <- as.matrix(cars_centered)
Sigma_formula <- (1 / (n - 1)) * (t(Xc) %*% Xc)
cat("Sigma from cov():\n")Sigma from cov():print(Sigma_centered) speed dist
speed 27.95918 109.9469
dist 109.94694 664.0608cat("\nSigma from (1/(n-1)) Xc^T Xc:\n")
Sigma from (1/(n-1)) Xc^T Xc:print(Sigma_formula) speed dist
speed 27.95918 109.9469
dist 109.94694 664.0608cat("\nAre they identical?", all.equal(Sigma_centered, Sigma_formula), "\n")
Are they identical? TRUE # -----------------------------------------------------------
# Part C: Eigendecomposition and sum comparison
# -----------------------------------------------------------
decomp_cov <- eigen(Sigma_centered)
cat("\nEigenvalues of Sigma_centered:\n")
Eigenvalues of Sigma_centered:print(decomp_cov$values)[1] 682.528426 9.491574cat("Sum of eigenvalues:", sum(decomp_cov$values), "\n")Sum of eigenvalues: 692.02 cat("Sum of marginal variances (trace):", sum(diag(Sigma_centered)), "\n")Sum of marginal variances (trace): 692.02 cat("Are they equal?", all.equal(sum(decomp_cov$values), sum(diag(Sigma_centered))), "\n")Are they equal? TRUE # -----------------------------------------------------------
# Part D: Rotate centered data and plot
# -----------------------------------------------------------
V <- decomp_cov$vectors
cars_rotated <- as.data.frame(as.matrix(cars_centered) %*% V)
colnames(cars_rotated) <- c("Axis1", "Axis2")
ggplot(cars_rotated, aes(x = Axis1, y = Axis2)) +
geom_point(color = "firebrick", size = 2) +
geom_hline(yintercept = 0, linetype = "dashed") +
geom_vline(xintercept = 0, linetype = "dashed") +
theme_minimal() +
labs(title = "Part D: Rotated Centered Cars Data", x = "First Principal Direction", y = "Second Principal Direction")
# -----------------------------------------------------------
# Part E: Verify marginal variances of rotated data
# -----------------------------------------------------------
rotated_variances <- apply(cars_rotated, 2, var)
cat("\nVariances of rotated columns:\n")
Variances of rotated columns:print(rotated_variances) Axis1 Axis2
682.528426 9.491574 cat("Eigenvalues:\n")Eigenvalues:print(decomp_cov$values)[1] 682.528426 9.491574cat("Are they equal?", all.equal(unname(rotated_variances), decomp_cov$values), "\n")Are they equal? TRUE # -----------------------------------------------------------
# Part F: Scale data and verify Sigma_scaled == R_centered
# -----------------------------------------------------------
cars_scaled <- as.data.frame(scale(cars, center = TRUE, scale = TRUE))
Sigma_scaled <- cov(cars_scaled)
cat("\nCovariance of scaled data:\n")
Covariance of scaled data:print(Sigma_scaled) speed dist
speed 1.0000000 0.8068949
dist 0.8068949 1.0000000cat("\nCorrelation of centered data:\n")
Correlation of centered data:print(R_centered) speed dist
speed 1.0000000 0.8068949
dist 0.8068949 1.0000000cat("Are they equal?", all.equal(Sigma_scaled, R_centered), "\n")Are they equal? TRUE # -----------------------------------------------------------
# Part G: Rotate scaled data and plot
# -----------------------------------------------------------
decomp_scaled <- eigen(Sigma_scaled)
cars_scaled_rotated <- as.data.frame(as.matrix(cars_scaled) %*% decomp_scaled$vectors)
colnames(cars_scaled_rotated) <- c("Axis1", "Axis2")
ggplot(cars_scaled_rotated, aes(x = Axis1, y = Axis2)) +
geom_point(color = "forestgreen", size = 2) +
geom_hline(yintercept = 0, linetype = "dashed") +
geom_vline(xintercept = 0, linetype = "dashed") +
theme_minimal() +
labs(title = "Part G: Rotated Scaled Cars Data", x = "Scaled PC 1", y = "Scaled PC 2")
# -----------------------------------------------------------
# Part H: Verify marginal variances of scaled rotated data
# -----------------------------------------------------------
scaled_rotated_variances <- apply(cars_scaled_rotated, 2, var)
cat("\nVariances of scaled rotated data:\n")
Variances of scaled rotated data:print(scaled_rotated_variances) Axis1 Axis2
1.8068949 0.1931051 cat("Eigenvalues of scaled covariance:\n")Eigenvalues of scaled covariance:print(decomp_scaled$values)[1] 1.8068949 0.1931051cat("Are they equal?", all.equal(unname(scaled_rotated_variances), decomp_scaled$values), "\n")Are they equal? TRUE cat("Sum of marginal variances:", sum(scaled_rotated_variances), "\n")Sum of marginal variances: 2 cat("Sum of diagonal of Sigma_scaled (Trace):", sum(diag(Sigma_scaled)), "\n")Sum of diagonal of Sigma_scaled (Trace): 2