A note on Linear Algebra

Eigenvalue and Eigenvector

Introduction

This blog discusses some of properties of the eigenvalues and eigenvectors of square matrix. Some of the mathematics parts covered in the field of (Bayesian) statistics and in many classic, rigorous textbook (for example Mike West book or DeGroot) are heavy in linear algebra. Hence, this post is served as a revision for me and all you guys who want to venture on the path of being a quantitative-approach persons.

P/S: I will update this blog weekly or if I finish all the urgent deadlines (((=

Properties

Eigenvalues and Eigenventors of the Inverse matrix

Proposition. Let \(A\) be a \(N \times N\) invertible matrix. Then \(\lambda\) is an eigenvalue of \(A\) corresponding to an eigenvector \(x\) if and only of \(\lambda^{-1}\) is an eigenvalue of \(A^{-1}\) corresponding to the same eigenvector \(x\)

Proof: By definition, a scalar \(\lambda\) is an eigenvalue of a matrix \(A\) corresponding to an eigenvector \(x\) if and only if \[ Ax = \lambda x\]

Since A is invertible with \(\lambda \neq 0\), we can multiply both side of the above equation by a quantity \(\lambda^{-1} A^{-1}\) to obtain \[ \lambda^{-1} A^{-1} Ax = \lambda^{-1} A^{-1} \lambda x\] or \[ A^{-1} x = \lambda^{-1} x\]

using the associative property of matrix multiplication. Thus, the above relation is only true if and only if \(\lambda^{-1}\) is an eigenvalue of \(A^{-1}\) associated to the eigenvector \(x\).

Here is an example of such case

import numpy as np
from numpy.linalg import eig, inv
A = np.array([[1, 0, 0], 
              [0, 2, 2],
              [1, 0, 3]])

w, v = eig(A)

A_inv = inv(A)
w_inv, v_inv = eig(A_inv)

print(f"Eigenvalues of the original matrix: {w} \n")

## Eigenvalues of the original matrix: [2. 3. 1.]

print(f"Eigenvalues of the inverse matrix: {w_inv} \n")

## Eigenvalues of the inverse matrix: [0.5        0.33333333 1.        ]

assert all(1/w == w_inv)

Eigenvalues and Eigenvectors of a Matrix plus some constant c

Proposition. Let \(A\) be a \(N \times N\) invertible matrix. Then \(\lambda\) is an eigenvalue of \(A\) corresponding to an eigenvector \(x\). Proof that \(x\) is also the corresponding eigenvector of the matrix \(A + cI\) with \(\lambda + c\) being the eigenvalue, where \(c\) is a non-zero constant

Proof: By definition, a scalar \(\lambda\) is an eigenvalue of a matrix \(A\) corresponding to an eigenvector \(x\) if and only if \[ Ax = \lambda x\]

Moreover, \(cI x = cv\) for some constant \(c\) and identity matrix \(I\). Then, \[ \begin{aligned} Ax cIx & = \lambda x c x \\ (A + cI) x & = (\lambda + c) x \end{aligned} \] Then, by definition, \(x\) is also the eigenvector of the matrix \(A + cI\) with its corresponding eigenvalue \(\lambda + c\).

Here is an example of such case

A = np.array([[1, 0, 0], 
              [0, 2, 2],
              [1, 0, 3]])
A_c = A + np.eye(3)
w_c, v_c = eig(A_c)
print(f"Eigenvalues of the matrix plus constant c: {w_c} \n")

## Eigenvalues of the matrix plus constant c: [3. 4. 2.]

print(f"Eigenvalues of the original matrix: {w} \n")

## Eigenvalues of the original matrix: [2. 3. 1.]

print(f"Eigenvectors of the matrix plus constant c: {v_c} \n")

## Eigenvectors of the matrix plus constant c: [[ 0.          0.          0.66666667]
##  [ 1.          0.89442719  0.66666667]
##  [ 0.          0.4472136  -0.33333333]]

print(f"Eigenvectors of the original matrix: {v} \n")

## Eigenvectors of the original matrix: [[ 0.          0.          0.66666667]
##  [ 1.          0.89442719  0.66666667]
##  [ 0.          0.4472136  -0.33333333]]