In this problem set we’ll work out some properties of the least squares solution that we reviewed in the weekly readings. Consider the unsolvable system Ax = b as given below:
\[ \left[\begin{array}{cc} 1 & 0\\ 1 & 1\\ 1 & 3\\ 1 & 4\\ \end{array}\right] \] *
\[ \left[\begin{array}{cc} x1\\ x2\\ \end{array}\right] \]
= \[ \left[\begin{array}{cc} 0\\ 8\\ 8\\ 20\\ \end{array}\right] \]
A = matrix(c(1,1,1,1,0,1,3,4),nrow=4,ncol=2)
b = matrix(c(0,8,8,20),nrow=4,ncol=1)
#A^TA
t(A) %*% A## [,1] [,2]
## [1,] 4 8
## [2,] 8 26#A^Tb
t(A) %*% b## [,1]
## [1,] 36
## [2,] 112\(\\cap(x)\)
x_cap = Inverse(t(A) %*% A) %*% (t(A) %*% b)
#x*(x cap) after calculation
x_cap## [,1]
## [1,] 1
## [2,] 4#Projection vector
p = A %*% x_cap
#Error vector. Difference between b and Ax*
e = b - p
#sum of squared error
sum(e*e)## [1] 44#Function to calculate least squares
lsfit(A, b,intercept=FALSE)$residuals## [1] -1 3 -5 3(i.e. show that this system is solvable as all equations are consistent with each other. This should result in an error vector e = 0).
b = matrix(c(0,8,8,20),nrow=4,ncol=1)
p = matrix(c(1,5,13,17),nrow=4,ncol=1)
x_cap = Inverse(t(A) %*% A) %*% (t(A) %*% b)
#Projection vector
p2 = A %*% x_cap
#Error vector. Difference between p and p2
e = p2 - p
#sum of squared error - This is approximately 0
sum(e*e)## [1] 2.603241e-29e = b-p
#error vector between b and p(projection vector)
e## [,1]
## [1,] -1
## [2,] 3
## [3,] -5
## [4,] 3e = b-p
# e is orthogonal to p. If orthogonal p^t * e = 0
t(p) %*% e## [,1]
## [1,] 0# e is orthogonal to A. If orthogonal A^t * e = 0
t(A) %*% e## [,1]
## [1,] 0
## [2,] 0Your code should be able to print all of the above requested quantities. Please include enough comments to make it easy to follow your R markdown document.
Consider the modified auto-mpg data (obtained from the UC Irvine Machine Learning dataset). This dataset contains 5 columns: displacement, horsepower, weight, acceleration, mpg. We are going to model mpg as a function of the other four variables.
Write an R markdown script that takes in the auto-mpg data, extracts an A matrix from the first 4 columns and b vector from the fifth (mpg) column. Using the least squares approach, your code should compute the best fitting solution. That is, and the best fitting equation that expresses mpg in terms of the other 4 variables. Finally, calculate the fitting error between the predicted mpg of your model and the actual mpg. Your script should be able to load in the 5 column data set, extract A and b, and perform the rest of the calculations. Please have adequate comments in your code to make it easy to follow your work.
Please complete both problem set 1 & problem set 2 in one R markdown document and upload it to the site. You don’t have to attach the auto-mpg data. Just write your markdown document in such a way that it expects and loads the auto-mpg data file from the current working directory. As always, your code is expected to compile and run successfully. Adding test cases to demonstrate that your code is working will be very helpful.
auto = read.table("G:/Google_drive/CUNY/Courses/CUNY-repository/605 - Computational Mathematics/Wek 5 - Least Squares/data/auto-mpg.data",header = FALSE)
auto = setNames(auto, c('displacement', 'horsepower','weight', 'acceleration','mpg'))
# Matrix A
A1 = matrix(c(auto$displacement,auto$horsepower,auto$weight,auto$acceleration),nrow=392,ncol=4)
# Matrix b
b1 = matrix(c(auto$mpg),nrow=392,ncol=1)
#Transpose of A * A
ATA= t(A1) %*% A1
#Transpose of A * b
ATB= t(A1) %*% b1
#x_cap of the matrix
x_cap = inv(ATA) %*% ATB
#Final fitted line
p = A1 %*% x_cap
#error b- p
e = b1 - p
sum(e)## [1] 135.2898#validation of residuals(e)
v = lsfit(A1, b1,intercept=FALSE)$residuals
#It is almost equal to zero
sum(e-v)## [1] -3.944638e-11#Validation via Lm function from statistics
summary(lm(mpg ~ displacement + horsepower + weight + acceleration, data = auto))##
## Call:
## lm(formula = mpg ~ displacement + horsepower + weight + acceleration,
## data = auto)
##
## Residuals:
## Min 1Q Median 3Q Max
## -11.378 -2.793 -0.333 2.193 16.256
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 45.2511397 2.4560447 18.424 < 2e-16 ***
## displacement -0.0060009 0.0067093 -0.894 0.37166
## horsepower -0.0436077 0.0165735 -2.631 0.00885 **
## weight -0.0052805 0.0008109 -6.512 2.3e-10 ***
## acceleration -0.0231480 0.1256012 -0.184 0.85388
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 4.247 on 387 degrees of freedom
## Multiple R-squared: 0.707, Adjusted R-squared: 0.704
## F-statistic: 233.4 on 4 and 387 DF, p-value: < 2.2e-16Equation of mpg:
\[mpg = -0.030037938 * displacement + 0.157115685 * horsepower -0.006217883 *weight +1.997320955* acceleration \]