In the Coursera ML class, Dr.Ng talked about the advanced optimization methods to find regression fits for logistic regression. He demonstrated the function fminunc in matlab; however, I wanted to find a way to implement this in R and apply it to linear regression instead.

It happens that the optim function in R do the same advanced optimization algorithm over the function cost.

First, let’s generate some data for linear regression:

set.seed(11)
x <- matrix(rnorm(400), ncol = 4) # so this is a 100x4 predictor matrix 
y <- rnorm(100)                   # this is our outcome variable, a 100x1 vector
m <- length(y)  
X<-cbind(rep(1, 100), x)      # adding a column of 1s as intercept to the design matrix X (100 x 5)
theta<-rep(0,5)               # set up the initial theta 5 x 1 vector

Next, let’s define the cost function for linear regression, this is a vectorized method:

compCost<-function(X, y, par){  
  m <- length(y)
  J <- sum((X%*%par- y)^2)/(2*m)
return(J) 
}

Then, we can use the optim function to find the best fit that minimizes the cost function. I happened to choose the “BFGS” method:

optim(par = theta, fn = compCost, X = X, y = y, method = "BFGS")
## $par
## [1]  0.04636348  0.09681103  0.10089706 -0.11856090 -0.20665898
## 
## $value
## [1] 0.5342071
## 
## $counts
## function gradient 
##        9        5 
## 
## $convergence
## [1] 0
## 
## $message
## NULL