Description of the Project

The following project is a work for Machine Learning course (DAT 315). It will solve a series of problems. All exercises have been made by Angel Sosa.

Rmarkdown Cache

Library

library(gbm)
library(mlbench)
library(e1071)
library(klaR)
library(ISLR)
library(data.table)
library(kernlab)
library(fmsb)
library(car)
library(stats)
library(leaps)
library(earth)
library(class)
library(caret)
library(MASS)
library(DMwR)
library(quadprog)
library(AppliedPredictiveModeling)
library(caretEnsemble)
library(neuralnet)
library(ROCR)
library(graphics)
library(randomForest)
library(rpart)

Problem 1

In the following exercises, I’m going to make use of functions to improve my way to build and compare models.

(A)

Making use of some functions, we have to remove any columns with zero variance and create our testing and training data.

ionos <- read.csv("C:/Users/cat_b/Desktop/Elizabethtown College/Machine Learning/Data/ionosphere.txt")
ionos <- ionos[,-nearZeroVar(ionos)]
ionos$X1 <- sapply(ionos$X1, as.numeric)
set.seed(12345)
ionosindex <- createDataPartition(ionos$g,p = 0.7, list = FALSE, times = 1)
ionostrain <- ionos[ionosindex,]
ionostest <- ionos[-ionosindex,]

(B)

Once we finished the previous exercises, now we have to make use of the function caretList. How it will help us? It will allow us to to build and compare more than one model at a time.

models1 <- caretList(g~. ,data=ionostrain, trControl=trainControl(method="cv", number=10, savePredictions = TRUE, classProbs = TRUE), methodList=c('knn', 'lda', 'rpart'))
results <- resamples(models1)
summary(results)
## 
## Call:
## summary.resamples(object = results)
## 
## Models: knn, lda, rpart 
## Number of resamples: 10 
## 
## Accuracy 
##       Min. 1st Qu. Median   Mean 3rd Qu. Max. NA's
## knn   0.75  0.8438 0.8800 0.8737  0.9200 0.96    0
## lda   0.76  0.8100 0.8575 0.8663  0.9075 1.00    0
## rpart 0.80  0.8762 0.9183 0.8983  0.9200 0.96    0
## 
## Kappa 
##         Min. 1st Qu. Median   Mean 3rd Qu.   Max. NA's
## knn   0.3846  0.6359 0.7261 0.7000  0.8175 0.9110    0
## lda   0.4231  0.5333 0.6648 0.6818  0.7902 1.0000    0
## rpart 0.5955  0.7211 0.8175 0.7787  0.8317 0.9153    0
dotplot(results)

On the chart above, we can visualize our models, either accuracy or kappa.

(C)

I’m going to use the previous code but now without the model lda. Also, in this exercise I’m comparing the results between the models.

modelCor(results)
##               knn        lda       rpart
## knn    1.00000000  0.7122985 -0.08362462
## lda    0.71229850  1.0000000 -0.29307963
## rpart -0.08362462 -0.2930796  1.00000000
models2 <- caretList(g~. ,data=ionostrain, trControl=trainControl(method="cv", number=10, savePredictions = TRUE, classProbs = TRUE), methodList=c('knn', 'rpart'))
results2 <- resamples(models2)
summary(results2)
## 
## Call:
## summary.resamples(object = results2)
## 
## Models: knn, rpart 
## Number of resamples: 10 
## 
## Accuracy 
##         Min. 1st Qu. Median   Mean 3rd Qu. Max. NA's
## knn   0.7917  0.8400 0.8723 0.8736    0.88 0.96    0
## rpart 0.7600  0.8709 0.9200 0.9018    0.95 1.00    0
## 
## Kappa 
##         Min. 1st Qu. Median   Mean 3rd Qu.  Max. NA's
## knn   0.5000  0.6154 0.6996 0.7020  0.7296 0.911    0
## rpart 0.4792  0.7253 0.8175 0.7846  0.8877 1.000    0
dotplot(results2)

modelCor(results2)
##             knn     rpart
## knn   1.0000000 0.3623974
## rpart 0.3623974 1.0000000

Why am I using modelCor function? It allows me to examine the correlations between the predictions of the different models. Therefore when I’m not using lda model, my matrix changes. Making a noticeable difference of results.

(D)

Now, I’m going to use the caretStack function to build a combined model. By combined I mean a model which combines the predictions from different models.

stacks <- caretStack(models1, method="glm", metric="Accuracy",trControl=trainControl(method="cv", number=10, savePredictions = TRUE, classProbs = TRUE))
stacks
## A glm ensemble of 2 base models: knn, lda, rpart
## 
## Ensemble results:
## Generalized Linear Model 
## 
## 246 samples
##   3 predictor
##   2 classes: 'b', 'g' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold) 
## Summary of sample sizes: 221, 221, 221, 222, 223, 221, ... 
## Resampling results:
## 
##   Accuracy   Kappa    
##   0.9309855  0.8468774
## 
##

How does it help to my model or results? Well, we can see how this function improved the model by comparing the accuracy of kNN and logistic regression models on the training data. The model in this excercise has a better accuracy than the other two models, thus, is a better option.

(E)

How does the accuracy of the stacked model compare to the accuracies of the separate kNN and CART models on the testing data?

models2 <- caretList(g~. ,data=ionostest, trControl=trainControl(method="cv", number=10, savePredictions = TRUE, classProbs = TRUE), methodList=c('knn', 'lda', 'rpart'))
results <- resamples(models2)
summary(results)
## 
## Call:
## summary.resamples(object = results)
## 
## Models: knn, lda, rpart 
## Number of resamples: 10 
## 
## Accuracy 
##       Min. 1st Qu. Median   Mean 3rd Qu. Max. NA's
## knn    0.6  0.6523    0.7 0.7218  0.7818  0.9    0
## lda    0.6  0.7455    0.8 0.8064  0.8182  1.0    0
## rpart  0.5  0.8182    0.9 0.8464  0.9000  1.0    0
## 
## Kappa 
##           Min. 1st Qu. Median   Mean 3rd Qu.   Max. NA's
## knn    0.00000  0.0000 0.1429 0.2426  0.4836 0.7368    0
## lda    0.00000  0.3860 0.5346 0.5373  0.5954 1.0000    0
## rpart -0.08696  0.6071 0.7597 0.6582  0.7900 1.0000    0
stacks2 <- caretStack(models2, method="glm", metric="Accuracy",trControl=trainControl(method="cv", number=10, savePredictions = TRUE, classProbs = TRUE))
stacks2
## A glm ensemble of 2 base models: knn, lda, rpart
## 
## Ensemble results:
## Generalized Linear Model 
## 
## 104 samples
##   3 predictor
##   2 classes: 'b', 'g' 
## 
## No pre-processing
## Resampling: Cross-Validated (10 fold) 
## Summary of sample sizes: 95, 94, 94, 93, 93, 94, ... 
## Resampling results:
## 
##   Accuracy   Kappa    
##   0.8681818  0.7039061
## 
##

The accuracy of the stacked model is even better by far than the other model using testing data.

Problem 2

In this part of the project I’m going to use four models to compare their accuracy given a testing and a training data. A remarkable aspect is the time it takes to build the models and the amount of results that you can get of them.

(A)

I created the data frames to get the testing and training data.

data("Khan")
khan <- Khan
khantrain<-as.data.frame(khan$xtrain)
khantest<-as.data.frame(khan$xtest)
khantrain$response<-as.factor(khan$ytrain)
khantest$response<-as.factor(khan$ytest)
set.seed(12345)

(B)

By using the function train I can build all the models.

CART2 <- train(response~., data=khantrain, method="rpart")
RF2 <- train(response~., data=khantrain, method="rf",preProcess=c("center","scale", "pca"), importance=TRUE)
GBM2 <- train(response~., data=khantrain, method="gbm")
SVM2 <- train(response~., data=khantrain, method="svmLinear")

(C)

In this exercise I’m going to compare the accuracy of the models using training data.

predictedcart2 <- predict(CART2, newdata=khantrain)
CM1 <- confusionMatrix(predictedcart2, khantrain$response)
CM1
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction  1  2  3  4
##          1  0  0  0  0
##          2  0 22  0  0
##          3  7  1 12  0
##          4  1  0  0 20
## 
## Overall Statistics
##                                           
##                Accuracy : 0.8571          
##                  95% CI : (0.7461, 0.9325)
##     No Information Rate : 0.3651          
##     P-Value [Acc > NIR] : 1.023e-15       
##                                           
##                   Kappa : 0.7977          
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 3 Class: 4
## Sensitivity             0.000   0.9565   1.0000   1.0000
## Specificity             1.000   1.0000   0.8431   0.9767
## Pos Pred Value            NaN   1.0000   0.6000   0.9524
## Neg Pred Value          0.873   0.9756   1.0000   1.0000
## Prevalence              0.127   0.3651   0.1905   0.3175
## Detection Rate          0.000   0.3492   0.1905   0.3175
## Detection Prevalence    0.000   0.3492   0.3175   0.3333
## Balanced Accuracy       0.500   0.9783   0.9216   0.9884
predictedrf2 <- predict(RF2, newdata=khantrain)
CM2 <- confusionMatrix(predictedrf2, khantrain$response)
CM2
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction  1  2  3  4
##          1  8  0  0  0
##          2  0 23  0  0
##          3  0  0 12  0
##          4  0  0  0 20
## 
## Overall Statistics
##                                      
##                Accuracy : 1          
##                  95% CI : (0.9431, 1)
##     No Information Rate : 0.3651     
##     P-Value [Acc > NIR] : < 2.2e-16  
##                                      
##                   Kappa : 1          
##  Mcnemar's Test P-Value : NA         
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 3 Class: 4
## Sensitivity             1.000   1.0000   1.0000   1.0000
## Specificity             1.000   1.0000   1.0000   1.0000
## Pos Pred Value          1.000   1.0000   1.0000   1.0000
## Neg Pred Value          1.000   1.0000   1.0000   1.0000
## Prevalence              0.127   0.3651   0.1905   0.3175
## Detection Rate          0.127   0.3651   0.1905   0.3175
## Detection Prevalence    0.127   0.3651   0.1905   0.3175
## Balanced Accuracy       1.000   1.0000   1.0000   1.0000
predictedgbm2 <- predict(GBM2, newdata=khantrain)
CM3 <- confusionMatrix(predictedgbm2, khantrain$response)
CM3
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction  1  2  3  4
##          1  8  0  0  0
##          2  0 23  0  0
##          3  0  0 12  0
##          4  0  0  0 20
## 
## Overall Statistics
##                                      
##                Accuracy : 1          
##                  95% CI : (0.9431, 1)
##     No Information Rate : 0.3651     
##     P-Value [Acc > NIR] : < 2.2e-16  
##                                      
##                   Kappa : 1          
##  Mcnemar's Test P-Value : NA         
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 3 Class: 4
## Sensitivity             1.000   1.0000   1.0000   1.0000
## Specificity             1.000   1.0000   1.0000   1.0000
## Pos Pred Value          1.000   1.0000   1.0000   1.0000
## Neg Pred Value          1.000   1.0000   1.0000   1.0000
## Prevalence              0.127   0.3651   0.1905   0.3175
## Detection Rate          0.127   0.3651   0.1905   0.3175
## Detection Prevalence    0.127   0.3651   0.1905   0.3175
## Balanced Accuracy       1.000   1.0000   1.0000   1.0000
predictedsvm2 <- predict(SVM2, newdata=khantrain)
CM4 <- confusionMatrix(predictedsvm2, khantrain$response)
CM4
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction  1  2  3  4
##          1  8  0  0  0
##          2  0 23  0  0
##          3  0  0 12  0
##          4  0  0  0 20
## 
## Overall Statistics
##                                      
##                Accuracy : 1          
##                  95% CI : (0.9431, 1)
##     No Information Rate : 0.3651     
##     P-Value [Acc > NIR] : < 2.2e-16  
##                                      
##                   Kappa : 1          
##  Mcnemar's Test P-Value : NA         
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 3 Class: 4
## Sensitivity             1.000   1.0000   1.0000   1.0000
## Specificity             1.000   1.0000   1.0000   1.0000
## Pos Pred Value          1.000   1.0000   1.0000   1.0000
## Neg Pred Value          1.000   1.0000   1.0000   1.0000
## Prevalence              0.127   0.3651   0.1905   0.3175
## Detection Rate          0.127   0.3651   0.1905   0.3175
## Detection Prevalence    0.127   0.3651   0.1905   0.3175
## Balanced Accuracy       1.000   1.0000   1.0000   1.0000

Using training data, just one out of the four models has an accuracy less than 1. There’s no difference with the other three. ##(D)

predictedcart21 <- predict(CART2, newdata=khantest)
CM12 <- confusionMatrix(predictedcart21, khantest$response)
CM12
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction 1 2 3 4
##          1 0 0 0 0
##          2 0 4 0 1
##          3 3 1 5 1
##          4 0 1 1 3
## 
## Overall Statistics
##                                           
##                Accuracy : 0.6             
##                  95% CI : (0.3605, 0.8088)
##     No Information Rate : 0.3             
##     P-Value [Acc > NIR] : 0.005138        
##                                           
##                   Kappa : 0.4386          
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 3 Class: 4
## Sensitivity              0.00   0.6667   0.8333   0.6000
## Specificity              1.00   0.9286   0.6429   0.8667
## Pos Pred Value            NaN   0.8000   0.5000   0.6000
## Neg Pred Value           0.85   0.8667   0.9000   0.8667
## Prevalence               0.15   0.3000   0.3000   0.2500
## Detection Rate           0.00   0.2000   0.2500   0.1500
## Detection Prevalence     0.00   0.2500   0.5000   0.2500
## Balanced Accuracy        0.50   0.7976   0.7381   0.7333
predictedrf22 <- predict(RF2, newdata=khantest)
CM22 <- confusionMatrix(predictedrf22, khantest$response)
CM22
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction 1 2 3 4
##          1 0 0 0 0
##          2 0 5 0 2
##          3 0 0 0 0
##          4 3 1 6 3
## 
## Overall Statistics
##                                           
##                Accuracy : 0.4             
##                  95% CI : (0.1912, 0.6395)
##     No Information Rate : 0.3             
##     P-Value [Acc > NIR] : 0.2277          
##                                           
##                   Kappa : 0.1809          
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 3 Class: 4
## Sensitivity              0.00   0.8333      0.0   0.6000
## Specificity              1.00   0.8571      1.0   0.3333
## Pos Pred Value            NaN   0.7143      NaN   0.2308
## Neg Pred Value           0.85   0.9231      0.7   0.7143
## Prevalence               0.15   0.3000      0.3   0.2500
## Detection Rate           0.00   0.2500      0.0   0.1500
## Detection Prevalence     0.00   0.3500      0.0   0.6500
## Balanced Accuracy        0.50   0.8452      0.5   0.4667
predictedgbm23 <- predict(GBM2, newdata=khantest)
CM32 <- confusionMatrix(predictedgbm23, khantest$response)
CM32
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction 1 2 3 4
##          1 3 0 0 0
##          2 0 5 0 0
##          3 0 0 6 0
##          4 0 1 0 5
## 
## Overall Statistics
##                                           
##                Accuracy : 0.95            
##                  95% CI : (0.7513, 0.9987)
##     No Information Rate : 0.3             
##     P-Value [Acc > NIR] : 1.662e-09       
##                                           
##                   Kappa : 0.9322          
##  Mcnemar's Test P-Value : NA              
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 3 Class: 4
## Sensitivity              1.00   0.8333      1.0   1.0000
## Specificity              1.00   1.0000      1.0   0.9333
## Pos Pred Value           1.00   1.0000      1.0   0.8333
## Neg Pred Value           1.00   0.9333      1.0   1.0000
## Prevalence               0.15   0.3000      0.3   0.2500
## Detection Rate           0.15   0.2500      0.3   0.2500
## Detection Prevalence     0.15   0.2500      0.3   0.3000
## Balanced Accuracy        1.00   0.9167      1.0   0.9667
predictedsvm24 <- predict(SVM2, newdata=khantest)
CM42 <- confusionMatrix(predictedsvm24, khantest$response)
CM42
## Confusion Matrix and Statistics
## 
##           Reference
## Prediction 1 2 3 4
##          1 3 0 0 0
##          2 0 6 2 0
##          3 0 0 4 0
##          4 0 0 0 5
## 
## Overall Statistics
##                                          
##                Accuracy : 0.9            
##                  95% CI : (0.683, 0.9877)
##     No Information Rate : 0.3            
##     P-Value [Acc > NIR] : 3.773e-08      
##                                          
##                   Kappa : 0.8639         
##  Mcnemar's Test P-Value : NA             
## 
## Statistics by Class:
## 
##                      Class: 1 Class: 2 Class: 3 Class: 4
## Sensitivity              1.00   1.0000   0.6667     1.00
## Specificity              1.00   0.8571   1.0000     1.00
## Pos Pred Value           1.00   0.7500   1.0000     1.00
## Neg Pred Value           1.00   1.0000   0.8750     1.00
## Prevalence               0.15   0.3000   0.3000     0.25
## Detection Rate           0.15   0.3000   0.2000     0.25
## Detection Prevalence     0.15   0.4000   0.2000     0.25
## Balanced Accuracy        1.00   0.9286   0.8333     1.00

Using testing data, GBM2 model got an accuracy of 0.95, then SVM2 an accuracy of 0.90. Finally, 0.65 and 0.55 for RF2 and CART2. Looking at this data we can agreed that the GBM2 has the best accuracy.

Problem 3 Finished

One of the main topics in machine learning are the models, however this time I’m going to work with a neuronal network

(A)

I have to normalize the data in order to be used. Also, get the testing and training data.

enefficiency <- read.csv("C:/Users/cat_b/Desktop/Elizabethtown College/Machine Learning/Data/enefficiency.csv")
colnames(enefficiency)[1] <- "X1"
normalize <- function(x){
  return((x-min(x))/(max(x)-min(x)))
}
enefficiency <- as.data.frame(lapply(enefficiency,normalize))
set.seed(12345)
trainindex3 <- createDataPartition(y=enefficiency$Y1+enefficiency$Y2, p = 0.7, list = FALSE)
enefficiencytrain <- enefficiency[trainindex3,]
enefficiencytest <- enefficiency[-trainindex3,]

(B)

I’m going to use the function train to build the neural network.

NN3b <- train(Y1~., data=enefficiencytrain, method="nnet",trControl=trainControl(method = "cv", number = 10))

(C)

Computing R^2 for the model.

nn3bpredicted <- predict(NN3b, newdata=enefficiencytest)
NN3b_R2_test <- cor(nn3bpredicted, enefficiencytest$Y1)^2
NN3b_R2_test
##           [,1]
## [1,] 0.9849403

(D)

In order to predict two responses, I’m going to build another model (neural network). As a requirement, I have to build the neural network with one hidden unit in one hidden layer.

NN3d <- neuralnet(Y1+Y2~X1+X2+X3+X4+X5+X6+X7+X8, data = enefficiencytrain, hidden=c(1))
plot(NN3d, rep="best")

nn3dpredicted <- compute(NN3d, enefficiencytest[,1:8])
NN3d_R2_testY1 <- cor(nn3dpredicted$net.result, enefficiencytest$Y1)^2
NN3d_R2_testY2 <- cor(nn3dpredicted$net.result, enefficiencytest$Y2)^2
NN3d_R2_testY1
##             [,1]
## [1,] 0.914034603
## [2,] 0.914034603
NN3d_R2_testY2
##              [,1]
## [1,] 0.8834792161
## [2,] 0.8834792161

As we can see the R^2 for each response varies.

(E)

Using the previous codes, I’m going to build another neural network, but this time with with two hidden layers, the first having 2 nodes and the second having one nodes. The nodes are important for the neural network model and have them could bea big difference in our results.

NN3e <- neuralnet(Y1+Y2~X1+X2+X3+X4+X5+X6+X7+X8, data = enefficiencytrain, hidden=c(2,1), linear.output = T)
plot(NN3e, rep="best")

nn3epredicted <-compute(NN3e, enefficiencytest[,1:8])
NN3e_R2_testY1 <- cor(nn3epredicted$net.result, enefficiencytest$Y1)^2
NN3e_R2_testY2 <- cor(nn3epredicted$net.result, enefficiencytest$Y2)^2
NN3e_R2_testY1
##              [,1]
## [1,] 0.9743982708
## [2,] 0.9743982708
NN3e_R2_testY2
##              [,1]
## [1,] 0.9462896524
## [2,] 0.9462896524

Problem 4

In this final exercise I’m going to build more neural network models.The importance between use normalize data or do not use it, will be covered here.

(A)

First, we have to get our data. Creating the partition and setting the seed are the first step of our exercise.

data("Boston")
Boston <- Boston[,13:14]
set.seed(12345)
trainindex4 <- createDataPartition(y=Boston$medv, p = 0.7, list = FALSE)
Bostontrain <- Boston[trainindex4,]
Bostontest <- Boston[-trainindex4,]

(B)

I’m building a model with one hiddem layer containningone hidden variable.

NN4b <- neuralnet(medv~lstat, data = Bostontrain)
plot(NN4b)
plot(Boston$lstat,Boston$medv)
par(new=TRUE)
curve(NN4b$result.matrix[6,1]+NN4b$result.matrix[7,1]*tanh(NN4b$result.matrix[4,1]+NN4b$result.matrix[5,1]*x),0,1, axes=F, xlab="", ylab="")
par(new=FALSE)

The plot of the curve is a little bit strange because there’s no curve. By annalyzing the plot of my neural network, I realized that I have one predictor to one response. Fact which makes my curve have more sense. By having just one predictor, one node and one output, the quality of the model is fine but simple. This model has been built with non-normalize data.

(C)

Now, I’m going to normalize my data. Then, I’m going to build another neural network modelto compare the results.

normalize <- function(x){
  return((x-min(x))/(max(x)-min(x)))
}
Boston2 <- as.data.frame(lapply(Boston,normalize))

(D)

With the previous data I’m building another nnetwork model.

set.seed(12345)
trainindex41 <- createDataPartition(y=Boston2$medv, p = 0.7, list = FALSE)
Boston2train <- Boston2[trainindex41,]
Boston2test <- Boston2[-trainindex41,]
NN4d <- neuralnet(medv~lstat, data = Boston2train, hidden=c(1), linear.output = T)
plot(Boston2$lstat,Boston2$medv)
par(new=TRUE)
curve(NN4d$result.matrix[6,1]+NN4d$result.matrix[7,1]*tanh(NN4d$result.matrix[4,1]+NN4d$result.matrix[5,1]*x),0,1, axes=F, xlab="", ylab="", col= "red")

par(new=FALSE)

The quality of the model is better than the model built with non-normalize data.However, the curve is still looking with no changes. It could be the fact of just have one predictor but it’s almost the same than the previous curve. The formula is the same.

(E)

Here we can visualize the model. It looks almost the same then the previous.

plot(NN4d, rep="best")

(f)

In this final exercise, I’m building another neural network model. This time with two hidden layes containing two hidden variable. The curve looks different than the other 2 previous. But, the quality of the mother is better, the nodes help to make this important change.

NN4f <- neuralnet(medv~lstat, data = Boston2train, hidden=c(2,2))
plot(NN4f, rep="best")

plot(Boston2$lstat,Boston2$medv)
par(new=TRUE)
curve(NN4f$result.matrix[14,1]+NN4f$result.matrix[16,1]*(NN4f$result.matrix[12,1]*tanh(x*NN4f$result.matrix[5,1]+NN4f$result.matrix[4,1])+NN4f$result.matrix[11,1]+NN4f$result.matrix[13,1]*tanh(x*NN4f$result.matrix[7,1]+NN4f$result.matrix[6,1]))+NN4f$result.matrix[15,1]*(NN4f$result.matrix[9,1]*tanh(x*NN4f$result.matrix[5,1]+NN4f$result.matrix[4,1])+NN4f$result.matrix[8,1]+NN4f$result.matrix[10,1]*tanh(x*NN4f$result.matrix[7,1]+NN4f$result.matrix[6,1])),0,1, axes=F, xlab="", ylab="", col="blue")