SD_Assignment 2
Steve Dunn
June 26, 2016
Prof. Andy Catlin IS 643 - Special Topics: Recommender Systems Department of Data Analytics, City University of New York
======================================================================================
Using R package to build the similarity and Collaborative Filtering recommender System code and data set Mostly taken from example in Building a Recommendation System with R
======================================================================================
While the models use for recommendations do as is expected i.e. make recommendation the accuracy and precision of those recommendation have to be evaluated to see if they make sense either at a user -user or item-item filtering. The accuracy is measured with RMSE, MSE and MAE. Since the measure as calculated differently and apply different weightings I would choose between the RMSE or MAE.I also found that using k-fold cross validation is faster than splitting the data set manually. 
## [1] 3.587565## 1 2 3 4 5 6 7
## 1 0.0000000 0.9605820 0.8339504 0.9192637 0.9326136 0.9541710 0.9446653
## 2 0.9605820 0.0000000 0.9268716 0.9370341 0.9848027 0.9543931 0.9670869
## 3 0.8339504 0.9268716 0.0000000 0.9130323 1.0000000 0.8668857 0.8738971
## 4 0.9192637 0.9370341 0.9130323 0.0000000 0.9946918 0.9238226 0.8823323
## 5 0.9326136 0.9848027 1.0000000 0.9946918 0.0000000 0.9336269 0.9081632
## 6 0.9541710 0.9543931 0.8668857 0.9238226 0.9336269 0.0000000 0.9605997
## 7 0.9446653 0.9670869 0.8738971 0.8823323 0.9081632 0.9605997 0.0000000
## 8 0.9775049 0.9586588 0.8958898 0.9765327 0.9412276 0.9775950 0.9561888
## 9 0.9764039 0.9400556 0.9191450 0.9938837 0.8809189 0.9427809 0.9394858
## 10 0.9683044 0.9826137 0.9024436 0.9691166 0.9401144 0.9784868 0.9780770
## 8 9 10
## 1 0.9775049 0.9764039 0.9683044
## 2 0.9586588 0.9400556 0.9826137
## 3 0.8958898 0.9191450 0.9024436
## 4 0.9765327 0.9938837 0.9691166
## 5 0.9412276 0.8809189 0.9401144
## 6 0.9775950 0.9427809 0.9784868
## 7 0.9561888 0.9394858 0.9780770
## 8 0.0000000 0.8990017 0.9752306
## 9 0.8990017 0.0000000 0.9880545
## 10 0.9752306 0.9880545 0.0000000
## 1 2 3 4 5 6 7
## 1 0.0000000 0.9605820 0.8339504 0.9192637 0.9326136 0.9541710 0.9446653
## 2 0.9605820 0.0000000 0.9268716 0.9370341 0.9848027 0.9543931 0.9670869
## 3 0.8339504 0.9268716 0.0000000 0.9130323 1.0000000 0.8668857 0.8738971
## 4 0.9192637 0.9370341 0.9130323 0.0000000 0.9946918 0.9238226 0.8823323
## 5 0.9326136 0.9848027 1.0000000 0.9946918 0.0000000 0.9336269 0.9081632
## 6 0.9541710 0.9543931 0.8668857 0.9238226 0.9336269 0.0000000 0.9605997
## 8 9 10
## 1 0.9775049 0.9764039 0.9683044
## 2 0.9586588 0.9400556 0.9826137
## 3 0.8958898 0.9191450 0.9024436
## 4 0.9765327 0.9938837 0.9691166
## 5 0.9412276 0.8809189 0.9401144
## 6 0.9775950 0.9427809 0.9784868## Toy Story (1995)
## Toy Story (1995) 0.0000000
## GoldenEye (1995) 0.9487374
## Four Rooms (1995) 0.9132997
## Get Shorty (1995) 0.9429069
## Copycat (1995) 0.9613638
## Shanghai Triad (Yao a yao yao dao waipo qiao) (1995) 0.9551194
## Twelve Monkeys (1995) 0.9489155
## Babe (1995) 0.9600459
## Dead Man Walking (1995) 0.9387445
## Richard III (1995) 0.9430394
## GoldenEye (1995)
## Toy Story (1995) 0.9487374
## GoldenEye (1995) 0.0000000
## Four Rooms (1995) 0.9088797
## Get Shorty (1995) 0.9394926
## Copycat (1995) 0.9426876
## Shanghai Triad (Yao a yao yao dao waipo qiao) (1995) 0.9550903
## Twelve Monkeys (1995) 0.9411770
## Babe (1995) 0.9499076
## Dead Man Walking (1995) 0.9145017
## Richard III (1995) 0.9389799
## Four Rooms (1995)
## Toy Story (1995) 0.9132997
## GoldenEye (1995) 0.9088797
## Four Rooms (1995) 0.0000000
## Get Shorty (1995) 0.8991940
## Copycat (1995) 0.9424719
## Shanghai Triad (Yao a yao yao dao waipo qiao) (1995) 0.9683641
## Twelve Monkeys (1995) 0.9208737
## Babe (1995) 0.8787096
## Dead Man Walking (1995) 0.9084892
## Richard III (1995) 0.9269418
## Get Shorty (1995)
## Toy Story (1995) 0.9429069
## GoldenEye (1995) 0.9394926
## Four Rooms (1995) 0.8991940
## Get Shorty (1995) 0.0000000
## Copycat (1995) 0.8919936
## Shanghai Triad (Yao a yao yao dao waipo qiao) (1995) 0.9190369
## Twelve Monkeys (1995) 0.9484601
## Babe (1995) 0.9539981
## Dead Man Walking (1995) 0.9497018
## Richard III (1995) 0.9582736
## Copycat (1995)
## Toy Story (1995) 0.9613638
## GoldenEye (1995) 0.9426876
## Four Rooms (1995) 0.9424719
## Get Shorty (1995) 0.8919936
## Copycat (1995) 0.0000000
## Shanghai Triad (Yao a yao yao dao waipo qiao) (1995) 0.9962406
## Twelve Monkeys (1995) 0.9359823
## Babe (1995) 0.9452349
## Dead Man Walking (1995) 0.9340369
## Richard III (1995) 0.9041944
## Shanghai Triad (Yao a yao yao dao waipo qiao) (1995)
## Toy Story (1995) 0.9551194
## GoldenEye (1995) 0.9550903
## Four Rooms (1995) 0.9683641
## Get Shorty (1995) 0.9190369
## Copycat (1995) 0.9962406
## Shanghai Triad (Yao a yao yao dao waipo qiao) (1995) 0.0000000
## Twelve Monkeys (1995) 0.9072989
## Babe (1995) 0.8613908
## Dead Man Walking (1995) 0.9517965
## Richard III (1995) 0.9405701
## Twelve Monkeys (1995)
## Toy Story (1995) 0.9489155
## GoldenEye (1995) 0.9411770
## Four Rooms (1995) 0.9208737
## Get Shorty (1995) 0.9484601
## Copycat (1995) 0.9359823
## Shanghai Triad (Yao a yao yao dao waipo qiao) (1995) 0.9072989
## Twelve Monkeys (1995) 0.0000000
## Babe (1995) 0.9595148
## Dead Man Walking (1995) 0.9503334
## Richard III (1995) 0.9494393
## Babe (1995)
## Toy Story (1995) 0.9600459
## GoldenEye (1995) 0.9499076
## Four Rooms (1995) 0.8787096
## Get Shorty (1995) 0.9539981
## Copycat (1995) 0.9452349
## Shanghai Triad (Yao a yao yao dao waipo qiao) (1995) 0.8613908
## Twelve Monkeys (1995) 0.9595148
## Babe (1995) 0.0000000
## Dead Man Walking (1995) 0.9611934
## Richard III (1995) 0.9681040
## Dead Man Walking (1995)
## Toy Story (1995) 0.9387445
## GoldenEye (1995) 0.9145017
## Four Rooms (1995) 0.9084892
## Get Shorty (1995) 0.9497018
## Copycat (1995) 0.9340369
## Shanghai Triad (Yao a yao yao dao waipo qiao) (1995) 0.9517965
## Twelve Monkeys (1995) 0.9503334
## Babe (1995) 0.9611934
## Dead Man Walking (1995) 0.0000000
## Richard III (1995) 0.9476115
## Richard III (1995)
## Toy Story (1995) 0.9430394
## GoldenEye (1995) 0.9389799
## Four Rooms (1995) 0.9269418
## Get Shorty (1995) 0.9582736
## Copycat (1995) 0.9041944
## Shanghai Triad (Yao a yao yao dao waipo qiao) (1995) 0.9405701
## Twelve Monkeys (1995) 0.9494393
## Babe (1995) 0.9681040
## Dead Man Walking (1995) 0.9476115
## Richard III (1995) 0.0000000
## Toy Story (1995)
## Toy Story (1995) 0.0000000
## GoldenEye (1995) 0.9487374
## Four Rooms (1995) 0.9132997
## Get Shorty (1995) 0.9429069
## Copycat (1995) 0.9613638
## Shanghai Triad (Yao a yao yao dao waipo qiao) (1995) 0.9551194
## GoldenEye (1995)
## Toy Story (1995) 0.9487374
## GoldenEye (1995) 0.0000000
## Four Rooms (1995) 0.9088797
## Get Shorty (1995) 0.9394926
## Copycat (1995) 0.9426876
## Shanghai Triad (Yao a yao yao dao waipo qiao) (1995) 0.9550903
## Four Rooms (1995)
## Toy Story (1995) 0.9132997
## GoldenEye (1995) 0.9088797
## Four Rooms (1995) 0.0000000
## Get Shorty (1995) 0.8991940
## Copycat (1995) 0.9424719
## Shanghai Triad (Yao a yao yao dao waipo qiao) (1995) 0.9683641
## Get Shorty (1995)
## Toy Story (1995) 0.9429069
## GoldenEye (1995) 0.9394926
## Four Rooms (1995) 0.8991940
## Get Shorty (1995) 0.0000000
## Copycat (1995) 0.8919936
## Shanghai Triad (Yao a yao yao dao waipo qiao) (1995) 0.9190369
## Copycat (1995)
## Toy Story (1995) 0.9613638
## GoldenEye (1995) 0.9426876
## Four Rooms (1995) 0.9424719
## Get Shorty (1995) 0.8919936
## Copycat (1995) 0.0000000
## Shanghai Triad (Yao a yao yao dao waipo qiao) (1995) 0.9962406Splitting the Data set into train and test
#Subsetting the dataset
ratings_movies <- MovieLense[ rowCounts( MovieLense) > 50, colCounts( MovieLense) > 100]
which_train<-sample(x=c(TRUE,FALSE),size = nrow(ratings_movies),replace = TRUE,prob = c(.8,.2))
rec_data_train<-ratings_movies[which_train,]
rec_data_test<-ratings_movies[!which_train,]
#build recommender Model
rec_model<-Recommender(data=rec_data_train,method='IBCF')
#predict 10 movies for the users
n_rec<-10
rec_predict<-predict(rec_model,newdata=rec_data_test,n=n_rec)
rec_predict## Recommendations as 'topNList' with n = 10 for 103 users.rec_matrix<-sapply(rec_predict@items,
function(x){
colnames(rec_data_test)[x]
})
rec_matrix[,1:10]## 13
## [1,] "Men in Black (1997)"
## [2,] "Con Air (1997)"
## [3,] "Lost World: Jurassic Park, The (1997)"
## [4,] "Time to Kill, A (1996)"
## [5,] "Firm, The (1993)"
## [6,] "Fly Away Home (1996)"
## [7,] "Phenomenon (1996)"
## [8,] "Welcome to the Dollhouse (1995)"
## [9,] "Crash (1996)"
## [10,] "Fifth Element, The (1997)"
## 14 21
## [1,] "Babe (1995)" "Rock, The (1996)"
## [2,] "Blade Runner (1982)" "Cold Comfort Farm (1995)"
## [3,] "Sting, The (1973)" "Quiz Show (1994)"
## [4,] "Nikita (La Femme Nikita) (1990)" "Time to Kill, A (1996)"
## [5,] "Edge, The (1997)" "In the Line of Fire (1993)"
## [6,] "Great Escape, The (1963)" "Welcome to the Dollhouse (1995)"
## [7,] "Dante's Peak (1997)" "Miracle on 34th Street (1994)"
## [8,] "Phenomenon (1996)" "Sex, Lies, and Videotape (1989)"
## [9,] "Pretty Woman (1990)" "Fly Away Home (1996)"
## [10,] "Clear and Present Danger (1994)" "Little Women (1994)"
## 24
## [1,] "Rumble in the Bronx (1995)"
## [2,] "Apollo 13 (1995)"
## [3,] "Batman Forever (1995)"
## [4,] "Ed Wood (1994)"
## [5,] "Terminator 2: Judgment Day (1991)"
## [6,] "Truth About Cats & Dogs, The (1996)"
## [7,] "Cold Comfort Farm (1995)"
## [8,] "Twister (1996)"
## [9,] "Swingers (1996)"
## [10,] "Right Stuff, The (1983)"
## 38
## [1,] "Twelve Monkeys (1995)"
## [2,] "Seven (Se7en) (1995)"
## [3,] "Crow, The (1994)"
## [4,] "Mystery Science Theater 3000: The Movie (1996)"
## [5,] "Monty Python and the Holy Grail (1974)"
## [6,] "Cinema Paradiso (1988)"
## [7,] "Field of Dreams (1989)"
## [8,] "Star Trek IV: The Voyage Home (1986)"
## [9,] "Raising Arizona (1987)"
## [10,] "Starship Troopers (1997)"
## 48
## [1,] "Mighty Aphrodite (1995)"
## [2,] "Cold Comfort Farm (1995)"
## [3,] "Contact (1997)"
## [4,] "As Good As It Gets (1997)"
## [5,] "Everyone Says I Love You (1996)"
## [6,] "Boogie Nights (1997)"
## [7,] "Mrs. Doubtfire (1993)"
## [8,] "African Queen, The (1951)"
## [9,] "Magnificent Seven, The (1954)"
## [10,] "Crying Game, The (1992)"
## 59
## [1,] "Sneakers (1992)"
## [2,] "One Fine Day (1996)"
## [3,] "Lost World: Jurassic Park, The (1997)"
## [4,] "Welcome to the Dollhouse (1995)"
## [5,] "Cable Guy, The (1996)"
## [6,] "Hunchback of Notre Dame, The (1996)"
## [7,] "Lone Star (1996)"
## [8,] "Stargate (1994)"
## [9,] "Abyss, The (1989)"
## [10,] "101 Dalmatians (1996)"
## 60
## [1,] "Manchurian Candidate, The (1962)"
## [2,] "Volcano (1997)"
## [3,] "Ghost and the Darkness, The (1996)"
## [4,] "Monty Python's Life of Brian (1979)"
## [5,] "Nightmare on Elm Street, A (1984)"
## [6,] "Young Guns (1988)"
## [7,] "Batman Forever (1995)"
## [8,] "Starship Troopers (1997)"
## [9,] "Stargate (1994)"
## [10,] "Ghost (1990)"
## 72
## [1,] "Multiplicity (1996)"
## [2,] "Cable Guy, The (1996)"
## [3,] "Crying Game, The (1992)"
## [4,] "George of the Jungle (1997)"
## [5,] "Father of the Bride Part II (1995)"
## [6,] "Mr. Smith Goes to Washington (1939)"
## [7,] "Pink Floyd - The Wall (1982)"
## [8,] "It's a Wonderful Life (1946)"
## [9,] "Nutty Professor, The (1996)"
## [10,] "Pinocchio (1940)"
## 77
## [1,] "Professional, The (1994)"
## [2,] "Much Ado About Nothing (1993)"
## [3,] "Aladdin (1992)"
## [4,] "Room with a View, A (1986)"
## [5,] "When Harry Met Sally... (1989)"
## [6,] "Absolute Power (1997)"
## [7,] "Face/Off (1997)"
## [8,] "Fly Away Home (1996)"
## [9,] "Like Water For Chocolate (Como agua para chocolate) (1992)"
## [10,] "Philadelphia Story, The (1940)"evaluating Recommendationusing k fold Cross Validation
n_fold <- 10
items_to_keep <- 15
rating_threshold <- 3
model_to_evaluate <- "IBCF"
model_parameters <- NULLeval_sets <- evaluationScheme( data = ratings_movies,
method = "cross-validation"
, k = n_fold, given = items_to_keep
, goodRating = rating_threshold)
eval_prediction <- predict(object = rec_model,
newdata = getData( eval_sets, "known"),
n = 10, type = "ratings")
eval_accuracy <- calcPredictionAccuracy(
x = eval_prediction,
data = getData(eval_sets, "unknown"),
byUser = FALSE,
given = items_to_recommend)
head( eval_accuracy)## RMSE MSE MAE
## 1.485621 2.207070 1.104279RMSE MSE MAE
1.0114568 1.0230448 0.7419217
eval_recommender <- Recommender(data = getData(eval_sets, "train"),
method = model_to_evaluate,
parameter = model_parameters)
results <- evaluate(x = eval_sets,
method = model_to_evaluate,
n = seq(10, 100, 10))## IBCF run fold/sample [model time/prediction time]
## 1 [0.77sec/0.1sec]
## 2 [0.55sec/0.04sec]
## 3 [0.61sec/0.01sec]
## 4 [0.61sec/0.03sec]
## 5 [1.01sec/0.07sec]
## 6 [0.68sec/0.03sec]
## 7 [0.75sec/0.03sec]
## 8 [0.5sec/0.04sec]
## 9 [0.55sec/0.03sec]
## 10 [0.6sec/0.03sec]IBCF run fold/sample [model time/prediction time]
1 [0.71sec/0.05sec]
2 [0.73sec/0.05sec]
3 [0.77sec/0.04sec]
4 [0.77sec/0.03sec]
5 [0.75sec/0.05sec]
6 [0.79sec/0.05sec]
7 [0.77sec/0.04sec]
8 [0.76sec/0.05sec]
9 [0.73sec/0.05sec]
10 [0.75sec/0.05sec]
head( getConfusionMatrix( results)[[ 1]])## TP FP FN TN precision recall TPR
## 10 3.053571 6.946429 74.48214 232.5179 0.3053571 0.04059486 0.04059486
## 20 5.946429 14.053571 71.58929 225.4107 0.2973214 0.08053030 0.08053030
## 30 8.875000 21.125000 68.66071 218.3393 0.2958333 0.11826206 0.11826206
## 40 11.446429 28.553571 66.08929 210.9107 0.2861607 0.14945523 0.14945523
## 50 13.821429 36.107143 63.71429 203.3571 0.2767081 0.17978363 0.17978363
## 60 15.964286 43.785714 61.57143 195.6786 0.2668866 0.20719043 0.20719043
## FPR
## 10 0.02874900
## 20 0.05852716
## 30 0.08769312
## 40 0.11838159
## 50 0.15034333
## 60 0.18281125TP FP FN TN precision recall TPR
10 3.232143 6.767857 73.08929 233.9107 0.3232143 0.03688216 0.03688216
20 6.607143 13.392857 69.71429 227.2857 0.3303571 0.08214411 0.08214411
30 10.089286 19.910714 66.23214 220.7679 0.3363095 0.12863596 0.12863596
40 13.375000 26.625000 62.94643 214.0536 0.3343750 0.17386720 0.17386720
50 16.750000 33.250000 59.57143 207.4286 0.3350000 0.22388116 0.22388116
60 20.178571 39.821429 56.14286 200.8571 0.3363095 0.27266253 0.27266253
FPR
10 0.02767918
20 0.05434228
30 0.08065479
40 0.10792881
50 0.13525171
60 0.16214875
table_wide <- reshape( data =table_Users,#convert table to wide
direction = "wide",
idvar ="UserID",
timevar = "MovieID",
v.names = "Rating" )## Warning in reshapeWide(data, idvar = idvar, timevar = timevar, varying =
## varying, : some constant variables (Timestamp) are really varying###Item Based Collaborative Filtering
head(table_wide[, 1: 10])## UserID Timestamp Rating.1193 Rating.661 Rating.914 Rating.3408
## 1 1 12 5 3 3 4
## 54 2 65 5 NA NA NA
## 183 3 194 NA NA NA NA
## 234 4 245 NA NA NA NA
## 255 5 266 NA NA NA 3
## 453 6 464 NA NA 5 5
## Rating.2355 Rating.1197 Rating.1287 Rating.2804
## 1 5 3 5 5
## 54 NA NA NA NA
## 183 5 5 NA NA
## 234 NA NA NA NA
## 255 5 NA NA NA
## 453 NA NA NA NAW_movies<-table_wide
W_movies <- (table_wide[,!(names(table_wide) %in% c("UserID"))])
W_movies[is.na(W_movies)] = 0
head(W_movies[, 1:10])## Timestamp Rating.1193 Rating.661 Rating.914 Rating.3408 Rating.2355
## 1 12 5 3 3 4 5
## 54 65 5 0 0 0 0
## 183 194 0 0 0 0 5
## 234 245 0 0 0 0 0
## 255 266 0 0 0 3 5
## 453 464 0 0 5 5 0
## Rating.1197 Rating.1287 Rating.2804 Rating.594
## 1 3 5 5 4
## 54 0 0 0 0
## 183 5 0 0 0
## 234 0 0 0 0
## 255 0 0 0 0
## 453 0 0 0 0# Function to calculate cosine Similarity
Sim_Cos <- function(a,b)
{
cos_ine <- sum(a*b) / (sqrt(sum(a*a)) * sqrt(sum(b*b)))
return(cos_ine)
}
#Fill table with item rating
W_movies_sim <- matrix(NA, nrow=ncol(W_movies)
,ncol=ncol(W_movies)
,dimnames=list(colnames(W_movies)
,colnames(W_movies)))
for(i in 1:ncol(W_movies)) {
# Loop through the columns
for(j in 1:ncol(W_movies)) {
# Fill matrix with cosine similarities
W_movies_sim[i,j] <- Sim_Cos(as.matrix(W_movies[i]),as.matrix(W_movies[j]))
}
}image(as.matrix(W_movies_sim),main="Similarity of Items")
W_movies_sim <- as.data.frame(W_movies_sim)
W_movies_n <- matrix(NA, nrow=ncol(W_movies_sim),ncol=5,dimnames=list(colnames(W_movies_sim)))
for(i in 1:ncol(W_movies))
{
W_movies_n[i,] <- (t(head(n=5,
rownames(W_movies_sim
[order(W_movies_sim[,i]
,decreasing=TRUE),][i]))))
}
#convert to rating Matrix
ratings_movies <- as(W_movies_sim, "realRatingMatrix")
ratings_movies## 521 x 270 rating matrix of class 'realRatingMatrix' with 522 ratings.#Splitting the Data set into train and test
which_train<-sample(x=c(TRUE,FALSE),size = nrow(ratings_movies),replace = TRUE,prob = c(.8,.2))
rec_data_train<-ratings_movies[which_train,]
rec_data_test<-ratings_movies[!which_train,]
#using the training dataset to create recommender model
rec <- Recommender(rec_data_train, method = "IBCF")
#predict recommended Movie for users on test dataset
rec_pred <- predict(rec, rec_data_test, n=10)
rec_pred## Recommendations as 'topNList' with n = 10 for 121 users.pre<-sapply(rec_pred@items,
function(x){
colnames(ratings_movies)[x]
})
pre## $`0.0113267275681244`
## character(0)
##
## $`0.046352454355709`
## character(0)
##
## $`0.0986296585162831`
## character(0)
##
## $`0.105972972699387`
## character(0)
##
## $`0.107939451445173`
## character(0)
##
## $`0.112456835409752`
## character(0)
##
## $`0.118028963173405`
## character(0)
##
## $`0.122851429777349`
## character(0)
##
## $`0.148728426356123`
## character(0)
##
## $`0.149477952429801`
## character(0)
##
## $`0.156610779437004`
## character(0)
##
## $`0.160075250492499`
## character(0)
##
## $`0.163918613399546`
## character(0)
##
## $`0.168345810492017`
## character(0)
##
## $`0.169671945936077`
## character(0)
##
## $`0.18211701967576`
## character(0)
##
## $`0.193467350518731`
## character(0)
##
## $`0.19387556690047`
## character(0)
##
## $`0.204711486196333`
## character(0)
##
## $`0.205502081866601`
## character(0)
##
## $`0.207915852126973`
## character(0)
##
## $`0.208952358599355`
## character(0)
##
## $`0.209159564210644`
## character(0)
##
## $`0.212185658478326`
## character(0)
##
## $`0.215368933897561`
## character(0)
##
## $`0.217348763170466`
## character(0)
##
## $`0.21822005537603`
## character(0)
##
## $`0.219000382114917`
## character(0)
##
## $`0.219601741569622`
## character(0)
##
## $`0.227249034598172`
## character(0)
##
## $`0.228137718956745`
## character(0)
##
## $`0.228597257538363`
## character(0)
##
## $`0.231134945328618`
## character(0)
##
## $`0.232378999390064`
## character(0)
##
## $`0.233574191111548`
## character(0)
##
## $`0.233896518647658`
## character(0)
##
## $`0.238297675254607`
## character(0)
##
## $`0.242290029774689`
## character(0)
##
## $`0.24729963361925`
## character(0)
##
## $`0.258085897049859`
## character(0)
##
## $`0.265857722213741`
## character(0)
##
## $`0.26844041659147`
## character(0)
##
## $`0.268983641140874`
## character(0)
##
## $`0.269354680461069`
## character(0)
##
## $`0.270412551387929`
## character(0)
##
## $`0.278678591109201`
## character(0)
##
## $`0.29056607303551`
## character(0)
##
## $`0.297283034326465`
## character(0)
##
## $`0.299875754025306`
## character(0)
##
## $`0.300027282199484`
## character(0)
##
## $`0.30320778413133`
## character(0)
##
## $`0.306080643027735`
## character(0)
##
## $`0.306187971048113`
## character(0)
##
## $`0.306269062579458`
## character(0)
##
## $`0.315586401534462`
## character(0)
##
## $`0.318750133020688`
## character(0)
##
## $`0.322844729104474`
## character(0)
##
## $`0.331553345248235`
## character(0)
##
## $`0.332174953978052`
## character(0)
##
## $`0.332267181641785`
## character(0)
##
## $`0.332857888066388`
## character(0)
##
## $`0.356795683751536`
## character(0)
##
## $`0.372193461505738`
## character(0)
##
## $`0.384228118317613`
## character(0)
##
## $`0.38894553661996`
## character(0)
##
## $`0.389863498142156`
## character(0)
##
## $`0.395398387352179`
## character(0)
##
## $`0.399026847060118`
## character(0)
##
## $`0.399181192489625`
## character(0)
##
## $`0.401080088040019`
## character(0)
##
## $`0.404939223427807`
## character(0)
##
## $`0.404955258670905`
## character(0)
##
## $`0.40625517421211`
## character(0)
##
## $`0.406466028571279`
## character(0)
##
## $`0.414698356193079`
## character(0)
##
## $`0.41523438799323`
## character(0)
##
## $`0.415841240233906`
## character(0)
##
## $`0.416012121313744`
## character(0)
##
## $`0.419008389054128`
## character(0)
##
## $`0.419103669553709`
## character(0)
##
## $`0.424708353787647`
## character(0)
##
## $`0.428487454576841`
## character(0)
##
## $`0.436239757683357`
## character(0)
##
## $`0.437172530416781`
## character(0)
##
## $`0.439680010465343`
## character(0)
##
## $`0.448811360957152`
## character(0)
##
## $`0.451486684947547`
## character(0)
##
## $`0.462702882728254`
## character(0)
##
## $`0.463548192165231`
## character(0)
##
## $`0.467125183483087`
## character(0)
##
## $`0.468851194134537`
## character(0)
##
## $`0.471993450631103`
## character(0)
##
## $`0.476129158338028`
## character(0)
##
## $`0.479116641152919`
## character(0)
##
## $`0.485042704782015`
## character(0)
##
## $`0.489815202849059`
## character(0)
##
## $`0.493636213153581`
## character(0)
##
## $`0.497242638060713`
## character(0)
##
## $`0.499306215920241`
## character(0)
##
## $`0.499916641835373`
## character(0)
##
## $`0.501272996974356`
## character(0)
##
## $`0.50504978210278`
## character(0)
##
## $`0.505997918449279`
## character(0)
##
## $`0.515559168591166`
## character(0)
##
## $`0.517453950379684`
## character(0)
##
## $`0.518066042661292`
## character(0)
##
## $`0.51862688485472`
## character(0)
##
## $`0.525203159094338`
## character(0)
##
## $`0.529759761228537`
## character(0)
##
## $`0.532102150706667`
## character(0)
##
## $`0.53487004939629`
## character(0)
##
## $`0.548857375340069`
## character(0)
##
## $`0.54906677584502`
## character(0)
##
## $`0.558323480947938`
## character(0)
##
## $`0.566376600366737`
## character(0)
##
## $`0.567210758062445`
## character(0)
##
## $`0.570389423793479`
## character(0)
##
## $`0.626663204544742`
## character(0)
##
## $`0.632425242177905`
## character(0)
##
## $`0.667189890594394`
## character(0)
##
## $`0.671488651151846`
## character(0)