Clustering Airbnb in New York
Pola Parol
2023-02-07
Introduction
I decided to explore Airbnb database in New York. Using clustering methods I chose the number of clusters. Clusters were selected in both scenarios: using only locations atributes: longitude and latitude and apart from them: price and type of the rented airbnb. They made it possible to divide the acomodations in New York. In the resulting clusters, I conducted, among other statistics, an analysis of the price in the districts of New York. The main purpose is to examine the price of airbnb in clusters. We can expect that price of booking depend on its location and airbnb type as well. For example in the Manhattan district the price is supposed to be higher than in Queens or Brooklyn.
Data comes from Kaggle database. This data file includes all needed information to find out more about hosts, geographical availability, necessary metrics to make predictions and draw conclusions. I randomly selected 1000-size sample and chose variables were said to have influence on distinguishing airbnb between each other.
https://www.kaggle.com/datasets/dgomonov/new-york-city-airbnb-open-data
Reading data
At the beggining, I read airbnb dataset and create and edit necessary variables.
airbnb <- read.csv("airbnb.csv")
airbnb$private_room <- airbnb$room_type
airbnb$entire_apartment <- airbnb$room_type
airbnb$shared_room <- airbnb$room_type
airbnb$private_room[airbnb$private_room == "Private room"] <- 1
airbnb$private_room[airbnb$private_room != 1] <- 0
airbnb$entire_apartment[airbnb$entire_apartment == "Entire home/apt"] <- 1
airbnb$entire_apartment[airbnb$entire_apartment != 1] <- 0
airbnb$shared_room[airbnb$shared_room == "Shared room"] <- 1
airbnb$shared_room[airbnb$shared_room != 1] <- 0
airbnb$private_room <- as.numeric(airbnb$private_room)
airbnb$entire_apartment <- as.numeric(airbnb$entire_apartment)
airbnb$shared_room <- as.numeric(airbnb$shared_room)
airbnb <- airbnb %>%
filter(price < 1000)
set.seed(200)
airbnb_sample <- airbnb[sample(nrow(airbnb), 1000), ]
set.seed(200)
airbnb_sample_ <- airbnb[sample(nrow(airbnb), 1000), ]
airbnb_map <- airbnb_sample[,c(7,8)]Choosing Clustering variables
clustering on location, price and type of the airbnb
Data Pre Processing
airbnb2 = subset(airbnb, select=-c(id,name,host_name,host_id,last_review,calculated_host_listings_count, availability_365, neighbourhood_group,neighbourhood, room_type, reviews_per_month, minimum_nights, number_of_reviews))
airbnb2 <- scale(airbnb2)
set.seed(200)
airbnb1_sample <- airbnb2[sample(nrow(airbnb2), 1000), ]
airbnb_clus <- airbnb1_sampleData Visualisation
To have a better visualisation, the airbnb locations are marked on the map of NY
my.sf.point <- st_as_sf(x = airbnb_map,
coords = c("longitude", "latitude"),
crs = "+proj=longlat +datum=WGS84 +ellps=WGS84 +towgs84=0,0,0")
# simple plot
# plot(my.sf.point)
# interactive map:
mapview(my.sf.point)## Warning in cbind(`Feature ID` = fid, mat): number of rows of result is not a
## multiple of vector length (arg 1)# convert to sp object if needed
my.sp.point <- as(my.sf.point, "Spatial")Hopkins Statistics
Before clustering it is necessary to check whether the data is clusterable. Accoring to Hopkins Statisitcs equal to 0.97 I can assume that dataset is significanly a clusterable data.
get_clust_tendency(airbnb_clus, 2, graph=TRUE, gradient=list(low="red", mid="white", high="blue"))## $hopkins_stat
## [1] 0.973009
##
## $plot
Clustering
Afterwards, as it is known that we can cluster is to choose the most optimal number of clusters. I will aply clustering algorithms such as kmeans.
At the beginning, I will find the most optimal number of clusters. I will use silhouette and wss statistics and then apply it to clustering algorithms: kmeans, PAM and CLARA. I have chosen a sample consisted of 1000 observation, so the hierarchical clustering is not visable and transparent.
Silhouette
In the first step, the silhouette method was used to choose optimal number of clusters.The silhouette value is a measure of how similar an object is to its own cluster (cohesion) compared to other clusters (separation). The silhouette ranges from −1 to +1, where a high value indicates that the object is well matched to its own cluster and poorly matched to neighboring clusters.[wikipedia]
Kmeans charts suggests dividing data into 2 clusters. PAM method chose the 2 clusters and CLARA grouped data into 2 clusters as well. Lastly, hcut recommends 3 clusters. However, the difference in silhouette width between the two and three clusters in hcut is not so visible.
library(gridExtra)
a <- fviz_nbclust(airbnb_clus, FUNcluster = kmeans, method = "silhouette") +
labs(subtitle = "silhouette - kmeans")
b <- fviz_nbclust(airbnb_clus, FUNcluster = cluster::pam, method = "silhouette") +
labs(subtitle = "silhouette - pam")
c <- fviz_nbclust(airbnb_clus, FUNcluster = cluster::clara, method = "silhouette") +
labs(subtitle = "silhouette - clara")
d <- fviz_nbclust(airbnb_clus, FUNcluster = hcut, method = "silhouette") +
labs(subtitle = "silhouette - hcut")
grid.arrange(a, b, c, d, ncol=2)
Total within sum of squares (WSS)
The Squared Error for each point is the square of the distance of the point from its representation i.e. its predicted cluster center. The WSS score is the sum of these Squared Errors for all the points. [https://medium.com/analytics-vidhya/how-to-determine-the-optimal-k-for-k-means-708505d204eb]
Elbow method
To compare results obtained by silhouette method, I will analyze charts of elbow method with WSS. In CLARA and PAM method the end of elbow in the graph visually suggests 4 clusters. Kmeans and hcut in silhouette method suggest 3 clusters.
e1 <- fviz_nbclust(airbnb_clus, kmeans, method = "wss",k.max = 10) +
labs(subtitle = "Elbow method - kmeans")
e2 <- fviz_nbclust(airbnb_clus, clara, method = "wss",k.max = 10) +
labs(subtitle = "Elbow method - clara")
e3 <- fviz_nbclust(airbnb_clus, pam, method = "wss",k.max = 10) +
labs(subtitle = "Elbow method - pam")
e4 <- fviz_nbclust(airbnb_clus, hcut, method = "wss",k.max = 10) +
labs(subtitle = "Elbow method - hcut")
grid.arrange(e1,e2,e3,e4, ncol=2, top = "Optimal number of clusters")
library(factoextra)
cl_kmeans <- eclust(airbnb_clus, k=2, FUNcluster="kmeans", hc_metric="pearson", graph=FALSE)
a <- fviz_silhouette(cl_kmeans)## cluster size ave.sil.width
## 1 1 473 0.43
## 2 2 527 0.47b <- fviz_cluster(cl_kmeans, data = airbnb_clus, elipse.type = "convex") + theme_minimal()
grid.arrange(a, b, ncol=2)
Kmeans
The Κ-means clustering algorithm uses iterative refinement to produce a final result. The algorithm inputs are the number of clusters Κ and the data set. The data set is a collection of features for each data point. The algorithms starts with initial estimates for the Κ centroids, which can either be randomly generated or randomly selected from the data set. The algorithm then iterates between two steps: Data assigment step and Centroid update step. [https://blogs.oracle.com/ai-and-datascience/post/introduction-to-k-means-clustering]
km2 <- eclust(airbnb_clus,k=2,hc_metric = 'euclidean', graph = FALSE)
km3 <- eclust(airbnb_clus,k=3,hc_metric = 'euclidean', graph = FALSE)
km4 <- eclust(airbnb_clus,k=4,hc_metric = 'euclidean', graph = FALSE)
km5 <- eclust(airbnb_clus,k=5,hc_metric = 'euclidean', graph = FALSE)
k1 <- fviz_cluster(km2, geom = c("point")) + ggtitle('K-means with 2 clusters')
k2 <- fviz_cluster(km3, geom = c("point")) + ggtitle('K-means with 3 clusters')
k3 <- fviz_cluster(km4, geom = c("point")) + ggtitle('K-means with 4 clusters')
k4 <- fviz_cluster(km5, geom = c("point")) + ggtitle('K-means with 5 clusters')
grid.arrange(arrangeGrob(k1, k2, k3, k4, ncol=2 , top = "Clustering"))
PAM
-small datasets PAM is an implementation of the k-medoid method, i.e. a clustering technique that divides a dataset containing n objects into k groups (clusters) known a priori. A useful tool for assessing the quality of grouping is the silhouette. It is calculated for each object subject to grouping. On its basis, it is possible to determine whether the objects have been grouped correctly or whether they have been placed in the wrong clusters.
pam2 <- eclust(airbnb_clus,'pam',k=2,hc_metric = 'euclidean', graph = FALSE)
pam3 <- eclust(airbnb_clus,'pam',k=3,hc_metric = 'euclidean', graph = FALSE)
pam4 <- eclust(airbnb_clus,'pam',k=4,hc_metric = 'euclidean', graph = FALSE)
pam5<- eclust(airbnb_clus,'pam',k=5,hc_metric = 'euclidean', graph = FALSE)
p1 <- fviz_cluster(pam2, geom = c("point")) + ggtitle('PAM with 2 clusters')
p2 <- fviz_cluster(pam3, geom = c("point")) + ggtitle('PAM with 3 clusters')
p3 <- fviz_cluster(pam4, geom = c("point")) + ggtitle('PAM with 4 clusters')
p4 <- fviz_cluster(pam5, geom = c("point")) + ggtitle('PAM with 5 clusters')
grid.arrange(arrangeGrob(p1, p2, p3, p4, ncol=2 , top = "Clustering"))
CLARA
-larger datasets CLARA draws multiple samples of the dataset, then applies PAM on each sample, and gives the best clustering as the oputput. [from presentation]
clara2 <- eclust(airbnb_clus,'clara',k=2,hc_metric = 'euclidean', graph = FALSE)
clara3 <- eclust(airbnb_clus,'clara',k=3,hc_metric = 'euclidean', graph = FALSE)
clara4 <- eclust(airbnb_clus,'clara',k=4,hc_metric = 'euclidean', graph = FALSE)
clara5 <- eclust(airbnb_clus,'clara',k=5,hc_metric = 'euclidean', graph = FALSE)
c1 <- fviz_cluster(clara2, geom = c("point")) + ggtitle('Clara with 2 clusters')
c2 <- fviz_cluster(clara3, geom = c("point")) + ggtitle('Clara with 3 clusters')
c3 <- fviz_cluster(clara4, geom = c("point")) + ggtitle('Clara with 4 clusters')
c4 <- fviz_cluster(clara5, geom = c("point")) + ggtitle('Clara with 5 clusters')
grid.arrange(arrangeGrob(c1, c2, c3, c4, ncol=2 , top = "Clustering"))
Post diagnostics
According to silhouete method and k-means charts, the most interesting number of clusters is 2. Moreover the size of them is similar, which cannot be said about the division in the case of 3 or 4 clusters.
km2$silinfo$clus.avg.widths## [1] 0.4340356 0.4679409km3$silinfo$clus.avg.widths## [1] 0.4391868 0.4976030 0.2191956km4$silinfo$clus.avg.widths## [1] 0.5338543 0.3078273 0.1656116 0.4157589km2_s <- fviz_silhouette(km2) + ggtitle('K-means with 2 clusters')## cluster size ave.sil.width
## 1 1 473 0.43
## 2 2 527 0.47km3_s <- fviz_silhouette(km3) + ggtitle('K-means with 3 clusters')## cluster size ave.sil.width
## 1 1 438 0.44
## 2 2 499 0.50
## 3 3 63 0.22km4_s <- fviz_silhouette(km4) + ggtitle('K-means with 4 clusters')## cluster size ave.sil.width
## 1 1 418 0.53
## 2 2 48 0.31
## 3 3 64 0.17
## 4 4 470 0.42grid.arrange(arrangeGrob(km2_s,km3_s, km4_s, ncol=2 , top = "Clustering"))
Clusters’ analysys
To get to know better data, I will analyze and compare data divided into 2 clusters.
km_ = kmeans(airbnb_clus, centers = 2, nstart=25)
airbnb_clus_map_ <- cbind(airbnb_sample_, cluster = km_$cluster)
new_airbnb_ <- cbind(airbnb_sample_, cluster = km_$cluster)
cluster1 <- new_airbnb_[new_airbnb_$cluster == 1,]
cluster2 <- new_airbnb_[new_airbnb_$cluster == 2,]
cluster1_map_ <- new_airbnb_[airbnb_clus_map_$cluster == 1,]
cluster2_map_ <- new_airbnb_[airbnb_clus_map_$cluster == 2,]Results
Cluster 1
Cluster 1 consists of 473 airbnbs. Most of the airbnb is located in Brooklyn (46,5%). Also a relative huge group is in Manhattan district and 15,56 % of airbnb is in Queens. 454 out of 473 is private rooms. Based on summary table and density plot, mean price is around 80 dollars per night, 1st quartile is 50 and 3rd quartile is 95.
my.sf.point <- st_as_sf(x = cluster1_map_,
coords = c("longitude", "latitude"),
crs = "+proj=longlat +datum=WGS84 +ellps=WGS84 +towgs84=0,0,0")
# simple plot
# plot(my.sf.point)
# interactive map:
mapview(my.sf.point)# convert to sp object if needed
my.sp.point <- as(my.sf.point, "Spatial")
summary(cluster1)## id name host_id host_name
## Min. : 8025 Length:473 Min. : 7549 Length:473
## 1st Qu.:11434339 Class :character 1st Qu.: 12672019 Class :character
## Median :21261549 Mode :character Median : 40632179 Mode :character
## Mean :20266917 Mean : 78282972
## 3rd Qu.:29831517 3rd Qu.:137358866
## Max. :36477588 Max. :273139430
##
## neighbourhood_group neighbourhood latitude longitude
## Length:473 Length:473 Min. :40.58 Min. :-74.08
## Class :character Class :character 1st Qu.:40.69 1st Qu.:-73.97
## Mode :character Mode :character Median :40.72 Median :-73.95
## Mean :40.73 Mean :-73.94
## 3rd Qu.:40.76 3rd Qu.:-73.92
## Max. :40.89 Max. :-73.74
##
## room_type price minimum_nights number_of_reviews
## Length:473 Min. : 21.00 Min. : 1.000 Min. : 0.00
## Class :character 1st Qu.: 50.00 1st Qu.: 1.000 1st Qu.: 1.00
## Mode :character Median : 70.00 Median : 2.000 Median : 5.00
## Mean : 80.69 Mean : 5.307 Mean : 23.61
## 3rd Qu.: 95.00 3rd Qu.: 4.000 3rd Qu.: 24.00
## Max. :525.00 Max. :60.000 Max. :356.00
##
## last_review reviews_per_month calculated_host_listings_count
## Length:473 Min. :0.020 Min. : 1.000
## Class :character 1st Qu.:0.230 1st Qu.: 1.000
## Mode :character Median :0.860 Median : 1.000
## Mean :1.482 Mean : 3.882
## 3rd Qu.:2.110 3rd Qu.: 3.000
## Max. :9.000 Max. :103.000
## NA's :92
## availability_365 private_room entire_apartment shared_room
## Min. : 0.0 Min. :0.0000 Min. :0 Min. :0.00000
## 1st Qu.: 0.0 1st Qu.:1.0000 1st Qu.:0 1st Qu.:0.00000
## Median : 62.0 Median :1.0000 Median :0 Median :0.00000
## Mean :111.9 Mean :0.9598 Mean :0 Mean :0.04017
## 3rd Qu.:203.0 3rd Qu.:1.0000 3rd Qu.:0 3rd Qu.:0.00000
## Max. :365.0 Max. :1.0000 Max. :0 Max. :1.00000
##
## cluster
## Min. :1
## 1st Qu.:1
## Median :1
## Mean :1
## 3rd Qu.:1
## Max. :1
## number_of_private <- filter(cluster1, room_type =='Private room')
number_of_entire <- filter(cluster1, room_type =='Entire home/apt')
number_of_shared <- filter(cluster1, room_type =='Shared room')
nrow(number_of_private)## [1] 454nrow(number_of_entire)## [1] 0nrow(number_of_shared)## [1] 19number_of_bronx <- filter(cluster1, neighbourhood_group =='Bronx')
number_of_manhattan <- filter(cluster1, neighbourhood_group =='Manhattan')
number_of_brooklyn <- filter(cluster1, neighbourhood_group =='Brooklyn')
number_of_queens <- filter(cluster1, neighbourhood_group =='Queens')
number_of_island <- filter(cluster1, neighbourhood_group =='Staten Island')
nrow(number_of_bronx)## [1] 18nrow(number_of_manhattan)## [1] 159nrow(number_of_brooklyn)## [1] 220nrow(number_of_queens)## [1] 74nrow(number_of_island)## [1] 2nrow(cluster1)## [1] 473mean(cluster1$price)## [1] 80.69133hist(cluster1$price,xlab = "Airbnb Price",col = "gray",border = "black", xlim = c(0,400) , ylim = c(0,400), breaks = 8)
d1 <- density(cluster1$price)
plot(d1)
summary(cluster1$price)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 21.00 50.00 70.00 80.69 95.00 525.00Cluster 2
Cluster 2 consists of 526 airbnbs. Half of the airbnb is located in Manhattan. Also a relative huge group is in Brooklyn district (37,26%) and 11,21 % of airbnb is in Queens. 526 out of 527 is ‘Entire home or apartment’. Based on summary table and density plot, mean price is around 190 dollars per night, 1st quartile is 120 and 3rd quartile is 223.5.
my.sf.point <- st_as_sf(x = cluster2_map_,
coords = c("longitude", "latitude"),
crs = "+proj=longlat +datum=WGS84 +ellps=WGS84 +towgs84=0,0,0")
# simple plot
# plot(my.sf.point)
# interactive map:
mapview(my.sf.point)# convert to sp object if needed
my.sp.point <- as(my.sf.point, "Spatial")
summary(cluster2)## id name host_id host_name
## Min. : 27644 Length:527 Min. : 52394 Length:527
## 1st Qu.: 8503370 Class :character 1st Qu.: 5598420 Class :character
## Median :19408844 Mode :character Median : 24467359 Mode :character
## Mean :18578594 Mean : 59844159
## 3rd Qu.:29442508 3rd Qu.: 85272560
## Max. :36404936 Max. :273568164
##
## neighbourhood_group neighbourhood latitude longitude
## Length:527 Length:527 Min. :40.51 Min. :-74.24
## Class :character Class :character 1st Qu.:40.69 1st Qu.:-73.99
## Mode :character Mode :character Median :40.73 Median :-73.96
## Mean :40.73 Mean :-73.96
## 3rd Qu.:40.76 3rd Qu.:-73.94
## Max. :40.87 Max. :-73.73
##
## room_type price minimum_nights number_of_reviews
## Length:527 Min. : 16.0 Min. : 1.000 Min. : 0.00
## Class :character 1st Qu.:120.0 1st Qu.: 2.000 1st Qu.: 1.00
## Mode :character Median :160.0 Median : 3.000 Median : 6.00
## Mean :190.1 Mean : 8.283 Mean : 25.06
## 3rd Qu.:223.5 3rd Qu.: 5.500 3rd Qu.: 25.00
## Max. :900.0 Max. :222.000 Max. :334.00
##
## last_review reviews_per_month calculated_host_listings_count
## Length:527 Min. : 0.010 Min. : 1.00
## Class :character 1st Qu.: 0.210 1st Qu.: 1.00
## Mode :character Median : 0.720 Median : 1.00
## Mean : 1.409 Mean : 12.51
## 3rd Qu.: 2.115 3rd Qu.: 2.00
## Max. :15.780 Max. :327.00
## NA's :95
## availability_365 private_room entire_apartment shared_room cluster
## Min. : 0.0 Min. :0.000000 Min. :0.0000 Min. :0 Min. :2
## 1st Qu.: 0.0 1st Qu.:0.000000 1st Qu.:1.0000 1st Qu.:0 1st Qu.:2
## Median : 44.0 Median :0.000000 Median :1.0000 Median :0 Median :2
## Mean :111.4 Mean :0.001898 Mean :0.9981 Mean :0 Mean :2
## 3rd Qu.:228.0 3rd Qu.:0.000000 3rd Qu.:1.0000 3rd Qu.:0 3rd Qu.:2
## Max. :365.0 Max. :1.000000 Max. :1.0000 Max. :0 Max. :2
## number_of_private <- filter(cluster2, room_type =='Private room')
number_of_entire <- filter(cluster2, room_type =='Entire home/apt')
number_of_shared <- filter(cluster2, room_type =='Shared room')
nrow(number_of_private)## [1] 1nrow(number_of_entire)## [1] 526nrow(number_of_shared)## [1] 0number_of_bronx <- filter(cluster2, neighbourhood_group =='Bronx')
number_of_manhattan <- filter(cluster2, neighbourhood_group =='Manhattan')
number_of_brooklyn <- filter(cluster2, neighbourhood_group =='Brooklyn')
number_of_queens <- filter(cluster2, neighbourhood_group =='Queens')
number_of_island <- filter(cluster2, neighbourhood_group =='Staten Island')
nrow(number_of_bronx)## [1] 6nrow(number_of_manhattan)## [1] 263nrow(number_of_brooklyn)## [1] 196nrow(number_of_queens)## [1] 59nrow(number_of_island)## [1] 3nrow(cluster2)## [1] 527mean(cluster2$price)## [1] 190.1347hist(cluster2$price,xlab = "Airbnb Price",col = "gray",border = "black", xlim = c(0,400) , ylim = c(0,400), breaks = 8)
d2 <- density(cluster2$price)
plot(d2)
summary(cluster2$price)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 16.0 120.0 160.0 190.1 223.5 900.0clustering only on location
Data Pre Processing
airbnb_sample_subset = subset(airbnb_sample,select=c(latitude, longitude))
airbnb_clus <- airbnb_sample_subsetData Visualisation
To have a better visualisation, the airbnb locations are marked on the map of NY
my.sf.point <- st_as_sf(x = airbnb_map,
coords = c("longitude", "latitude"),
crs = "+proj=longlat +datum=WGS84 +ellps=WGS84 +towgs84=0,0,0")
# simple plot
# plot(my.sf.point)
# interactive map:
mapview(my.sf.point)## Warning in cbind(`Feature ID` = fid, mat): number of rows of result is not a
## multiple of vector length (arg 1)# convert to sp object if needed
my.sp.point <- as(my.sf.point, "Spatial")Hopkins Statistics
Before clustering it is necessary to check whether the data is clusterable. Accoring to Hopkins Statisitcs equal to 0.95 I can assume that dataset is significanly a clusterable data.
get_clust_tendency(airbnb_clus, 2, graph=TRUE, gradient=list(low="red", mid="white", high="blue"))## $hopkins_stat
## [1] 0.9571082
##
## $plot
Clustering
Afterwards, as it is known that we can cluster is to choose the most optimal number of clusters. I will aply clustering algorithms such as kmeans.
At the beginning, I will find the most optimal number of clusters. I will use silhouette and wss statistics and then apply it to clustering algorithms: kmeans, PAM and CLARA. I have chosen a sample consisted of 1000 observation, so the hierarchical clustering is not visable and transparent.
Silhouette
In the first step, the silhouette method was used to choose optimal number of clusters.The silhouette value is a measure of how similar an object is to its own cluster (cohesion) compared to other clusters (separation). The silhouette ranges from −1 to +1, where a high value indicates that the object is well matched to its own cluster and poorly matched to neighboring clusters.[wikipedia]
Kmeans chart suggests dividing data into 7 clusters. PAM method chose the 5 clusters and CLARA grouped data into 4 clusters. Lastly, hcut recommends 6 clusters. However, the difference in silhouette width between the 2 and 10 clusters is not so visible in aforementioned methods.
library(gridExtra)
a <- fviz_nbclust(airbnb_clus, FUNcluster = kmeans, method = "silhouette") +
labs(subtitle = "silhouette - kmeans")
b <- fviz_nbclust(airbnb_clus, FUNcluster = cluster::pam, method = "silhouette") +
labs(subtitle = "silhouette - pam")
c <- fviz_nbclust(airbnb_clus, FUNcluster = cluster::clara, method = "silhouette") +
labs(subtitle = "silhouette - clara")
d <- fviz_nbclust(airbnb_clus, FUNcluster = hcut, method = "silhouette") +
labs(subtitle = "silhouette - hcut")
grid.arrange(a, b, c, d, ncol=2)
Total within sum of squares (WSS)
The Squared Error for each point is the square of the distance of the point from its representation i.e. its predicted cluster center. The WSS score is the sum of these Squared Errors for all the points. [https://medium.com/analytics-vidhya/how-to-determine-the-optimal-k-for-k-means-708505d204eb]
Elbow method
To compare results obtained by silhouette method, I will analyze charts of elbow method with WSS. Due to small changes in WSS between clusters, it is hard to verify the correct number of clusters. However, for each method according to elbow method, I shoyld choose 4 clusters.
e1 <- fviz_nbclust(airbnb_clus, kmeans, method = "wss",k.max = 10) +
labs(subtitle = "Elbow method - kmeans")
e2 <- fviz_nbclust(airbnb_clus, clara, method = "wss",k.max = 10) +
labs(subtitle = "Elbow method - clara")
e3 <- fviz_nbclust(airbnb_clus, pam, method = "wss",k.max = 10) +
labs(subtitle = "Elbow method - pam")
e4 <- fviz_nbclust(airbnb_clus, hcut, method = "wss",k.max = 10) +
labs(subtitle = "Elbow method - hcut")
grid.arrange(e1,e2,e3,e4, ncol=2, top = "Optimal number of clusters")
library(factoextra)
cl_kmeans <- eclust(airbnb_clus, k=4, FUNcluster="kmeans", hc_metric="pearson", graph=FALSE)
a <- fviz_silhouette(cl_kmeans)## cluster size ave.sil.width
## 1 1 235 0.37
## 2 2 312 0.51
## 3 3 70 0.37
## 4 4 383 0.36b <- fviz_cluster(cl_kmeans, data = airbnb_clus, elipse.type = "convex") + theme_minimal()
grid.arrange(a, b, ncol=2)
Kmeans
The Κ-means clustering algorithm uses iterative refinement to produce a final result. The algorithm inputs are the number of clusters Κ and the data set. The data set is a collection of features for each data point. The algorithms starts with initial estimates for the Κ centroids, which can either be randomly generated or randomly selected from the data set. The algorithm then iterates between two steps: Data assigment step and Centroid update step. [https://blogs.oracle.com/ai-and-datascience/post/introduction-to-k-means-clustering]
km4 <- eclust(airbnb_clus,k=4,hc_metric = 'euclidean', graph = FALSE)
km5 <- eclust(airbnb_clus,k=5,hc_metric = 'euclidean', graph = FALSE)
km6 <- eclust(airbnb_clus,k=6,hc_metric = 'euclidean', graph = FALSE)
k1 <- fviz_cluster(km4, geom = c("point")) + ggtitle('K-means with 4 clusters')
k2 <- fviz_cluster(km5, geom = c("point")) + ggtitle('K-means with 5 clusters')
k3 <- fviz_cluster(km6, geom = c("point")) + ggtitle('K-means with 6 clusters')
grid.arrange(arrangeGrob(k1, k2, k3, ncol=2 , top = "Clustering"))
PAM
-small datasets PAM is an implementation of the k-medoid method, i.e. a clustering technique that divides a dataset containing n objects into k groups (clusters) known a priori. A useful tool for assessing the quality of grouping is the silhouette. It is calculated for each object subject to grouping. On its basis, it is possible to determine whether the objects have been grouped correctly or whether they have been placed in the wrong clusters.
pam4 <- eclust(airbnb_clus,'pam',k=4,hc_metric = 'euclidean', graph = FALSE)
pam5<- eclust(airbnb_clus,'pam',k=5,hc_metric = 'euclidean', graph = FALSE)
pam6 <- eclust(airbnb_clus,'pam',k=6,hc_metric = 'euclidean', graph = FALSE)
p1 <- fviz_cluster(pam4, geom = c("point")) + ggtitle('PAM with 4 clusters')
p2 <- fviz_cluster(pam5, geom = c("point")) + ggtitle('PAM with 5 clusters')
p3 <- fviz_cluster(pam6, geom = c("point")) + ggtitle('PAM with 6 clusters')
grid.arrange(arrangeGrob(p1, p2, p3, ncol=2 , top = "Clustering"))
CLARA
-larger datasets CLARA draws multiple samples of the dataset, then applies PAM on each sample, and gives the best clustering as the oputput. [from presentation]
clara4 <- eclust(airbnb_clus,'clara',k=4,hc_metric = 'euclidean', graph = FALSE)
clara5 <- eclust(airbnb_clus,'clara',k=5,hc_metric = 'euclidean', graph = FALSE)
clara6 <- eclust(airbnb_clus,'clara',k=6,hc_metric = 'euclidean', graph = FALSE)
c1 <- fviz_cluster(clara4, geom = c("point")) + ggtitle('Clara with 4 clusters')
c2 <- fviz_cluster(clara5, geom = c("point")) + ggtitle('Clara with 5 clusters')
c3 <- fviz_cluster(clara6, geom = c("point")) + ggtitle('Clara with 6 clusters')
grid.arrange(arrangeGrob(c1, c2, c3, ncol=2 , top = "Clustering"))
Post diagnostics
km4$silinfo$clus.avg.widths## [1] 0.3738984 0.5111021 0.3655410 0.3590772km5$silinfo$clus.avg.widths## [1] 0.3656397 0.4962927 0.3640703 0.3576738 0.4113436km6$silinfo$clus.avg.widths## [1] 0.4430948 0.5222087 0.2732862 0.4301002 0.4937648 0.1974620km4_s <- fviz_silhouette(km4) + ggtitle('K-means with 4 clusters')## cluster size ave.sil.width
## 1 1 235 0.37
## 2 2 312 0.51
## 3 3 70 0.37
## 4 4 383 0.36km5_s <- fviz_silhouette(km5) + ggtitle('K-means with 5 clusters')## cluster size ave.sil.width
## 1 1 217 0.37
## 2 2 298 0.50
## 3 3 40 0.36
## 4 4 382 0.36
## 5 5 63 0.41km6_s <- fviz_silhouette(km6) + ggtitle('K-means with 6 clusters')## cluster size ave.sil.width
## 1 1 167 0.44
## 2 2 287 0.52
## 3 3 52 0.27
## 4 4 293 0.43
## 5 5 78 0.49
## 6 6 123 0.20grid.arrange(arrangeGrob(km4_s,km5_s, km6_s, ncol=2 , top = "Clustering"))
Clusters’ analysys
Taking into consideration all aforementioned methods, I eventually chose 4 clusters. Below I am joining two tabels to see cluster1, cluster2, cluster3 and cluster4 statistics to compare them.
km_ = kmeans(airbnb_clus, centers = 4, nstart=25)
airbnb_clus_map_ <- cbind(airbnb_sample_, cluster = km_$cluster)
new_airbnb_ <- cbind(airbnb_sample_, cluster = km_$cluster)
cluster1 <- new_airbnb_[new_airbnb_$cluster == 1,]
cluster2 <- new_airbnb_[new_airbnb_$cluster == 2,]
cluster3 <- new_airbnb_[new_airbnb_$cluster == 3,]
cluster4 <- new_airbnb_[new_airbnb_$cluster == 4,]
cluster1_map_ <- new_airbnb_[airbnb_clus_map_$cluster == 1,]
cluster2_map_ <- new_airbnb_[airbnb_clus_map_$cluster == 2,]
cluster3_map_ <- new_airbnb_[airbnb_clus_map_$cluster == 3,]
cluster4_map_ <- new_airbnb_[airbnb_clus_map_$cluster == 4,]Results
Cluster 1
Cluster 1 consists of 231 airbnbs. Most of the airbnb is located in Manhattan (70%). The rest aribnbs are located in Queens. Based on summary table and density plot, mean price is around 125 dollars per night, 1st quartile is 25 and 3rd quartile is 150. 103 airbnbs are clasified as ‘entire apartemnt’ and 123 of them are private rooms.
my.sf.point <- st_as_sf(x = cluster1_map_,
coords = c("longitude", "latitude"),
crs = "+proj=longlat +datum=WGS84 +ellps=WGS84 +towgs84=0,0,0")
# simple plot
# plot(my.sf.point)
# interactive map:
mapview(my.sf.point)# convert to sp object if needed
my.sp.point <- as(my.sf.point, "Spatial")
nrow(cluster1)## [1] 231summary(cluster1)## id name host_id host_name
## Min. : 9518 Length:231 Min. : 31374 Length:231
## 1st Qu.: 8832204 Class :character 1st Qu.: 9398728 Class :character
## Median :18503701 Mode :character Median : 32625342 Mode :character
## Mean :18772126 Mean : 70102918
## 3rd Qu.:28868838 3rd Qu.:113395182
## Max. :36477588 Max. :272557714
##
## neighbourhood_group neighbourhood latitude longitude
## Length:231 Length:231 Min. :40.74 Min. :-73.98
## Class :character Class :character 1st Qu.:40.77 1st Qu.:-73.96
## Mode :character Mode :character Median :40.80 Median :-73.94
## Mean :40.80 Mean :-73.94
## 3rd Qu.:40.82 3rd Qu.:-73.92
## Max. :40.89 Max. :-73.85
##
## room_type price minimum_nights number_of_reviews
## Length:231 Min. : 25.0 Min. : 1.000 Min. : 0.00
## Class :character 1st Qu.: 66.5 1st Qu.: 1.000 1st Qu.: 1.00
## Mode :character Median : 99.0 Median : 2.000 Median : 6.00
## Mean :124.6 Mean : 6.294 Mean : 21.09
## 3rd Qu.:150.0 3rd Qu.: 5.000 3rd Qu.: 21.00
## Max. :900.0 Max. :122.000 Max. :268.00
##
## last_review reviews_per_month calculated_host_listings_count
## Length:231 Min. :0.020 Min. : 1.000
## Class :character 1st Qu.:0.210 1st Qu.: 1.000
## Mode :character Median :0.710 Median : 1.000
## Mean :1.351 Mean : 3.926
## 3rd Qu.:2.030 3rd Qu.: 2.000
## Max. :9.000 Max. :103.000
## NA's :48
## availability_365 private_room entire_apartment shared_room
## Min. : 0.0 Min. :0.0000 Min. :0.0000 Min. :0.00000
## 1st Qu.: 0.0 1st Qu.:0.0000 1st Qu.:0.0000 1st Qu.:0.00000
## Median : 33.0 Median :1.0000 Median :0.0000 Median :0.00000
## Mean :109.9 Mean :0.5325 Mean :0.4459 Mean :0.02165
## 3rd Qu.:242.5 3rd Qu.:1.0000 3rd Qu.:1.0000 3rd Qu.:0.00000
## Max. :365.0 Max. :1.0000 Max. :1.0000 Max. :1.00000
##
## cluster
## Min. :1
## 1st Qu.:1
## Median :1
## Mean :1
## 3rd Qu.:1
## Max. :1
## number_of_private <- filter(cluster1, room_type =='Private room')
number_of_entire <- filter(cluster1, room_type =='Entire home/apt')
number_of_shared <- filter(cluster1, room_type =='Shared room')
nrow(number_of_private)## [1] 123nrow(number_of_entire)## [1] 103nrow(number_of_shared)## [1] 5number_of_bronx <- filter(cluster1, neighbourhood_group =='Bronx')
number_of_manhattan <- filter(cluster1, neighbourhood_group =='Manhattan')
number_of_brooklyn <- filter(cluster1, neighbourhood_group =='Brooklyn')
number_of_queens <- filter(cluster1, neighbourhood_group =='Queens')
number_of_island <- filter(cluster1, neighbourhood_group =='Staten Island')
nrow(number_of_bronx)## [1] 19nrow(number_of_manhattan)## [1] 162nrow(number_of_brooklyn)## [1] 0nrow(number_of_queens)## [1] 50nrow(number_of_island)## [1] 0mean(cluster1$price)## [1] 124.5714hist(cluster1$price,xlab = "Airbnb Price",col = "gray",border = "black", xlim = c(0,400) , ylim = c(0,200), breaks = 8)
d1 <- density(cluster1$price)
plot(d1)
summary(cluster1$price)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 25.0 66.5 99.0 124.6 150.0 900.0Cluster 2
Cluster 2 consists of 70 airbnbs. Most of the airbnb is located in East Queens (93%). The rest aribnbs are located in Bronx Based on summary table and density plot, mean price is around 90 dollars per night, 1st quartile is 50 and 3rd quartile is 100. 32 airbnbs are clasified as ‘entire apartemnt’ and 38 of them are private rooms.
my.sf.point <- st_as_sf(x = cluster2_map_,
coords = c("longitude", "latitude"),
crs = "+proj=longlat +datum=WGS84 +ellps=WGS84 +towgs84=0,0,0")
# simple plot
# plot(my.sf.point)
# interactive map:
mapview(my.sf.point)# convert to sp object if needed
my.sp.point <- as(my.sf.point, "Spatial")
nrow(cluster2)## [1] 70summary(cluster2)## id name host_id host_name
## Min. : 765563 Length:70 Min. : 445894 Length:70
## 1st Qu.:17410253 Class :character 1st Qu.: 23425614 Class :character
## Median :26764064 Mode :character Median : 94315680 Mode :character
## Mean :24049823 Mean :106834910
## 3rd Qu.:31906802 3rd Qu.:183438968
## Max. :36412355 Max. :263504959
##
## neighbourhood_group neighbourhood latitude longitude
## Length:70 Length:70 Min. :40.58 Min. :-73.89
## Class :character Class :character 1st Qu.:40.71 1st Qu.:-73.86
## Mode :character Mode :character Median :40.73 Median :-73.83
## Mean :40.73 Mean :-73.83
## 3rd Qu.:40.76 3rd Qu.:-73.80
## Max. :40.86 Max. :-73.73
##
## room_type price minimum_nights number_of_reviews
## Length:70 Min. : 28.00 Min. : 1.000 Min. : 0.0
## Class :character 1st Qu.: 50.00 1st Qu.: 1.000 1st Qu.: 4.0
## Mode :character Median : 65.00 Median : 2.000 Median : 12.0
## Mean : 89.69 Mean : 4.557 Mean : 30.4
## 3rd Qu.: 99.75 3rd Qu.: 3.000 3rd Qu.: 37.0
## Max. :359.00 Max. :60.000 Max. :356.0
##
## last_review reviews_per_month calculated_host_listings_count
## Length:70 Min. : 0.0200 Min. : 1.0
## Class :character 1st Qu.: 0.6675 1st Qu.: 1.0
## Mode :character Median : 1.8800 Median : 2.0
## Mean : 2.5628 Mean : 4.9
## 3rd Qu.: 3.2800 3rd Qu.: 4.0
## Max. :15.7800 Max. :103.0
## NA's :10
## availability_365 private_room entire_apartment shared_room cluster
## Min. : 0.0 Min. :0.0000 Min. :0.0000 Min. :0 Min. :2
## 1st Qu.: 58.0 1st Qu.:0.0000 1st Qu.:0.0000 1st Qu.:0 1st Qu.:2
## Median :134.5 Median :1.0000 Median :0.0000 Median :0 Median :2
## Mean :157.3 Mean :0.5429 Mean :0.4571 Mean :0 Mean :2
## 3rd Qu.:282.5 3rd Qu.:1.0000 3rd Qu.:1.0000 3rd Qu.:0 3rd Qu.:2
## Max. :365.0 Max. :1.0000 Max. :1.0000 Max. :0 Max. :2
## number_of_private <- filter(cluster2, room_type =='Private room')
number_of_entire <- filter(cluster2, room_type =='Entire home/apt')
number_of_shared <- filter(cluster2, room_type =='Shared room')
nrow(number_of_private)## [1] 38nrow(number_of_entire)## [1] 32nrow(number_of_shared)## [1] 0number_of_bronx <- filter(cluster2, neighbourhood_group =='Bronx')
number_of_manhattan <- filter(cluster2, neighbourhood_group =='Manhattan')
number_of_brooklyn <- filter(cluster2, neighbourhood_group =='Brooklyn')
number_of_queens <- filter(cluster2, neighbourhood_group =='Queens')
number_of_island <- filter(cluster2, neighbourhood_group =='Staten Island')
nrow(number_of_bronx)## [1] 5nrow(number_of_manhattan)## [1] 0nrow(number_of_brooklyn)## [1] 0nrow(number_of_queens)## [1] 65nrow(number_of_island)## [1] 0mean(cluster2$price)## [1] 89.68571hist(cluster2$price,xlab = "Airbnb Price",col = "gray",border = "black", xlim = c(0,1000) , ylim = c(0,100), breaks = 4)
d2 <- density(cluster2$price)
plot(d2)
summary(cluster2$price)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 28.00 50.00 65.00 89.69 99.75 359.00Cluster 3
Cluster 1 consists of 378 airbnbs. Most of the airbnb is located in Brooklyn (98%). Based on summary table and density plot, mean price is around 114 dollars per night, 1st quartile is 60 and 3rd quartile is 139. 199 airbnbs are clasified as ‘entire apartemnt’ and 171 of them are private rooms.
my.sf.point <- st_as_sf(x = cluster3_map_,
coords = c("longitude", "latitude"),
crs = "+proj=longlat +datum=WGS84 +ellps=WGS84 +towgs84=0,0,0")
# simple plot
# plot(my.sf.point)
# interactive map:
mapview(my.sf.point)# convert to sp object if needed
my.sp.point <- as(my.sf.point, "Spatial")
nrow(cluster3)## [1] 378summary(cluster3)## id name host_id host_name
## Min. : 8025 Length:378 Min. : 22486 Length:378
## 1st Qu.: 9799367 Class :character 1st Qu.: 6070007 Class :character
## Median :20332798 Mode :character Median : 27250906 Mode :character
## Mean :19534555 Mean : 63309123
## 3rd Qu.:29323210 3rd Qu.: 90805038
## Max. :36351543 Max. :273139430
##
## neighbourhood_group neighbourhood latitude longitude
## Length:378 Length:378 Min. :40.51 Min. :-74.24
## Class :character Class :character 1st Qu.:40.67 1st Qu.:-73.97
## Mode :character Mode :character Median :40.69 Median :-73.95
## Mean :40.68 Mean :-73.95
## 3rd Qu.:40.70 3rd Qu.:-73.93
## Max. :40.72 Max. :-73.88
##
## room_type price minimum_nights number_of_reviews
## Length:378 Min. : 21.0 Min. : 1.000 Min. : 0.00
## Class :character 1st Qu.: 60.0 1st Qu.: 1.000 1st Qu.: 1.00
## Mode :character Median : 90.0 Median : 2.000 Median : 6.00
## Mean :114.4 Mean : 5.847 Mean : 23.94
## 3rd Qu.:139.0 3rd Qu.: 4.750 3rd Qu.: 24.00
## Max. :900.0 Max. :222.000 Max. :323.00
##
## last_review reviews_per_month calculated_host_listings_count
## Length:378 Min. :0.010 Min. : 1.000
## Class :character 1st Qu.:0.220 1st Qu.: 1.000
## Mode :character Median :0.780 Median : 1.000
## Mean :1.417 Mean : 2.548
## 3rd Qu.:2.150 3rd Qu.: 2.000
## Max. :8.970 Max. :43.000
## NA's :61
## availability_365 private_room entire_apartment shared_room
## Min. : 0.0 Min. :0.0000 Min. :0.0000 Min. :0.00000
## 1st Qu.: 0.0 1st Qu.:0.0000 1st Qu.:0.0000 1st Qu.:0.00000
## Median : 54.5 Median :1.0000 Median :0.0000 Median :0.00000
## Mean :108.4 Mean :0.5265 Mean :0.4524 Mean :0.02116
## 3rd Qu.:217.0 3rd Qu.:1.0000 3rd Qu.:1.0000 3rd Qu.:0.00000
## Max. :365.0 Max. :1.0000 Max. :1.0000 Max. :1.00000
##
## cluster
## Min. :3
## 1st Qu.:3
## Median :3
## Mean :3
## 3rd Qu.:3
## Max. :3
## number_of_private <- filter(cluster3, room_type =='Private room')
number_of_entire <- filter(cluster3, room_type =='Entire home/apt')
number_of_shared <- filter(cluster3, room_type =='Shared room')
nrow(number_of_private)## [1] 199nrow(number_of_entire)## [1] 171nrow(number_of_shared)## [1] 8number_of_bronx <- filter(cluster3, neighbourhood_group =='Bronx')
number_of_manhattan <- filter(cluster3, neighbourhood_group =='Manhattan')
number_of_brooklyn <- filter(cluster3, neighbourhood_group =='Brooklyn')
number_of_queens <- filter(cluster3, neighbourhood_group =='Queens')
number_of_island <- filter(cluster3, neighbourhood_group =='Staten Island')
nrow(number_of_bronx)## [1] 0nrow(number_of_manhattan)## [1] 0nrow(number_of_brooklyn)## [1] 367nrow(number_of_queens)## [1] 6nrow(number_of_island)## [1] 5mean(cluster3$price)## [1] 114.3757hist(cluster3$price, xlim = c(0,500) , ylim = c(0,400))
d3 <- density(cluster3$price)
plot(d3)
summary(cluster3$price)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 21.0 60.0 90.0 114.4 139.0 900.0Cluster 4
Cluster 4 consists of 321 airbnbs. Most of the airbnb is located in Manhattan (81%). 15% aribnbs are located in North West Brooklyn. Based on summary table and density plot, mean price is around 187 dollars per night, 1st quartile is 105 and 3rd quartile is 225. 220 airbnbs are clasified as ‘entire apartemnt’ and 95 of them are private rooms.
my.sf.point <- st_as_sf(x = cluster4_map_,
coords = c("longitude", "latitude"),
crs = "+proj=longlat +datum=WGS84 +ellps=WGS84 +towgs84=0,0,0")
# simple plot
# plot(my.sf.point)
# interactive map:
mapview(my.sf.point)# convert to sp object if needed
my.sp.point <- as(my.sf.point, "Spatial")
nrow(cluster4)## [1] 321summary(cluster4)## id name host_id host_name
## Min. : 12048 Length:321 Min. : 7549 Length:321
## 1st Qu.: 8615062 Class :character 1st Qu.: 7339399 Class :character
## Median :19413230 Mode :character Median : 30933227 Mode :character
## Mean :18608288 Mean : 65304210
## 3rd Qu.:29704596 3rd Qu.:107434423
## Max. :36426720 Max. :273568164
##
## neighbourhood_group neighbourhood latitude longitude
## Length:321 Length:321 Min. :40.69 Min. :-74.02
## Class :character Class :character 1st Qu.:40.72 1st Qu.:-74.00
## Mode :character Mode :character Median :40.73 Median :-73.99
## Mean :40.74 Mean :-73.98
## 3rd Qu.:40.75 3rd Qu.:-73.98
## Max. :40.78 Max. :-73.92
##
## room_type price minimum_nights number_of_reviews
## Length:321 Min. : 16.0 Min. : 1.000 Min. : 0.00
## Class :character 1st Qu.:105.0 1st Qu.: 2.000 1st Qu.: 1.00
## Mode :character Median :160.0 Median : 3.000 Median : 4.00
## Mean :187.2 Mean : 9.009 Mean : 25.94
## 3rd Qu.:225.0 3rd Qu.: 7.000 3rd Qu.: 26.00
## Max. :900.0 Max. :180.000 Max. :334.00
##
## last_review reviews_per_month calculated_host_listings_count
## Length:321 Min. :0.020 Min. : 1.00
## Class :character 1st Qu.:0.160 1st Qu.: 1.00
## Mode :character Median :0.660 Median : 1.00
## Mean :1.278 Mean : 19.37
## 3rd Qu.:1.950 3rd Qu.: 2.00
## Max. :6.430 Max. :327.00
## NA's :68
## availability_365 private_room entire_apartment shared_room
## Min. : 0.0 Min. :0.000 Min. :0.0000 Min. :0.00000
## 1st Qu.: 0.0 1st Qu.:0.000 1st Qu.:0.0000 1st Qu.:0.00000
## Median : 28.0 Median :0.000 Median :1.0000 Median :0.00000
## Mean :106.7 Mean :0.296 Mean :0.6854 Mean :0.01869
## 3rd Qu.:207.0 3rd Qu.:1.000 3rd Qu.:1.0000 3rd Qu.:0.00000
## Max. :365.0 Max. :1.000 Max. :1.0000 Max. :1.00000
##
## cluster
## Min. :4
## 1st Qu.:4
## Median :4
## Mean :4
## 3rd Qu.:4
## Max. :4
## number_of_private <- filter(cluster4, room_type =='Private room')
number_of_entire <- filter(cluster4, room_type =='Entire home/apt')
number_of_shared <- filter(cluster4, room_type =='Shared room')
nrow(number_of_private)## [1] 95nrow(number_of_entire)## [1] 220nrow(number_of_shared)## [1] 6number_of_bronx <- filter(cluster4, neighbourhood_group =='Bronx')
number_of_manhattan <- filter(cluster4, neighbourhood_group =='Manhattan')
number_of_brooklyn <- filter(cluster4, neighbourhood_group =='Brooklyn')
number_of_queens <- filter(cluster4, neighbourhood_group =='Queens')
number_of_island <- filter(cluster4, neighbourhood_group =='Staten Island')
nrow(number_of_bronx)## [1] 0nrow(number_of_manhattan)## [1] 260nrow(number_of_brooklyn)## [1] 49nrow(number_of_queens)## [1] 12nrow(number_of_island)## [1] 0mean(cluster4$price)## [1] 187.1651hist(cluster4$price, xlim = c(0,300) , ylim = c(0,500))
d4 <- density(cluster4$price)
plot(d4)
summary(cluster4$price)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 16.0 105.0 160.0 187.2 225.0 900.0Conclusion
To sum up, the airbnb location were clustered in two scenarios:- clustering on latitude and longitude
- clustering on: latitude, longitude, price and type of airbnb.
- clustering on locations The algorithms like K-means, PAM, CLARA suggested dividing data into 4-7 clusters depending on the method. However, basing on silhouette method, average silhouette width increases slightly, increasing number of clusters from 4. Thus, after analyzing created plots, I chose 4 clusters. Unfortunately, Hierarchical Clustering is not so helpful, as the dataset is quite huge.
-
clustering on locations, price and type of airbnb The algorithms like K-means, PAM, CLARA suggested dividing data into 4-7 clusters depending on the method. However, basing on silhouette method, average silhouette width increases slightly, increasing number of clusters from 4. Thus, after analyzing created plots, I chose 4 clusters. Unfortunately, Hierarchical Clustering is not so helpful, as the dataset is quite huge.
Comparing two scenarios, even though clustering on locations, price and type only have 2 clusters, clusters are divided in a significant way. Whereas, as expected, clustering by location is very successful in dividing the data into New York City districts. So we can compare prices and airbnb locations in 4 areas that can be considered as NY districts