For the purposes of this post I will be reviewing Airbnb prices, ratings, and aspects in Cincinnati Ohio during their Oktoberfest celebration.
Cincinnati is a fairly large city. While no comparison to that of Chicago and New York, we still have some landmarks to write home about. I was curious to see how our rental/ vacation options seemed in our city.
We would want to look at things like price, both the current and the discounted prices. Furthermore we can see the star rating as well as the number of reviews. We can also see the rental types and descriptions which can be used for characterizing houses.
Initially there were errors where many of the scrapes were returning zero after a few tests. This is mainly because there was no system sleep input into the scraping process. It is most likely that AirBNB put us on hold until we learned to stop scraping so rapidly.
Additionally before the system sleep function was utilized there is a feature on the site that caused issues. Whilst scrolling through potential rentals AirBNB will show you a sliding row of additional rentals that paid for a little extra advertising space. Our scrapes were pulling these despite them not actually always fitting our criteria added to the filters above. So first we looked to remove their elements. However, upon further analysis we determined that the system sleep allowed us to pull consistently without interruption.
The page loops
Because of how dynamic the site is we could not simply set up a page vector. We had to actually pull each url directly. This was because AirBNB thinks they are special and change seemingly random parts of their page urls. The code can be seen below.
# | echo: falselibrary(tidyverse)
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr 1.1.4 ✔ readr 2.1.5
✔ forcats 1.0.0 ✔ stringr 1.5.1
✔ ggplot2 3.4.4 ✔ tibble 3.2.1
✔ lubridate 1.9.3 ✔ tidyr 1.3.0
✔ purrr 1.0.2
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag() masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(rvest)
Warning: package 'rvest' was built under R version 4.3.3
Attaching package: 'rvest'
The following object is masked from 'package:readr':
guess_encoding
library(lubridate)library(httr)library(chromote)
Warning: package 'chromote' was built under R version 4.3.3
library(dplyr)library(readr)
ben_fun <-function(url) {airbnb_url <-read_html_live(url)Sys.sleep(2)## Rental type ##Rental_type <- airbnb_url %>%html_elements("div.t1jojoys") %>%html_text2()## Rental Description ##Rental_desc <- airbnb_url %>%html_elements("div.lxq01kf") %>%html_elements("div.fb4nyux") %>%html_elements("span.t6mzqp7") %>%html_text2()## Bed n Bath (too be separated in data frame) ##Bedrooms <- airbnb_url %>%html_elements("div.lxq01kf") %>%html_elements("div.fb4nyux") %>%html_text2() %>%as.character() %>%data.frame() %>%filter(row_number()%%2==0) %>%transmute(beds =as.numeric(substr(.,1,2)))## Per night price ##Current_price <- airbnb_url %>%html_elements("div.lxq01kf") %>%html_elements("div.pquyp1l") %>%html_elements("span._1y74zjx") %>%html_text2()# If the previous price exists, scrape the previous price.# If the previous price does not exist, record an NAPrevious_price <-ifelse(airbnb_url %>%html_elements("div.lxq01kf") %>%html_elements("div.pquyp1l") %>%as.character() %>%str_detect("_1l85cgq") ==TRUE, airbnb_url %>%html_elements("div.lxq01kf") %>%html_elements("div.pquyp1l") %>%html_elements("span._1l85cgq") %>%html_text2(),NA)## Stars ##Rating <- airbnb_url %>%html_elements("div.lxq01kf") %>%html_elements("div.g1qv1ctd") %>%html_elements("div.t1a9j9y7") %>%html_elements("span.ru0q88m.atm_cp_1ts48j8.dir.dir-ltr") %>%html_text2() airbnb_df <-data.frame(Rental_type, Rental_desc, Current_price, Previous_price, Rating) %>%bind_cols(Bedrooms)return(airbnb_df)}pages <-c("page1link","page2link","page3link","page4link","page5link","page6link")## Loop#### Our scraping function was labeled "ben_fun"Airbnb_Full <-data.frame()## Here is the for loop comented out so that it will run ;) #for (i in seq_along(pages)){# Airbnb_Full <- # ben_fun(pages[i]) %>% # bind_rows(Airbnb_Full)}
Data cleaning
After successfully scraping 6 pages we had to fix our data set. Our price variables were not being counted as numeric. Additionally we decided to remove the latter half of our rating pull which happened to be our number of ratings.
Results from data
Some basic pulls we managed to handle. How many rentals were available with only 2 beds, despite us saying we had three people staying in the rental. This could help you look for cheaper options, or tell you to pack light if you book late.
We were additionally able to discover our average rating of 4.92 stars as well as our highest per night price during the Cincinnati Oktoberfest season.
Conclusion
With a bit more dplyr work and some significant more expertise this data can help you weed through rental options in a more efficient manner. It’s even possible that we can sort by description, which AirBNB does not allow you to do. With some proper coding you can search for rentals marked as “romantic getaway”, even if they didn’t pay for some extra ad space! This is some fun data and a great challenge and I hope to improve.
