1. Sentiment Analysis of Uber and Airbnb in New York

The analysis examines the sentiment trends in online media coverage of Uber and Airbnb, utilizing sentiment scores derived from text data, based on articles collected from 2011 till 2024 in a Excel file named "Perception_media_news.xls".

Load Required Libraries

library(readxl)
library(dplyr)
library(lubridate)
library(tidyr)
library(tidytext)
library(ggplot2)
library(rvest)
library(tinytex)
library(writexl)
library(grid)
library(ggimage)
library(png)
library(gridExtra)
library(magick)
library(cowplot)
library(lmtest)
library(sandwich)
library(nnet)

Load and Combine Data

Load the dataset of the online news created in Excel named "Perception_media_news.xls" and combine it into one dataset.

file_path <- "/Users/mariamrochi/Library/Containers/com.microsoft.Excel/Data/Desktop/Master Thesis/Public Sentiment/Perception_media_news.xls"


uber_data <- read_excel(file_path, sheet = "Uber")
airbnb_data <- read_excel(file_path, sheet = "Airbnb")


combined_data <- bind_rows(
  uber_data %>% mutate(company = "Uber"),
  airbnb_data %>% mutate(company = "Airbnb")
)

Scrape Text Data from URLs

A function is defined to scrape text data from URLs of the articles collected in the Excel file Perception_media_news.xls. The scrape_article function processes URLs to extract text, which is then stored in a new column as a basis for sentiment analysis.

scrape_article <- function(url) {
  tryCatch({
    page <- read_html(url)
    article_text <- page %>%
      html_nodes("body") %>%
      html_text() %>%
      paste(collapse = " ")
    return(article_text)
  }, error = function(e) {
    return(NA)
  })
}

combined_data$article_text <- sapply(combined_data$url, scrape_article)

Load Sentiment Lexicon

The Bing sentiment lexicon is loaded. This lexicon categorizes words as either positive or negative, providing a reference for classifying sentiment in the text data.

sentiment_lexicon <- get_sentiments("bing")

Sentiment Analysis

Text data is tokenized into individual words, and sentiment scores are calculated for each word using the sentiment lexicon. A sentiment score is assigned (1 for positive, -1 for negative), and the total sentiment score is aggregated by article.

article_sentiments <- combined_data %>%
  unnest_tokens(word, article_text) %>%
  inner_join(sentiment_lexicon, by = "word") %>%
  mutate(sentiment_score = ifelse(sentiment == "positive", 1, -1)) %>%
  group_by(url) %>%
  summarize(total_score = sum(sentiment_score, na.rm = TRUE), .groups = "drop")


combined_data <- left_join(combined_data, article_sentiments, by = "url")

Sentiment by Company

The overall sentiment is calculated for each company. The total number of articles, and the counts of positive and negative articles, are summarized. A sentiment score is calculated for each company by normalizing the positive and negative article counts.

summary_table_by_company <- combined_data %>%
  group_by(company) %>%
  summarize(
    mean_sentiment = mean(total_score, na.rm = TRUE),
    total_articles = n(),
    positive_articles = sum(total_score > 0, na.rm = TRUE),
    negative_articles = sum(total_score < 0, na.rm = TRUE)
  ) %>%
  mutate(
    sentiment_score = ifelse(
      (positive_articles + negative_articles) > 0,  
      (positive_articles - negative_articles) / (positive_articles + negative_articles),
      0  
    )
  )

print(summary_table_by_company)
## # A tibble: 2 × 6
##   company mean_sentiment total_articles positive_articles negative_articles
##   <chr>            <dbl>          <int>             <int>             <int>
## 1 Airbnb           -158.            113                37                66
## 2 Uber             -148.            134                63                62
## # ℹ 1 more variable: sentiment_score <dbl>
table_grob <- tableGrob(summary_table_by_company)


png("summary_table.png", width = 900, height = 200, res = 100)
grid.draw(table_grob)  
dev.off()  
## quartz_off_screen 
##                 2

Filter “Unkown” Data

The code classifies articles as either related to Uber or Airbnb based on their titles and filters out irrelevant articles that do not mention either company.

combined_data <- combined_data %>%
  mutate(company = case_when(
    grepl("uber", title_article, ignore.case = TRUE) ~ "Uber",
    grepl("airbnb", title_article, ignore.case = TRUE) ~ "Airbnb",
    TRUE ~ "Unknown"
  )) %>%
  filter(company != "Unknown")

Aggregate Data and Calculate Sentiment Scores by Month

The code performs monthly aggregation and sentiment analysis of articles and groups the data by company and month, summarizing the number of positive, negative, and total articles.

summary_table_positive_negative_month <- combined_data %>%
  mutate(
    publication_date = as.Date(date_article),
    month_year = floor_date(publication_date, "month")
  ) %>%
  group_by(company, month_year) %>%
  summarize(
    positive_articles = sum(total_score > 0, na.rm = TRUE),
    negative_articles = sum(total_score < 0, na.rm = TRUE),
    total_articles = n(),
    .groups = "drop"
  )


print(summary_table_positive_negative_month)
## # A tibble: 94 × 5
##    company month_year positive_articles negative_articles total_articles
##    <chr>   <date>                 <int>             <int>          <int>
##  1 Airbnb  2012-11-01                 1                 0              1
##  2 Airbnb  2013-01-01                 1                 0              1
##  3 Airbnb  2013-05-01                 0                 1              1
##  4 Airbnb  2013-06-01                 1                 1              2
##  5 Airbnb  2013-10-01                 2                 3              5
##  6 Airbnb  2014-04-01                 1                 0              1
##  7 Airbnb  2014-05-01                 1                 1              2
##  8 Airbnb  2014-08-01                 0                 1              1
##  9 Airbnb  2014-10-01                 2                 2              4
## 10 Airbnb  2015-01-01                 0                 3              3
## # ℹ 84 more rows

Calculate and Visualize Uber Sentiment Trend by Month

The code analyzes the sentiment trend of Uber over time by calculating the mean sentiment score per month and visualizing it with geomline using ggplot2. Prints the results in a table format.

uber_sentiment_trend <- combined_data %>%
  filter(company == "Uber") %>%
  mutate(
    publication_date = as.Date(date_article),
    month_year = floor_date(publication_date, "month")
  ) %>%
  group_by(month_year) %>%
  summarize(
    mean_sentiment = mean(total_score, na.rm = TRUE),
    .groups = "drop"
  )

uber_sentiment_trend$company <- "Uber"

ggplot(uber_sentiment_trend, aes(x = month_year, y = mean_sentiment, color = company)) +
  geom_line(size = 1) + 
  geom_point(color = "#DCAE96", size = 1.5) +  
  scale_x_date(
    date_breaks = "6 months",  
    date_labels = "%b %Y"
  ) +
  labs(
    title = "Mean Sentiment Trend for Uber",
    x = "",
    y = "Mean Sentiment Score",
    caption = "Source: dataset (perception_media_news)"
  ) +
  scale_color_manual(values = c("Uber" = "#FF69B4","Airbnb" = "#DCAE96"),
                     name = "Company"
  ) + 
  theme_linedraw() +
  theme(
    axis.text.x = element_text(angle = 90, hjust = 1),
    axis.text.y = element_text(size = 12),
    plot.title = element_text(size = 16, face = "bold"),
    plot.caption = element_text(hjust = 0, size = 10),
    legend.position = "right"
   
)

print(uber_sentiment_trend)
## # A tibble: 51 × 3
##    month_year mean_sentiment company
##    <date>              <dbl> <chr>  
##  1 2012-06-01            6   Uber   
##  2 2012-09-01         -175   Uber   
##  3 2012-10-01            6   Uber   
##  4 2012-11-01          511   Uber   
##  5 2013-04-01         -305   Uber   
##  6 2013-12-01            3   Uber   
##  7 2014-09-01          183   Uber   
##  8 2014-11-01          482   Uber   
##  9 2014-12-01          -38.5 Uber   
## 10 2015-01-01         -100   Uber   
## # ℹ 41 more rows

Calculate and Visualize Airbnb Sentiment Trend by Month

The code analyzes the sentiment trend of Airbnb over time by calculating the mean sentiment score per month and visualizing it with geomline using ggplot2. Prints the results in a table format.

airbnb_sentiment_trend <- combined_data %>%
  filter(company == "Airbnb") %>%
  mutate(
    publication_date = as.Date(date_article),
    month_year = floor_date(publication_date, "month")
  ) %>%
  group_by(month_year) %>%
  summarize(
    mean_sentiment = mean(total_score, na.rm = TRUE),
    .groups = "drop"
  )

airbnb_sentiment_trend$company <- "Airbnb"

ggplot(airbnb_sentiment_trend, aes(x = month_year, y = mean_sentiment, color = company)) +
  geom_line(size = 1) + 
  geom_point(color = "#FF69B4", size = 1.5) +  
  scale_x_date(
    date_breaks = "6 months",  
    date_labels = "%b %Y"
  ) +
  labs(
    title = "Mean Sentiment Trend for Airbnb",
    x = "",
    y = "Mean Sentiment Score",
    caption = "Source: dataset (perception_media_news)",
  ) +
scale_color_manual(values = c("Uber" = "#FF69B4","Airbnb" = "#DCAE96"),
                     name = "Company"
) +
  theme_linedraw() +
  theme(
    axis.text.x = element_text(angle = 90, hjust = 1),
    axis.text.y = element_text(size = 12),
    plot.title = element_text(size = 16, face = "bold"),
    legend.position = "right"
  )

print(airbnb_sentiment_trend)
## # A tibble: 43 × 3
##    month_year mean_sentiment company
##    <date>              <dbl> <chr>  
##  1 2012-11-01            5   Airbnb 
##  2 2013-01-01           22   Airbnb 
##  3 2013-05-01          -43   Airbnb 
##  4 2013-06-01         -166   Airbnb 
##  5 2013-10-01          -65   Airbnb 
##  6 2014-04-01          119   Airbnb 
##  7 2014-05-01          -21.5 Airbnb 
##  8 2014-08-01         -147   Airbnb 
##  9 2014-10-01          -46   Airbnb 
## 10 2015-01-01         -133.  Airbnb 
## # ℹ 33 more rows

Total Amount of Articles Collected

The dataset is transformed and summarized using the dplyr package to calculate the articles with a positive (1), neutral (0) and negative (-1) sentiment score for each company. The code then generates a geombar chart using ggplot2 to visualize how many articles were collected for each company.

combined_data <- combined_data %>%
  mutate(publication_date = as.Date(date_article),
         month_year = floor_date(publication_date, "month")) %>%
  group_by(company, month_year) %>%
  summarize(
    monthly_sentiment = mean(total_score, na.rm = TRUE),
    positive_articles = sum(total_score > 0, na.rm = TRUE),
    negative_articles = sum(total_score < 0, na.rm = TRUE),
    total_articles = n(),
    .groups = "drop"
  ) %>%
  mutate(sentiment_class = case_when(
    monthly_sentiment > 0 ~ 1,
    monthly_sentiment == 0 ~ 0,
    monthly_sentiment < 0 ~ -1
  ))

ggplot(combined_data, aes(x = company, y = total_articles, fill = company)) +
  geom_bar(stat = "identity") +
  labs(
    title = "Amount of Articles Collected",
    x = "",
    y = "Total Articles",
    fill= "Company",
    caption = "Source: dataset (perception_media_news)"
  ) +
  scale_fill_manual(values = c("Uber" = "#FF69B4", "Airbnb" = "#DCAE96")) +
  theme_linedraw()+
  theme(
    axis.text.y = element_text(size = 12),                        
    plot.title = element_text(size = 16, face = "bold"),         
    legend.title = element_text(size = 12),                      
    legend.text = element_text(size = 11)                         
  )

Mean Sentiment by Company

Calculate the mean sentiment score for each company over time to see the general sense of whether the overall sentiment for each company is positive or negative.

mean_sentiment_airbnb <- mean(combined_data %>% filter(company == "Airbnb") %>% pull(monthly_sentiment), na.rm = TRUE)
mean_sentiment_uber <- mean(combined_data %>% filter(company == "Uber") %>% pull(monthly_sentiment), na.rm = TRUE)

print(mean_sentiment_airbnb)
## [1] -168.4548
print(mean_sentiment_uber)
## [1] -28.3401

Standard Deviation Sentiment Score

Calculate the standard deviation of sentiment scores for each company over the years to assess how volatile the sentiment is.

sd_sentiment_airbnb <- sd(combined_data %>% filter(company == "Airbnb") %>% pull(monthly_sentiment), na.rm = TRUE)
sd_sentiment_uber <- sd(combined_data %>% filter(company == "Uber") %>% pull(monthly_sentiment), na.rm = TRUE)

print(sd_sentiment_airbnb)
## [1] 668.1916
print(sd_sentiment_uber)
## [1] 272.9562

Sentiment Distribution by Company

Total Amount of Positive and Negative Articles Collected

The code aggregates the total number of positive and negative articles for each company. A new column sentiment is created to indicate whether the count belongs to positive or negative sentiment and also another column count holds the corresponding number of articles for each sentiment. Finally, a geombar chart is created using ggplot2 wherebars are grouped by sentiment and placed side by side. Prints the results in a table format.

combined_data_long <- combined_data %>%
  group_by(company) %>%
  summarize(
    total_positive = sum(positive_articles, na.rm = TRUE),
    total_negative = sum(negative_articles, na.rm = TRUE),
    .groups = "drop"
  ) %>%
  pivot_longer(cols = c(total_positive, total_negative),
               names_to = "sentiment",
               values_to = "count")

ggplot(combined_data_long, aes(x = company, y = count, fill = sentiment)) +
  geom_bar(stat = "identity", position = "dodge") +
  labs(
    title = "Sentiment Distribution by Company",
    x = "",
    y = "Articles Collected",
    fill = "Sentiment",
    caption = "Source: dataset (perception_media_news)"
  ) +
  scale_fill_manual(values = c("total_positive" = "#00F0FF", "total_negative" = "red"),
                    labels = c("Negative", "Positive")) +
  theme_linedraw()+
  theme(
    axis.text.y = element_text(size = 12),                        
    plot.title = element_text(size = 16, face = "bold"),         
    legend.title = element_text(size = 12),                      
    legend.text = element_text(size = 11)                         
  )

print(combined_data_long)
## # A tibble: 4 × 3
##   company sentiment      count
##   <chr>   <chr>          <int>
## 1 Airbnb  total_positive    35
## 2 Airbnb  total_negative    63
## 3 Uber    total_positive    52
## 4 Uber    total_negative    58
table_grob1 <- tableGrob(combined_data_long)


png("combined_data_long.png", width = 800, height = 400, res = 100)
grid.draw(table_grob1)  
dev.off()  
## quartz_off_screen 
##                 2

Total Number of Articles Collected Over Time by Company

The code uses ggplot2 to visualize the number of media articles collected over time for Uber and Airbnb and creates a geombar chart that illustrates how media coverage has fluctuated for both companies, helping to analyze trends in public discourse over different time periods.

ggplot(combined_data, aes(x = month_year, y = total_articles, fill = company)) +
  geom_bar(stat = "identity", position = position_dodge(width = 0.8)) +  
  scale_x_date(
    date_breaks = "1 year",  
    date_labels = "%Y"       
  ) +
  scale_y_continuous(
    breaks = seq(0, max(combined_data$total_articles, na.rm = TRUE), by = 1)  
  ) +
  labs(
    title = "Number of Articles Collected Over Time by Company",
    x = "",
    y = "Articles",
    fill = "Company",
    caption = "Source: dataset (perception_media_news)"
  ) +
  theme_linedraw() +
  theme(
    axis.text.x = element_text(angle = 90, hjust = 1, size = 12),  
    axis.text.y = element_text(size = 12),                        
    plot.title = element_text(size = 16, face = "bold"),          
    legend.title = element_text(size = 12),                       
    legend.text = element_text(size = 11)                         
  ) +
  scale_fill_manual(
    values = c("Airbnb"= "#DCAE96", "Uber" = "#FF69B4")  
  )

Summarize Sentiment

The code aggregates sentiment scores for Uber and Airbnb over time and creates a summary table showing mean sentiment scores and total articles per month. The final output displays the company, month-year, the mean sentiment score (for that month) and the total number of articles collected (for that month).

summary_table_mean_sentiment <- combined_data %>%
  group_by(company, month_year) %>%
  summarize(
    mean_sentiment = mean(monthly_sentiment, na.rm = TRUE),
    total_articles = sum(total_articles)
  )

print(summary_table_mean_sentiment)
## # A tibble: 94 × 4
## # Groups:   company [2]
##    company month_year mean_sentiment total_articles
##    <chr>   <date>              <dbl>          <int>
##  1 Airbnb  2012-11-01            5                1
##  2 Airbnb  2013-01-01           22                1
##  3 Airbnb  2013-05-01          -43                1
##  4 Airbnb  2013-06-01         -166                2
##  5 Airbnb  2013-10-01          -65                5
##  6 Airbnb  2014-04-01          119                1
##  7 Airbnb  2014-05-01          -21.5              2
##  8 Airbnb  2014-08-01         -147                1
##  9 Airbnb  2014-10-01          -46                4
## 10 Airbnb  2015-01-01         -133.               3
## # ℹ 84 more rows

Export the Mean Sentiment Table as Excel

The code exports the summary_table_mean_sentiment data frame to an Excel file.

output_file <- "/Users/mariamrochi/Library/Containers/com.microsoft.Excel/Data/Desktop/Master Thesis/Public Sentiment/summary_table_mean_sentiment.xlsx"  


write_xlsx(summary_table_mean_sentiment, output_file)

2. Public Perception Sentiment and Policy Changes of Uber and Airbnb in New York

The code below analyzes the relationship between public sentiment and policy changes for Uber and Airbnb in New York. The process involves data loading, transformation, merging, and visualization of sentiment scores over time with corresponding policy changes.

Read and Process Policy Data

The policy data for Uber and Airbnb are read from an Excel file named "law_regulations_airbnb_uber.xlsx". The dataset includes the policy change from 2011 until 2024 of Uber and Airbnb in New York. Each company’s data is stored in a separate dataframe.The timeframe column, representing the date of policy events, is converted into a date format, and a month_year column is created by flooring the timeframe to the first day of each month.

policy_file <- "/Users/mariamrochi/Library/Containers/com.microsoft.Excel/Data/Desktop/Master Thesis/Public Sentiment/law_regulations_airbnb_uber.xlsx"

policy_uber <- read_excel(policy_file, sheet = "uber") %>%
  mutate(
    timeframe = as.Date(timeframe, format = "%Y-%m-%d"),
    month_year = floor_date(timeframe, "month")  
  )

policy_airbnb <- read_excel(policy_file, sheet = "airbnb") %>%
  mutate(
    timeframe = as.Date(timeframe, format = "%Y-%m-%d"),
    month_year = floor_date(timeframe, "month")  
  )

Load Sentiment Data

The sentiment data is loaded from another Excel file named "summary_table_mean_sentiment.xlsx", previously generated, which contains sentiment scores split by company, and the month_year column is created for sentiment data to align with the policy data.

sentiment_file <- "/Users/mariamrochi/Library/Containers/com.microsoft.Excel/Data/Desktop/Master Thesis/Public Sentiment/summary_table_mean_sentiment.xlsx"
mean_sentiment <- read_excel(sentiment_file) %>%
  mutate(month_year = as.Date(month_year))  

uber_sentiment <- mean_sentiment %>%
  filter(company == "Uber") %>%
  mutate(month_year = floor_date(month_year, "month"))  

airbnb_sentiment <- mean_sentiment %>%
  filter(company == "Airbnb") %>%
  mutate(month_year = floor_date(month_year, "month"))

Merge Sentiment Data with Policy Data

The sentiment data is merged with the respective policy data based on the month_year column for both Uber and Airbnb using the left_join function.The merged datasets for both companies are then combined into a single policy_data dataset, with an additional column to indicate the company (Uber or Airbnb). The merged policy_data dataset is written to an Excel file named "policy_sentiment.xlsx" for further analysis.

uber_combined <- policy_uber %>% left_join(uber_sentiment, by = "month_year")

airbnb_combined <- policy_airbnb %>% left_join(airbnb_sentiment, by = "month_year")

policy_data <- bind_rows( uber_combined %>% mutate(company = "Uber"), airbnb_combined %>% mutate(company = "Airbnb") )

print(policy_data)
## # A tibble: 31 × 11
##    timeframe  policy_name     details lobbying_strategies outcome results source
##    <date>     <chr>           <chr>   <chr>               <chr>   <chr>   <chr> 
##  1 2012-09-05 "Ban of Uber a… "The T… "Uber's CEO argued… The ba… Positi… "*htt…
##  2 2012-12-13 "Pilot Program… "The T… "Uber among other … The pi… Positi… "*htt…
##  3 2013-12-16 "Surge pricing" "Uber'… "The company defen… Public… Positi… "*htt…
##  4 2015-07-16 "Proposed Cap … "Mayor… "Uber framed the p… The ca… Positi… "*htt…
##  5 2018-08-08 "12-Month Cap … "The C… "Funded million-do… The ca… Negati… "*htt…
##  6 2018-08-08 "Minimum Wage"  "The C… "Urged customers t… The mi… Negati… "*htt…
##  7 2018-12-05 "Minimum Pay R… "The c… "Uber resisted the… The mi… Negati… "*htt…
##  8 2019-02-15 "Lawsuit again… "Uber … "Uber argued that … The la… Negati… "*htt…
##  9 2019-08-07 "Extension of … "The T… "Uber sued the cit… The Ma… Negati… "*htt…
## 10 2022-03-24 "Agreement to … "Uber … "Uber framed it as… The ag… Positi… "*htt…
## # ℹ 21 more rows
## # ℹ 4 more variables: month_year <date>, company <chr>, mean_sentiment <dbl>,
## #   total_articles <dbl>
write_xlsx(policy_data, "/Users/mariamrochi/Library/Containers/com.microsoft.Excel/Data/Desktop/Master Thesis/Public Sentiment/policy_sentiment.xlsx")

The code creates also a new column in the dataset called lobby_results andassigns 1 for positive outcomes, -1 for negative, and 0 for neutral/missing values.

policy_data$month_year <- as.Date(policy_data$month_year, format = "%Y-%m-%d")

policy_data$results <- tolower(trimws(policy_data$results))

policy_data$lobby_results <- ifelse(policy_data$results == "positive", 1, 
                             ifelse(policy_data$results == "negative", -1, 0))

policy_data$lobby_results <- as.factor(policy_data$lobby_results)

policy_data <- na.omit(policy_data)

print(policy_data)
## # A tibble: 26 × 12
##    timeframe  policy_name     details lobbying_strategies outcome results source
##    <date>     <chr>           <chr>   <chr>               <chr>   <chr>   <chr> 
##  1 2012-09-05 "Ban of Uber a… "The T… "Uber's CEO argued… The ba… positi… "*htt…
##  2 2013-12-16 "Surge pricing" "Uber'… "The company defen… Public… positi… "*htt…
##  3 2015-07-16 "Proposed Cap … "Mayor… "Uber framed the p… The ca… positi… "*htt…
##  4 2018-08-08 "12-Month Cap … "The C… "Funded million-do… The ca… negati… "*htt…
##  5 2018-08-08 "Minimum Wage"  "The C… "Urged customers t… The mi… negati… "*htt…
##  6 2018-12-05 "Minimum Pay R… "The c… "Uber resisted the… The mi… negati… "*htt…
##  7 2019-02-15 "Lawsuit again… "Uber … "Uber argued that … The la… negati… "*htt…
##  8 2019-08-07 "Extension of … "The T… "Uber sued the cit… The Ma… negati… "*htt…
##  9 2022-03-24 "Agreement to … "Uber … "Uber framed it as… The ag… positi… "*htt…
## 10 2022-12-19 "Planned Pay I… "The T… "Uber filed a laws… The pl… negati… "*htt…
## # ℹ 16 more rows
## # ℹ 5 more variables: month_year <date>, company <chr>, mean_sentiment <dbl>,
## #   total_articles <dbl>, lobby_results <fct>

Plot Sentiment and Policy Changes for Uber

The sentiment scores over time are visualized using ggplot2. A geomline graph is created to plot the mean sentiment for Uber, with vertical dashed lines marking policy changes.

policy_uber <- policy_data %>%
  filter(company == "Uber") %>%
  mutate(
    results = trimws(results),  
    policy_name_color = case_when(
      results == "positive" ~ "#00F0FF",  
      results == "negative" ~ "red",      
      TRUE ~ "black"                      
    )
  )

ggplot(policy_uber, aes(x = month_year, y = mean_sentiment, color = company)) +
  geom_line(size = 1) +  
  geom_point(color = "#DCAE96", size = 1.7) +   
  
  geom_vline(
    aes(xintercept = as.numeric(timeframe)), 
    linetype = "dashed",
    color = "red"
  ) +  

  geom_text(
    aes(x = timeframe, 
        y = min(mean_sentiment, na.rm = TRUE),
        label = policy_name, 
        color = policy_name_color),  
    angle = 90, 
    vjust = 1.5, 
    hjust = 0, 
    size = 2
  ) +  

  scale_x_date(
    breaks = unique(policy_uber$timeframe), 
    date_labels = "%b %Y"
  ) +
  labs(
    title = "Uber Mean Sentiment Over Time with Policy Changes",
    x = "",
    y = "Mean Sentiment",
    caption = "Dashed lines indicate Uber policy changes. Source: dataset (policy_sentiment)"
  ) +
  scale_color_manual(values = c("Uber" = "#FF69B4","Airbnb" = "#DCAE96"),
                     name = "Company") +
  theme_linedraw() +
  theme(
    axis.text.x = element_text(angle = 45, hjust = 1, size = 6),
    plot.title = element_text(size = 16, face = "bold"),
    axis.title = element_text(size = 10),
    legend.position = "right"
  )

Plot Sentiment and Policy Changes for Airbnb

The sentiment scores over time are visualized using ggplot2. A geomline graph is created to plot the mean sentiment for Airbnb, with vertical dashed lines marking policy changes.

policy_airbnb <- policy_data %>%
  filter(company == "Airbnb") %>%
  mutate(
    results = trimws(results),  
    policy_name_color = case_when(
      results == "positive" ~ "#00F0FF",  
      results == "negative" ~ "red",      
      TRUE ~ "black"                      
    )
  )


ggplot(policy_airbnb, aes(x = month_year, y = mean_sentiment, color = company)) +
  geom_line(size = 1) +  
  geom_point(color = "#FF69B4", size = 1.7) +   
  geom_vline(
    aes(xintercept = as.numeric(timeframe)), 
    linetype = "dashed", 
    color = "red"
  ) +  
  geom_text(
    aes(x = timeframe, 
        y = min(mean_sentiment, na.rm = TRUE), 
        label = policy_name, 
        color = "black"),
    angle = 90, 
    vjust = 1.5, 
    hjust = 0, 
    size = 2,
  ) +  
  scale_x_date(
    breaks = unique(policy_airbnb$timeframe), 
    date_labels = "%b %Y"
  ) +
  labs(
    title = "Airbnb Mean Sentiment Over Time with Policy Changes",
    x = "",
    y = "Mean Sentiment",
    caption = "Dashed lines indicate Airbnb policy changes. Source: dataset (policy_sentiment)"
  ) +
    scale_color_manual(values = c("Uber" = "#FF69B4","Airbnb" = "#DCAE96"),
                     name = "Company") +
  theme_linedraw() +
  theme(
    axis.text.x = element_text(angle = 45, hjust = 1, size = 6),
    plot.title = element_text(size = 16, face = "bold"),
    axis.title = element_text(size = 10),
    legend.position = "right"
  )

Faceted Plot Sentiment Trend and Policy Changes for Uber and Airbnb

The plot is faceted by company, showing the sentiment scores plotted over time and policy changes marked for each company, side by side for easy comparison.

policy_data <- policy_data %>%
  arrange(company, month_year)  

ggplot(policy_data, aes(x = month_year, y = mean_sentiment, color = company)) +
  geom_line(aes(color = company), size = 0.5) +  
  geom_point(aes(color = company), size = 1) +    
  geom_vline(
    aes(xintercept = as.numeric(timeframe)),
    linetype = "dashed",
    color = "red"
  ) +  
  geom_text(
    aes(x = timeframe, y = min(mean_sentiment, na.rm = TRUE), label = policy_name),
    angle = 90, 
    vjust = 1.5, 
    hjust = 0, 
    size = 1.5,
    color = "black"
  ) +  
  scale_x_date(
    breaks = unique(policy_data$month_year), 
    date_labels = "%b %Y"
  ) +
  facet_wrap(~company, scales = "free_x") +  
  scale_color_manual(values = c("Airbnb" = "#DCAE96", "Uber" = "#FF69B4"),
                     name = "Company") +
  labs(
    title = "Sentiment Over Time with Policy Changes",
    x = "",
    y = "Mean Sentiment",
    caption = "Dashed lines indicate policy changes. Source: dataset (policy_sentiment)"
  ) +
  theme_linedraw() +
  theme(
    axis.text.x = element_text(angle = 45, hjust = 1, size = 6),
    plot.title = element_text(size = 16, face = "bold"),
    axis.title = element_text(size = 10),
    legend.position = "right"
  )

Insights and Findings

The 3 plots shows that public sentiment towards both companies fluctuates over time, rather than remaining constant. The graphs are useful as it shows potential trends in sentiment and potential relationships between sentiment and policy changes. By visually looking at the plots, it can also be seen that the negative policy changes appear to have had a more pronounced effect on the sentiment for Airbnb than for Uber, with deeper drops in sentiment for Airbnb after policy changes. The limitations of the plots is that it cannot prove causation between policy changes and sentiment shifts and for these reasons the analysis will proceed further with a correlation analysis test.

3. Correlation Analysis Between Sentiment and Policy Outcomes

Data Preparation and Processing

The code run a correlation test between sentiment and policy outcome by creating a new column in the dataset called lobby_results.Given that policy outcomes were categorized as “Positive,” “Negative,” or “Neutral,” it was necessary to clean the results column by removing extra spaces and converting all text to lowercase to maintain consistency. Also, the code assigns 1 for positive outcomes, -1 for negative, and 0 for neutral/missing values.

policy_data$month_year <- as.Date(policy_data$month_year, format = "%Y-%m-%d")

policy_data$results <- tolower(trimws(policy_data$results))

policy_data$lobby_results <- ifelse(policy_data$results == "positive", 1, 
                             ifelse(policy_data$results == "negative", -1, 0))

policy_data$lobby_results <- as.numeric(policy_data$lobby_results)

table(policy_data$lobby_results)
## 
## -1  0  1 
## 15  1 10
print(policy_data)
## # A tibble: 26 × 12
##    timeframe  policy_name     details lobbying_strategies outcome results source
##    <date>     <chr>           <chr>   <chr>               <chr>   <chr>   <chr> 
##  1 2014-05-22 "Agreement to … Airbnb… "Framed the agreem… The de… positi… "*htt…
##  2 2014-10-16 "New York Stat… Report… "Described the fin… The re… negati… "*htt…
##  3 2015-01-21 "City Council … A hear… "Sent a letter to … Increa… negati… "*htt…
##  4 2016-06-22 "State Bill Ba… The Ne… "Claimed the bill … The la… negati… "*htt…
##  5 2016-12-06 "Settlement of… Airbnb… "Secured assurance… The se… positi… "*htt…
##  6 2017-08-07 "Hotel Industr… The Ho… "Called the ad cam… The ad… negati… "*htt…
##  7 2018-08-26 "New Legislati… New le… "Contested the leg… A judg… positi… "*htt…
##  8 2019-01-03 "Judge Blocks … Judge … "Celebrated the de… The ju… positi… "*htt…
##  9 2019-02-19 "Subpoena for … New Yo… "Claimed the subpo… The ci… negati… "*htt…
## 10 2019-05-17 "Manhattan Sup… Manhat… "Cited concerns ov… The co… negati… "*htt…
## # ℹ 16 more rows
## # ℹ 5 more variables: month_year <date>, company <chr>, mean_sentiment <dbl>,
## #   total_articles <dbl>, lobby_results <dbl>

Correlation Test

The code convert the data in the lobby_results column to numeric types and handles the missing data. Then, the code performs a correlation test (cor.test) between the mean_sentiment and lobby_results columns to measure the strength and direction of the relationship between these two variables.

policy_data$lobby_results <- as.numeric(policy_data$lobby_results)

policy_data <- na.omit(policy_data)

policy_data$mean_sentiment <- as.numeric(policy_data$mean_sentiment)

correlation_result <- cor.test(policy_data$mean_sentiment, policy_data$lobby_results)

print(correlation_result)
## 
##  Pearson's product-moment correlation
## 
## data:  policy_data$mean_sentiment and policy_data$lobby_results
## t = 0.70123, df = 24, p-value = 0.4899
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
##  -0.2599249  0.5015190
## sample estimates:
##      cor 
## 0.141693
correlation_result <- cor.test(policy_data$mean_sentiment, policy_data$lobby_results)


correlation_table <- data.frame(
  Statistic = c("Correlation Coefficient", "P-Value", "T-Statistic", "Degrees of Freedom"),
  Value = c(correlation_result$estimate, correlation_result$p.value, correlation_result$statistic, correlation_result$parameter)
)


table_grob3 <- tableGrob(correlation_table)


png("correlation_result.png", width = 800, height = 800, res = 200)
grid.draw(table_grob3)  
dev.off()  
## quartz_off_screen 
##                 2

Linear Regression Model

The code fits a linear regression model where lobby_results is the dependent (response) variable and mean_sentiment is the independent (predictor) variable to attempt to predict the relationship between the two variables. The goal of this model is to examine whether public sentiment influences lobbying outcomes.

policy_data$lobby_results <- as.numeric(policy_data$lobby_results)

model <- lm(lobby_results ~ mean_sentiment, data = policy_data)

summary(model)
## 
## Call:
## lm(formula = lobby_results ~ mean_sentiment, data = policy_data)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -0.8753 -0.8519 -0.7391  1.1473  1.3153 
## 
## Coefficients:
##                  Estimate Std. Error t value Pr(>|t|)
## (Intercept)    -0.1439876  0.2061506  -0.698    0.492
## mean_sentiment  0.0001872  0.0002669   0.701    0.490
## 
## Residual standard error: 0.9907 on 24 degrees of freedom
## Multiple R-squared:  0.02008,    Adjusted R-squared:  -0.02075 
## F-statistic: 0.4917 on 1 and 24 DF,  p-value: 0.4899
model_summary <- summary(model)
regression_table <- data.frame(
  Statistic = c("Intercept", "Slope (Mean Sentiment)", "R-Squared", "Adjusted R-Squared", "P-Value"),
  Value = c(
    coef(model_summary)[1,1],  
    coef(model_summary)[2,1], 
    model_summary$r.squared,   
    model_summary$adj.r.squared,  
    coef(model_summary)[2,4]   
  )
)

table_grob4 <- tableGrob(regression_table)


png("regression_table.png", width = 800, height = 800, res = 200)
grid.draw(table_grob4)  
dev.off()  
## quartz_off_screen 
##                 2

4. Conclusion

The weak relationship between sentiment and lobbying outcomes of Uber and Airbnb in New York supports the conclusion that public perception does not have a strong or direct correlation with the effectiveness of lobbying efforts. To test the hypothesis that “the effectiveness of platform firms in lobbying for better regulations is positively correlated with public perception,” the analysis used a combination of quantitative methods, including sentiment analysis, correlation tests, and linear regression. The statistical tests concluded that there is no strong or statistically significant correlation between public sentiment and lobbying outcomes and do not support the hypothesis that the effectiveness of lobbying efforts is positively correlated with public perception, as previously assumed from the research. The weak and statistically insignificant correlation shows that the public perception is not a good predictor of lobbying success. Therefore, it is likely that other factors, play a more significant role in shaping policy outcomes and this calls for further exploration of other potential factors influencing the success of lobbying.

#knitr::purl("SentimentAnalysisRochi.Rmd", output = "SentimentAnalysisRochi.R")
#knitr::purl("SentimentAnalysisRochi.Rmd", output = "SentimentAnalysisRochi.txt")