Character Count Exploration

Introduction

The goal here is to query as much of the contentapi as possible to get the text of articles, then get a count of the number of each type of character used.

First step here is to get all of the article URLs from the past two years. This document is on the internet so the SQL authentication info is redacted.

library(RPostgreSQL)

drv <- dbDriver("PostgreSQL")
con <- dbConnect(drv, 
                 dbname = "redacted",
                 host = "redacted",
                 port = 'redacted',
                 user = "redacted",
                 password = "redacted")

articles <- dbGetQuery(con, 'select url, pub_date from slateplus.articles_article where pub_date > \'2015-10-01\';')
head(articles)

Next we loop through each article and download a JSON file. Then we extract the text from the JSON file within another loop. The last step creates a dataframe with all of the article text and publication dates.

library(RCurl)
library(rjson)
library(dplyr)
article_text <- c()
article_jsons <- list()
for(u in seq(1,length(articles$url))){
  art <- getURL(url = paste0('https://my.slate.com/api/articles/url?url=', articles$url[u])) 
  article_jsons[[u]] <- art
  art <- art%>%fromJSON()
  
  text_list <- c()
  for(t in seq(1:length(art$body))){
    ifelse(nchar(art$body[[t]]$text) > 5, 
           text_list[t] <- art$body[[t]]$text,
           next)
  }
  article_text[u] <- paste(text_list, collapse = ' ')
}

articles_text <- data.frame(
  date = articles$pub_date,
  text = article_text
)

Split out each word onto its own line of a dataframe (that’s what unnest_tokens() does!). This is where the computationally expensive stuff starts!

library(tidytext)
article_words <- articles_text %>%
  mutate(text = as.character(text))%>%
  unnest_tokens(word, text)

Repeat this process but split out characters within words. I did it this way so we could identify weird characters and map them to the actual words they’re in. This is the last step which did not work.

chars <- article_words%>%
  mutate(word = as.character(word))%>%
  unnest_tokens(char, word, token = 'characters', drop = FALSE)%>%
  # filter(!char %in% c(letters, as.character(seq(0,9))))%>%
  group_by(char, date)%>%
  summarise(n = n())%>%
  distinct()