Enviromental Twitter Analysis

By Helge Peters (Oxford) and Nathanael Sheehan (UCL)

Part 1: Different actors problematise water in different ways

Social studies of environmental controversies suggest that different actors problematise ‘water’ in distinct ways, which likely reflect their organisational purpose, priorities, constraints, accountabilities, audiences etc Such distinct problematisations are liable to contribute to water management as a ‘wicked problem’ in which multiple ontologies of water are at stake. We propose to use text clustering in order to identify these distinct ontologies/problematisations of water and subsequently measure the extent to which these differ between actor categories.

Research question 1) How can natural language processing from tweets identify latent topics regarding water?

Research question 2) Can we measure the proximity/distance between these topics and map these onto actor categories?

Data collection

In order to understand how different actors speak about water on twitter, we needed to mine twitter data from the relevent actors. A list of 170 actors was produced and were labeled under six categories Activist, Charity, Industry, Government, Journalist and Partnership’s dependent on their affiliations.

To perform the analysis, a number of R libraries were used. In order to access and mine twitter data, we used the rtweet package. Collected data was cleaned using tm, stringr and qdapRegex. For data merging, formatting and munging we used the purrr, dplyr and readr packages.

library(httpuv)
library(rtweet)       
library(tm)            
library(stringr)       
library(qdapRegex)    
library(wordcloud2)    
library(purrr)
library(dplyr)
library(plyr)
library(readr)

Authetication to the twitter API was generated using our API keys and tokens. The rtweet package makes this easy through the create_token and get_token functions.

##Set up twitter auth - these are not our actual keys :p
api_key <- "hjksfdhkjeh342324"
api_secret_key <- "jh342kjhkjfhdkjh2jk3h4kjbfdskjhkjh"
access_token <- "123u098f90ds8fshjk234lnhkljfd"
access_secret <- "fdsjkhakjfh324fkjhkj3riuhi9"


token <- create_token(
  app = "Camellia Research",
  consumer_key = api_key,
  consumer_secret = api_secret_key,
  access_token = access_token,
  access_secret = access_secret)

get_token()

Once connection was secure, a function was writted to mine twitter data. The function mineTweets takes the string username as a variable to mine the relevent account. A maxiumum of 1000 tweets were mined per account (not including retweets) and a retryonratelimit was set to true in order to ensure we sucessfully mined everyones tweets. Mined tweets were subsetted to only include the twitter and the tweet. Each accounts tweets were the exported as a CSV.

mineTweets <- function(username){
  #Mine tweets
  tweets = get_timelines(username, n = 1000, retryonratelimit = TRUE, include_rts = FALSE)
  
  #Subset tweets
  tweets = subset(tweets,
         select = c(
           `screen_name`,
           `text`)
  )
  
  #Export dataframe
  write.csv(tweets, paste(username,"minedtweets.csv"))
}

A queue was then set up per category to mine each actors profile.

#Mining activist tweets
mineTweetsActivist {
  mineTweets(username = "rivercide_live")
  mineTweets(username = "cleansafethames")
  mineTweets(username = "GeorgeMonbiot")
  mineTweets(username = "Feargal_Sharkey")
  mineTweets(username = "AlisonJArcher")
  mineTweets(username = "deedeelea")
  mineTweets(username = "iantokelove")
  mineTweets(username = "FisherLady21")
  mineTweets(username = "sascampaigns")
  mineTweets(username = "LDNWaterkeeper")
  mineTweets(username = "PymmesBrookERS")
  mineTweets(username = "QWAG")
  mineTweets(username = "leaboaters")
  mineTweets(username = "StonebridgeLock")
  mineTweets(username = "SaveLeaMarshes")
  mineTweets(username = "silkstreamers")
  mineTweets(username = "LDNgreenspaces")
  mineTweets(username = "StopTheDrill")
  
}

Once all accounts had been mined, a second queue was established in order to merge all CSV files together.

mergeTweetsActivist {
  setwd("~/Twitter-Mining/mined-tweets")
  mydir = "activist"
  myfiles = list.files(path=mydir, pattern="*.csv", full.names=TRUE)
  myfiles
  activist = ldply(myfiles, read_csv)
  
  #Export dataframe
  write.csv(activist,"merged_activist_tweets.csv")
}

This process was then repeated for each category; generating 6 CSV files containing all over 70,00 tweets from our different categories.

Analysis