19 Oct 2017

Overview

Social Media Data

  • How to Collect
  • Querying Social Media
  • Biases & Ethics

Hands on: Twitter API

  • RESTful: Current Tweets, Profile, Friends/Followers
  • Streaming: Future Tweets
  • "Big Data" Streaming

Hands on: Facebook API

  • Searching public pages
  • Scraping public pages
  • Public page comments & likes

Collecting Social Media Data

Two Different Methods

  • Web scraping: extract data from tagged source code of website
  • Web APIs (application programming interface): use structured https requests that return JSON or XML files

Two Types of APIs

  • RESTful API's: queries for static data at current moment
  • Streaming API's: changes in users' data in real time (future)

Limitations

  • API's limited to public data, subject to rate and historical limits
  • Identify the best platform, not necessarily the easiest.

Potential Issues

While there are opportunities in social media data, there are difficulties:

Difficult Learning Curve

Bots, ethics, and experimentation

Discussion

Note about third party research

Twitter Blog

Studies of the impact of bots and automation on Twitter necessarily and systematically under-represent our enforcement actions because these defensive actions are not visible via our API, and because they take place shortly after content is created and delivered via our streaming API. Furthermore, researchers using an API often overlook the substantial in-product features that prioritize the most relevant content. Based on user interests and choices, we limit the visibility of low-quality content using tools such as Quality Filter and Safe Search – both of which are on by default for all of Twitter’s users and active for more than 97% of users.

Source: Twitter Blog, Sept 28, 2017

“Research conducted by third parties through our search API about the impact of bots and misinformation on Twitter is almost always inaccurate and methodologically flawed,” Twitter said.

Source: Washington Post, Sept 28, 2017

rtweet: REST and Streaming Twitter with R

Get/Call Package

lines 4-7

# install from CRAN (uncomment out)
# install.packages("rtweet")

# load rtweet & tidyverse
library(rtweet); library(tidyverse)

Create a Twitter API App

Create Twitter app

  • Go to apps.twitter.com and create a new app by providing a Name (rtweet_token), Description, and Website of your choosing.

Callback URL

Accept Terms

  • Check yes if you agree and then click “Create your Twitter application”.

Run Examples

  • /01-rtweet/01-rtweet-restapi.R

  • /01-rtweet/02-rtweet-streaming.R

  • /01-rtweet/03-rtweet-streaming-large.R

  • /02-Rfacebook/facebook-api.Rmd