Web scraping - round 1

In the web scraping portion of the project, I collected, among other things, the URL, title, author name, publish date, tags, number of comments and number of “claps” of all articles with the tag “Data Science” published on Medium (earliest of which was published in 2009, but really picked up in volume in 2013).

Since I was completely new to web scraping before this, I looked around for existing and fairly recent scripts scraping Medium articles. I had found two that use Selenium (here and here), but because of the large size of data to be scraped, I wanted to use Scrapy over Selenium (see a comparison of the two here). Plus, Scrapy has a built-in selector system that means I don’t have to use BeautifulSoup to parse the HTML. I ended up using the Scrapy workflow by May Yeung (posted on Medium, of course) as a starting point and made the script below, after much trial and error.

Looking at the archive of all articles tagged with “Data Science”, I see that I can iterate over each year (2009-2019 November), each month (01-12) and each day (01-31) to see the story cards of all articles tagged with “Data Science”. Even though each story card contains the title, author name, publication date, number of comments and claps, I needed to get to the actual article page to get the tags. So, the script below follows the article URL on each story card to access the article page and scrape all the desired elements. This was the major departure from May Yeung’s workflow, which scraped only the story cards.

As there are often ~20,000 articles/year published in more recent years, I decided to divide up the scraping by year. As an example, here is the script used to get all the articles tagged with “Data Science” published in 2018:

## Import libraries
import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.utils.log import configure_logging
import logging
 
## Set working directory
import os
os.chdir('/Users/nancy/PycharmProjects/medium-ds-articles/data/raw/')
 
## Create container for scraped data
class Article(scrapy.Item):
    nameOfAuthor = scrapy.Field()
    linkOfAuthorProfile = scrapy.Field()
    NumOfComments = scrapy.Field()
    article = scrapy.Field()
    postingTime = scrapy.Field()
    NumOfClaps = scrapy.Field()
    articleURL = scrapy.Field()
    articleTags = scrapy.Field()
    readingTime = scrapy.Field()
 
## Set-up logging
logger = logging.getLogger('scrapylogger')
 
## Create crawler
class MediumSpider(scrapy.Spider):
    name = "medium_spider"
 
    configure_logging(install_root_handler=False)
    logging.basicConfig(
        filename='medium_full_2018_log.txt',
        format='%(levelname)s: %(message)s',
        level=logging.INFO
    )
 
    custom_settings = {
        'FEED_FORMAT': 'csv',
        'FEED_URI': 'medium_full_2018.csv',
        'AUTOTHROTTLE_ENABLED' : True,
        'AUTOTHROTTLE_START_DELAY' : 1,
        'AUTOTHROTTLE_MAX_DELAY' : 3
    }
 
    def start_requests(self):
        urls = []
 
        for month in range(1, 13):
            for day in range(1, 32):
                urls.append(f"https://medium.com/tag/data-science/archive/2018/{month:02}/{day:02}")
 
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)
 
 
 
    def parse(self, response):
        item = Article()
 
        for story in response.css('div.postArticle'):
            if story.css('div.postArticle-readMore a::attr(href)').extract_first() is not None:
                url = story.css('div.postArticle-readMore a::attr(href)').extract_first()
                yield scrapy.Request(url=url, callback=self.parse_full, meta={'item': item})
 
    def parse_full(self, response):
 
        item = response.meta['item']
        item['articleURL'] = response.request.url
        item['article'] = response.css('div.postArticle-content section div.section-content div h1::text, \
                                        div.postArticle-content section div.section-content div h1 a::text, \
                                        div.postArticle-content section div.section-content div h1 strong::text,\
                                        div.postArticle-content section div.section-content div h1 em::text, \
                                        div.postArticle-content section div.section-content div h3::text, \
                                        div.postArticle-content section div.section-content div h4::text, \
                                        div.postArticle-content section div.section-content div p strong::text, \
                                        div.postArticle-content section div.section-content div p strong em::text, \
                                        div.postArticle-content section div.section-content div p::text').extract_first()

        try:
            item['linkOfAuthorProfile'] = response.css('div.u-paddingBottom3 a').attrib['href']
        except KeyError:
            item['linkOfAuthorProfile'] = ' '
 
        try:
            item['readingTime'] = response.css('span.readingTime').attrib['title']
        except KeyError:
            item['readingTime'] = ' '
 
 
        item['nameOfAuthor'] = response.css('div.u-paddingBottom3 a::text').extract_first()
        item['postingTime'] = response.css('time::text').extract_first()
        item['articleTags'] = response.css('div.u-paddingBottom10 ul.tags--postTags li a::text').getall()
        item['NumOfComments'] = response.css(
            'div.buttonSet.u-flex0 button.button.button--chromeless.u-baseColor--buttonNormal.u-marginRight12::text').extract_first()
        item['NumOfClaps'] = response.xpath(
            '//div/main/article/footer/div[1]/div[3]/div/div[1]/div/span/button//text()').extract_first()
 
 
        yield item
 
 
process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})
 
process.crawl(MediumSpider)
process.start()

Web scraping - round 2

Just to see if for whatever reason I am missing some articles, I wrote a second crawler where I just scraped the article link from each story card. I do realize that there must be much simpler and more elegant ways of doing everything. However, this was my first attempt at web scraping, so I settled for the quickest way to get some data that I can analyze.

Here is the script for the simplified crawler:

import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.utils.log import configure_logging
import logging

class Article(scrapy.Item):
    article = scrapy.Field()
    articleURL = scrapy.Field()

logger = logging.getLogger('scrapylogger')

class MediumSpider(scrapy.Spider):
    name = "medium_spider" # Name of the scraper

    configure_logging(install_root_handler=False)
    logging.basicConfig(
        filename='./data/raw/medium_titles_2018_log.txt',
        format='%(levelname)s: %(message)s',
        level=logging.INFO
    )

    custom_settings = {
        'FEED_FORMAT': 'csv',
        'FEED_URI': './data/raw/medium_titles_2018.csv',
        'AUTOTHROTTLE_ENABLED': True,
        'AUTOTHROTTLE_START_DELAY': 1,
        'AUTOTHROTTLE_MAX_DELAY': 3
    }

    def start_requests(self):
        urls = []

        for month in range(1, 13):
            for day in range(1, 32):
                urls.append(f"https://medium.com/tag/data-science/archive/2018/{month:02}/{day:02}")

        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)



    def parse(self, response):
        for story in response.css('div.postArticle'):
            yield {
                'article': story.css(
                    'div.postArticle-content section div.section-content div h3::text').getall(),
                'articleURL': story.css('div.postArticle-readMore a::attr(href)').extract_first(),
            }


process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})

process.crawl(IntroSpider)
process.start()

Combine scraping results

Now that this is done, let’s peek at the two data sets for 2018:

## Import libraries
import pandas as pd

## Import data sets
full = pd.read_csv("https://github.com/nchelaru/medium_scrapy/raw/master/raw/medium_full_2018.csv")
titles = pd.read_csv("https://github.com/nchelaru/medium_scrapy/raw/master/raw/medium_titles_2018.csv")

full.head(3)

titles.head(3)

Theoretically, if my scraping is perfect in both cases, I would get the same number of articles/links from the two crawlers. However, a quick comparison of the two data sets, namely the “full-featured” article data that I got using the crawler in Part 1 and the more simplified “title-only” data gotten using the crawler above, showed that there are differences:

## Check dimensions of the two datasets
print("Full:", full.shape)

Full: (19698, 9)

print("Titles:", titles.shape)

Titles: (20127, 2)

Looking over the data sets, I see that there are duplicate links that only differ in the last part of the URLs (namely, ?source=tag_archive---------0---------------------). As the links are still valid after removing everything after ?, I did just that and deduplicated rows with idential article links.

## Clean full info dataset
full['articleLink'] = full['articleURL'].str.split('?').str[0]

full.drop('articleURL', axis=1, inplace=True)

full.drop_duplicates(subset=['articleLink'], keep='first', inplace=True)

full['articleLink'].dropna(inplace=True)

full.shape

(19653, 9)

## Clean title info dataset
titles['articleLink'] = titles['articleURL'].str.split('?').str[0]

titles.drop('articleURL', axis=1, inplace=True)

titles.drop_duplicates(subset=['articleLink'], keep='first', inplace=True)

titles['articleLink'].dropna(inplace=True)

titles.shape

(19678, 2)

Now the number of articles/links in the two datasets are much closer. Again, as this is a quick proof-of-concept and hobbey project, I’m not as concerned with getting all the data that I can possibly can. So, I will combine these two datasets to get all the unique article titles and metadata that I can, to get the final datasets that I will use for analysis. This is done in the same manner for articles tagged with “Data science” published in all years between 2009 and November 2019.

Just out of interest, let’s see how many data science articles are published each year since 2009:

## Import library
library(ggplot2)

## Count number of articles published each year
year_list <- c("2009-2011", "2012", "2013", "2014", "2015", "2016", "2017", "2018", 
    "2019")

freq_list <- list()

i <- 1

for (year in year_list) {
    df <- read.csv(sprintf("https://github.com/nchelaru/medium_scrapy/raw/master/processed/y%s_clean_titles.csv", 
        year))
    
    freq_list[[year]] <- dim(df)[1]
}

df <- do.call(rbind, Map(data.frame, Year = year_list, num_articles = freq_list))

rownames(df) <- c()

## Plot
ggplot(df, aes(x = Year, y = num_articles, fill = num_articles, label = num_articles)) + 
    geom_bar(stat = "identity") + geom_text(size = 4, position = position_stack(vjust = 1.1)) + 
    scale_fill_gradient2(low = "red", high = "green") + labs(y = "Number of articles published", 
    size = 5) + theme_classic() + theme(legend.position = "none")

Wow!

Get title bigram counts

First, I will use the Python natural language processing packages NLTK and spaCy to tokenize each (parsed) article title, filter out non-English words, singularize nouns, generate bigrams and finally create a tally of the frequency of each bigram. Unlike many other examples I have seen, I opted to not remove stop words, as otherwise I would lose bigrams like ‘how’, ‘to’ and ‘need’, ‘to’ that are so prevalent in Medium articles and hallmarks for clickbait.

## Import libraries
import collections
import nltk
import inflection as inf
import spacy
from spacy_langdetect import LanguageDetector

## Get bigrams
nlp = spacy.load('en_core_web_sm')

nlp.add_pipe(LanguageDetector(), name="language_detector", last=True)

counts = collections.Counter()

for sent in final_df["names"]:
  if type(sent) == str:
    doc = nlp(sent)
    
    word_list = []
    
    for token in doc:
      word_list.append(token.text.lower())
      
    counts.update(nltk.bigrams(word_list))
    
## Get dataframe of bigram counts
bigram_counts = pd.DataFrame.from_dict(counts, orient='index').reset_index()

bigram_counts.columns = ["Bigrams", 'Count']

bigram_counts[['Term1', 'Term2']] = pd.DataFrame(bigram_counts['Bigrams'].tolist(), index=bigram_counts.index)  

bigram_counts = bigram_counts[['Term1', 'Term2', 'Count']]

bigram_counts.columns = ['word1', 'word2', 'n']

Unsurprisingly, “data science”, “machine learning” and “how to” are the most frequently appearing bigrams in article titles. Looks like we are on the right track.

Visualize bigram relationships

Finally, I will use the R packages igraph and ggraph to visualize the relationship between the top 60 (otherwise the figure is too crowded) most frequently appearing bigrams in titles of Medium articles tagged with “Data science” published in 2018.

Each word appears as a node and the directionality of the arrow connecting them to each other indicates the order in which they appears in a bigram. Finally, the darkness of the arrow connecting each pair of words is proportional to the frequency of appearance for that bigram. For example, we see much darker arrows connecting ‘data’, ‘science’ or ‘machine’, ‘learning’.

## Import libraries
library(igraph)
library(ggraph)

## Set seed for ggraph package
set.seed(2016)

## Get top 60 most frequent bigrams
bigram_counts <- head(bigram_counts[order(-bigram_counts$n), ], 60) %>% select(-X)

## Create graph
bigram_graph <- bigram_counts %>% graph_from_data_frame()

a <- grid::arrow(type = "closed")

ggraph(bigram_graph, layout = "fr") + geom_edge_link(aes(edge_alpha = n), show.legend = FALSE, 
    arrow = a, end_cap = circle(0.07, "inches")) + geom_node_point(color = "lightblue", 
    size = 5) + geom_node_text(aes(label = name), vjust = 1, hjust = 1, repel = TRUE) + 
    theme_void()

Yep, we are seeing the “need to”, “how to”, “the best” that just sends a stab of FOMO through your heart.

Anyways! Please head over to this nifty Shiny dashboard to explore the most frequently appearing bigrams in the titles of Medium data science articles published between 2009 and 2019 (as of November 21st).