DATA607-Week7

Introduction

In addition to being a fun and educational exercise in and of itself, I wanted to use the exercise to answer an interesting question (well, at least one interesting to me!). So I explored a few of the various APIs that NYT makes available, and many presented potentially interesting ideas. In the end, maybe because I had been struggling to find a good novel to read, I ended up choosing the books API. But I didn’t just want to pull down a list of best sellers. I wanted to dig into the data just a bit and see if I could uncover something interesting, so I decided to find out whether Fiction books stay on the best sellers list longer than do Non-Fiction books or vice versa. At this point, I’ve only confirmed feasibility of what I’d like to do, but I haven’t actually captured any data, so I don’t know what the answer is yet.

According to the NYT API README page, the best sellers list URI format is as follows:
http://api.nytimes.com/svc/books/{version}/lists/[date/]{list-name}[.response_format]? [optional-param1=value1]&[...]&api-key={your-API-key}

library(jsonlite)
library(dplyr)
library(tidyr)
library(ggplot2)
library(ggthemes)
api_key = 'api-key=368fbac19ccc4179ae2db761a426699d'
base_url = 'https://api.nytimes.com/svc/books/v3/lists/'
resp_format = '.json'
fict = 'combined-print-and-e-book-fiction'
nonfict = 'combined-print-and-e-book-nonfiction'
fict_url = paste0(base_url, fict, resp_format, '?', api_key)
fict_res = fromJSON(fict_url)
fict_df = fict_res$results$books
fict_df$category = 'fiction'
nonfict_url = paste0(base_url, nonfict, resp_format, '?', api_key)
nonfict_res = fromJSON(nonfict_url)
nonfict_df = nonfict_res$results$books
nonfict_df$category = 'nonfiction'
nytbestsellers = bind_rows(fict_df, nonfict_df)
nytbestsellers$vendor_link = paste0('[Amazon](', nytbestsellers$amazon_product_url, ')')
nytbestsellers$cover = paste0("<img src=", nytbestsellers$book_image, " width='50px'/>")

Top 5 in NYT Best Sellers List

Let’s first have a look at the top five fiction and non-fiction books currently on the list.

knitr::kable(nytbestsellers[1:5, c('category', 'rank', 'title', 'description', 'author', 'vendor_link', 'cover')], row.names = FALSE, caption = 'New York Times Best Sellers - Top 5 in Fiction') 

category	rank	title	description	author	vendor_link	cover
fiction	1	THE GIRL ON THE TRAIN	A psychological thriller set in the environs of London is full of complications and betrayals.	Paula Hawkins	Amazon
fiction	2	VINCE FLYNN: ORDER TO KILL	Flynn’s character, the C.I.A. operative Mitch Rapp, uncovers a dangerous Russian plot. Flynn died in 2013.	Kyle Mills	Amazon
fiction	3	SMALL GREAT THINGS	A medical crisis entangles a black nurse, a white supremacist father and a white lawyer.	Jodi Picoult	Amazon
fiction	4	TWO BY TWO	A man who became a single father when his marriage and business collapsed learns to take a chance on a new love.	Nicholas Sparks	Amazon
fiction	5	A MAN CALLED OVE	A curmudgeon's gruff exterior masks a generosity of spirit. Originally published in Sweden in 2014.	Fredrik Backman	Amazon

knitr::kable(nytbestsellers[21:25, c('category', 'rank', 'title', 'description', 'author', 'vendor_link', 'cover')], row.names = FALSE, caption = 'New York Times Best Sellers - Top 5 in Non-Fiction')

category	rank	title	description	author	vendor_link	cover
nonfiction	1	KILLING THE RISING SUN	The host of “The O’Reilly Factor” recounts the final years of World War II.	Bill O'Reilly and Martin Dugard	Amazon
nonfiction	2	BORN TO RUN	The singer-songwriter's autobiography.	Bruce Springsteen	Amazon
nonfiction	3	FILTHY RICH	The story of a South Florida financier convicted of soliciting sex from underage girls.	James Patterson and John Connolly with Tim Malloy	Amazon
nonfiction	4	HILLBILLY ELEGY	A Yale Law School graduate looks at the struggles of the white working class through the story of his own childhood in the Rust Belt.	J D Vance	Amazon
nonfiction	5	LOVE WARRIOR	After her husband confesses to multiple infidelities, a woman struggles to grow — and so does he.	Glennon Doyle Melton	Amazon

Weeks on NYT Best Sellers List

Next, we’ll take a look at how long books have been on their respective best sellers lists. Based on the histograms below, there doesn’t seem to be a discernible difference between the two book categories (aside from the fiction category having one book that has been on the list for a very long time, which happens to be The Girl on the Train).

nytbestsellers %>% ggplot(aes(x = weeks_on_list, col = category, fill = category)) +
  geom_histogram(alpha = 0.5, binwidth = 1, position = 'identity') +
  facet_grid(category ~ .) + 
  scale_fill_economist() + scale_color_economist() + 
  ggtitle('Number of Weeks on NYT Best Sellers List Histogram')

Turnover of NYT Best Sellers

Perhaps the amount of turnover within the top 20 of each category might prove more interesting. Let’s look at how many new books were added to each list that weren’t there the previous week. Fortunately, the API results provide each book’s ranking for the previous week. All we need to do is count how many were not in the top 20 in the previous week.

And again, there does not seem to be a significant difference. There were 10 new non-fiction additions compared to 9 new fiction additions. I suppose it’s interesting that there is that much turnover from week-to-week. Perhaps in a future study we could look at turnover over time.

nytbestsellers %>% mutate(newAddition = ifelse(rank_last_week == 0, 'New Addition', 'Not New Addition')) %>%
  group_by(category, newAddition) %>%
  summarise(n = n()) %>% 
  mutate(freq = n / sum(n)) %>%
  ggplot(aes(x = category, y = freq, fill = newAddition)) +
  geom_bar(stat = 'identity') +
  scale_fill_economist() + scale_color_economist() + 
  ggtitle('Proportion of New Additions to NYT Best Sellers List')

Rank Changes of NYT Best Sellers

I’m running out of things to compare here! One last comparison then - let’s have a look at how much movement in rankings took place within each category from the previous week to the current week. We’ll evaluate the change in rank from week-to-week. For those books that were not ranked in the previous week, we’ll assume that they were all ranked 21st (just outside of the best sellers list).

Alas, when comparing the average absolute value of rank change, fiction books seem to have moved around just a bit more than non-fiction books; however, without a benchmark, it’s hard to say how big of a difference this really represents.

nytbestsellers %>% mutate(rankChange = ifelse(rank_last_week == 0, 21, rank_last_week) - rank) %>%
  group_by(category) %>%
  summarise(avgAbsRankChange = mean(abs(rankChange)))

All in all, this was a fun exercise even if the takeaways from the data analysis were not earth shattering. Perhaps looking at this data over time would prove more interesting…