PodCruncher Reviews Recap

We continue (see first article) analysis of PodCruncher Podcast Player reviews on iTunes. This time we attack reviews using text analytics and sentiment analysis.

Short recap first. Before we pulled, parsed and enhanced online iTunes ratings with reviews for PodCruncher resulting in R dataframe containing:

Figure 1. Fragment of the Dataframe with PodCruncher Reviews.
.id id title rating date feeling nchar dateday wday year month yearmonth hour dayPart content
entry 1588794507 Gmanstro ***** 2017-04-14 19:21:25 POSITIVE (4 or 5) 60 2017-04-14 Fri 2017 Apr 2017-04-01 19 Evening Solid app, The Best Easy to use best sma…
entry 1588723073 Nope Nope Nope * 2017-04-14 16:36:24 NEGATIVE (1,2, or 3) 761 2017-04-14 Fri 2017 Apr 2017-04-01 16 Day Been using this app for a few years and …
entry 1588633218 Needs update ** 2017-04-14 14:12:48 NEGATIVE (1,2, or 3) 90 2017-04-14 Fri 2017 Apr 2017-04-01 14 Day Great app but the interface doesn’t fit …

Text Sentiment Scoring

Sentiment Extraction (Scoring)

User sentiment may indicate if and how perception of the app changes and help recongnize leading factors behind such processes. Sentiment extraction is a necessary part of such analysis. There are several packages available on CRAN to analyze text sentiment and package syuzhet was chosen for its concise API, flexibility and feature set.

Using syuzhet extracting and scoring sentiment from the reviews progresses as an iterative process:

  • iterating over each review
  • combining title and review into one text
  • parsing combined text into sentences
  • scoring sentiment for each sentence
  • save sentiment scores for each sentence
  • aggregate (sum up) sentiment scores for each review
  • saving variation of sentence sentiment for each review as a standard deviation of sentence scores
  • do exactly the same for title only

Consolidating Sentiment Scores and Reviews

Finally analytical data set is compiled from both reviews and sentiment scores. This results in sentiment several sentiment scores compute for each review:

  • title sentiment scores
  • overall review (title + content) sentiment scores
  • sentiment score for each sentence in review content

Sentiment analysis below will analyze all 3 types of sentiment trying to understand how and where to find interesting insight from user sentiment.

podcruncherReviewsByTitle =
  merge(podcruncherReviews, sentiments_by_title, by = "id", suffixes = c(".total", ".title"))
podcruncherReviewsBySentences = 
  merge(podcruncherReviews, sentiments_by_sentences, by = "id", suffixes = c(".total", ".sentence"))
podcruncherReviews = cbind(podcruncherReviews, 
                           sentiment=sentiments, 
                           sentiment_variability = sentiment_variability,
                           title_sentiment=title_sentiments,
                           title_sentiment_variability=title_sentiment_variability)

podcruncherReviewsBySentences$nchar =  nchar(podcruncherReviewsBySentences$sentence)

Sentiment Analysis

Sentiment analysis references available online contain plenty of information on how to extract and score sentiment from text but for the most part lack in specifics and examples of concrete analytics using extracted sentiment. With examples below we attempt to showcase how extracted sentiment scores fuse into analysis of actual artifacts that carried sentiment - PodCruncher reviews in this case.

Sentiment Distributions

Simple validation of sentiment scores using histograms shows that 1-,2-, and 3-star reviews indeed shifted towards negative sentiment when compared to 4-, and 5-star reviews:

Figure 2. Review Sentiment Distributions by User Feeling.

Figure 2. Review Sentiment Distributions by User Feeling.

Observe similar shift for title sentiments only:
Figure 3. Title Sentiment Distributions by User Feeling.

Figure 3. Title Sentiment Distributions by User Feeling.

In addition we can see how sentiment distribution shifts lower with each year:
Figure 4. Review Sentiment Distributions by Year.

Figure 4. Review Sentiment Distributions by Year.

There are more sentiment distributions that we will analyze. For example, the distribution of sentiment of each review sentence:
Figure 5. Sentence Sentiment Distributions by User Feeling.

Figure 5. Sentence Sentiment Distributions by User Feeling.

The Most Extreme Reviews by Sentiment

Before diving into analysis of sentiment scores let’s see actual reviews in the top and bottom of sentiment scoring:
Figure 6. Reviews with the highest total sentiment.
rating sentiment date title content
**** 8.75 2016-09-08 03:31:02 Worth it for the playlist options + show notes I listen to a lot of podcasts and for a variety of different purposes: work, self improvement, general tips and how-to, entertainment, meditating, etc. Podcruncher is my fav podcast app because of the options for playlists: create a playlist or a smart playlists, and the ability to download playlists (vs. other apps that only let you stream, not great for NYers who listen on the subway). I also really appreciate the “Show notes” feature that shows the entire description of a podcast episode with working links. I haven’t had any problems syncing with paid podcast subscriptions or login info. The only way this app could be improved are more options to sort within playlists and a more robust search option to find new podcasts (by episode like the native app).
**** 8.00 2014-12-16 06:03:47 Great app

This is my favorite podcast app. Does a great job and even though I keep trying to find better I can’t.

Only reason to not give 5 stars is it doesn’t sync across devices to know what I have already listened to.

I would also really love to have some chromecast integration that would tell chromecast to stream the podcasts from the source.

Great app. Best out there. I hope they keep making it better.

12/15/2014… Still the best and I keep trying others. This is the only one that gets daily use. Would still like to see the improvements mentioned.
**** 7.65 2015-07-30 00:48:57 Very Good . . .

Seems to be a very good app. As many have already said, it just works.

Couple of feature for the developers consideration though . . .

When an episode is deleted, make a way to add it back into the available lineup, instead of having to unsubscribe & resubscribe to that podcast. Apples podcast app does a pretty good job of this with the ‘feed’ button, where all episodes can be seen & selected even after they are “deleted”. (Currently, just mark as played, or Delete Media File for downloaded episodes, instead of deleting, if anyone hasn’t figured this out & is wondering. This will keep it in the lineup.)

Also, an EQ would be nice. People’s voices, as well as recording equipment vary greatly. An EQ would add the ability to tweak the sound so it’s more understandable on some of the “less than stellar” podcasts.

Overall though, GREAT JOB ! ! ! !
while the lowest scoring reviews are:
Figure 7. Reviews with the lowest total sentiment.
rating sentiment date title content
**** -3.90 2015-06-18 14:51:43 Love it but small recent bug is driving me crazy

Hey guys, love this app. I’ve used it for over a year, and besides it occasionally crashing or refusing to load a podcast, requiring restart, it’s awesome.

Lately though there’s a little bug that’s annoying. The search bar in the “Add” screen (little plus sign) displays directly overtop of the All/Title/Author toggle. You can’t really read it and it’s difficult to search. Also, and this isn’t a bug so much as an annoyance: if the search bar could auto-clear after returning to it from elsewhere in the app, that’d be great. Usually I hit the plus sign and “search” and start typing so quickly I don’t even realize I’m typing at the end of my last search phrase, and I have to take the time to clear the search bar and retype. It’s a little thing but it’s annoying for someone who uses the app on a daily basis.
-3.25 2017-01-04 17:47:44 Disappointed Decided to stop using apple’s podcast app due to performance issues. Went with this one because the glowing reviews but it lacks many features I thought were standard. The biggest blind spot is the inability to play oldest to newest. So anything with a narrative is played in reverse. Another issue is that while the play button on my headphones will start a podcast it seems incapable of pausing one.
** -3.00 2016-07-26 04:53:43 Are they studying their own app? When searching for podcasts, the toggle of subject, title, or author actually overlays the input field itself. So you can’t use it. Totally a stupid and bizarre problem and I’m annoyed they haven’t noticed or fixed it.
Figure 8. Reviews with the highest title sentiment.
rating title_sentiment date title content
***** 1.85 2014-10-16 14:47:12 Great app, worth the money

This app is great in general and I’m very happy with it.

One suggestion to make it perfect: I wish there were a way to sync my podcast subscriptions across multiple devices.
**** 1.80 2015-11-16 12:00:23 Works well, perfect for my use Not much to say other than it works well and has just the right features for my purposes. Only thing keeping it from being 5 stars is that the UI doesn’t scale up well on the iPhone 6S (everything is just bigger, doesn’t appear as crisp as it did on the 5S). Not a functional issue at all, but mildly annoying. Still great.
***** 1.80 2015-09-05 11:54:48 Intuitive with easy controls and great playback I hated podcasts because the standard player was buggy and terrible. This app was recommended as a much better alternative. And it is! Easy management with no bugs or hiccups and an easy to use interface. I recommend this app for everyone.
Figure 9. Reviews with the lowest title sentiment.
rating title_sentiment date title content
-1.5 2017-02-28 23:53:08 Interface ugly as sin The interface on this app is about 15 years behind everything else out there. I know that’s picky, but it makes a HORRIBLE first impression. It’s not correctly optimized for a Plus screen. I also encountered bugs just a few minutes into using the app. Skip this one. I regret buying it.
-1.0 2017-04-10 12:26:36 This app is dead This once great app is now a ghost town. Check the update list, and you’ll see that it hasn’t been updated since 2014. Save your money for one of the podcast apps that’s still being actively updated and supported. This app is dead.
*** -1.0 2017-04-05 21:47:05 Too many glitches

Although this podcast app looks pretty good, there are a few frustrating glitches that made me look elsewhere.

One is that on iOS, the search bar is obscured by the title, author, etc., categories.

It was also hit or miss as to whether a podcast would even play. I would have to change what I planned on listening to based on what worked at the time.

You also can’t see every podcast episode done by the creators (though I don’t know if that is an issue with this app or the podcasters themselves). Not having the option to go back and listen to an old episode is frustrating, and this occurred on multiple shows. My friends don’t have the same issue on their apps.

I don’t love the player interface either, though I can’t tell you why because I just tried playing an episode and it wouldnt work so I can’t remind myself of the cons. (I already deleted the app so I no longer have all my podcasts available. I tried downloading a random one to do a proper review.)

This app has potential, but it needs some big changes first.
Figure 10. Review sentences with the highest sentiment.
rating sentiment date title sentence
***** 4.65 2014-10-09 13:25:13 Works great!! It really has made my life so much simpler, grabs new content automatically, and easy to adjust and use .. The ONLY feature to make this better would be a way to select overriding the screen lock from in the App itself.. Unlocking the ip5 while on the bike, strapped to my arm isn’t fun.. Overall tho, highly recommended.
**** 4.05 2016-09-08 03:31:02 Worth it for the playlist options + show notes I listen to a lot of podcasts and for a variety of different purposes: work, self improvement, general tips and how-to, entertainment, meditating, etc. Podcruncher is my fav podcast app because of the options for playlists: create a playlist or a smart playlists, and the ability to download playlists (vs.
*** 3.70 2016-01-31 17:03:43 Need update! With a paid app that’s a clear winner, I expect some love in terms of fresh features according to user requests and to stay ahead of the pack.
Figure 11. Review sentences with the lowest sentiment.
rating sentiment date title sentence
** -3.00 2016-07-26 04:53:43 Are they studying their own app? Totally a stupid and bizarre problem and I’m annoyed they haven’t noticed or fixed it.
** -2.85 2017-01-23 06:29:07 Interface needs work The more minor glitch is that while deleting a podcast by left swiping, as you begin to swipe, the sort/settings/filter header bar disappears causing the entire list of podcasts to shift up resulting in an unsettling impression that the podcast you’re deleting is actually the one beneath the one you intended.
*** -2.60 2016-01-31 17:03:43 Need update! Frequent crashes alone yields any app dead weight.

Review Sentiment by Submission Time

By Day of Week

As in part I let’s look how reivew sentiment correlates with time of review submission. We will look into how sentiment deciles are associated with days of week first:
Figure 18. Reviews Submitted by Rating and Day of Week.

Figure 18. Reviews Submitted by Rating and Day of Week.

If you remember we just defined sentiment polarity using sentiment deciles, so the same plot with deciles gives more precise but busier picture:

Figure 19. Reviews Submitted by Rating and Day of Week.

Figure 19. Reviews Submitted by Rating and Day of Week.

By Time of Day

Figure 20. Reviews Submitted by Rating and Time of Day.

Figure 20. Reviews Submitted by Rating and Time of Day.

Review Lengths

By review length we simply mean a number of characters in each review. This usually indicates user’s involvement or commitment to provide information about the produc and thus longer reviews indicate stronger intent and/or interest.

Figure 21. Review Length Distributions by Rating

Figure 21. Review Length Distributions by Rating

Text Analytics

As before we parse reviews to find words driving user sentiment. Before we formed quasi-documents (combined from multiple reviews) based on feeling (attribute derived from user rating). Similarly we now use polarity based on extracted sentiment from each sentence to form quasi-documents spanning multiple sentences and reviews. Such sentiment-based approach should improve both precision and bias. Indeed, using extracted sentiment improves text correspondence to actual user sentiment, while scoring and analyzing each sentence separately and independently removes bias towards review rating.

Top Words by Polarity

Figure 22. Top 25 Negative and Positive Review Words by Frequency.

Figure 22. Top 25 Negative and Positive Review Words by Frequency.

Using TF-IDF instead of frequency delves into differences between negative and positive polarity in review sentences:
Figure 23. Top 25 Negative and Positive Review Words by Frequency.

Figure 23. Top 25 Negative and Positive Review Words by Frequency.

Top Non-Sentiment Words by Polarity

One may notice that term DISAPPOINTED in the negative barplot above carries little to no new information. Indeed, it belongs to negative polarity top ranking and is obviously expected to be there. What if we can eliminate sentiment scoring terms all together from the plots to narrow down analysis to most relevant words about podcast player app? Actually, that is how same plots appear after removing all sentiment carrying terms:
Figure 24. Top 25 Negative and Positive Non-Sentiment Review Words by Frequency.

Figure 24. Top 25 Negative and Positive Non-Sentiment Review Words by Frequency.

Figure 25. Top 25 Negative and Positive Non-Sentiment Review Words by TF-IDF.

Figure 25. Top 25 Negative and Positive Non-Sentiment Review Words by TF-IDF.

Combining Ratings with Sentiment

So far we completely ignored review ratings instead focusing only on sentiment. Combining them together using feeling (based on rating) and polarity (based on sentiment deciles) gives us opportunity to analyze more intricate drivers behind user reviews:

Figure 26. Top 10 Non-Sentiment Words by Review Feeling and Polarity (Frequency).

Figure 26. Top 10 Non-Sentiment Words by Review Feeling and Polarity (Frequency).

Figure 27. Top 10 Non-Sentiment Words by Review Feeling and Polarity (TF-IDF).

Figure 27. Top 10 Non-Sentiment Words by Review Feeling and Polarity (TF-IDF).