## [1] "Using direct authentication"

Measuring Town sentiment by using Twitter as a proxy.

One of the great things about R algorithms and the public’s total disconnect from the concept of privacy, is that it is now possible to aggregate twitter feeds and combine individuals’ frames of mind or moods to pass judgment on an aggregate sentiment of geographical areas. Although mapping sentiment geographically is the next step following this analysis, in this report we simply create “mood metrics” from sentiment frequencies to see how we think towns behave. I used the centroid latitude and longitude from the 29 most populous towns in Maine and looped through each pair of coordinates to battle ultimately successfully with twitter and pull a decent number of messages per town.

The feeds were then aggregated by each town with various metrics created to see which towns are in the holiday mood and which towns might be better off hosting a convention of scrooges. All towns are apparently not the same as the results will show.

## [[1]]
## # A tibble: 29 x 7
##                 town latitude longitude radiuslimit   pop  area density
##                <chr>    <dbl>     <dbl>       <dbl> <dbl> <dbl>   <dbl>
##  1       Portland,ME 43.66147 -70.25533         3.3 66937 21.35  3135.2
##  2       Lewiston,ME 44.10035 -70.21478         4.2 36140 35.51  1017.7
##  3         Bangor,ME 44.80118 -68.77781         4.1 31985 34.23   934.4
##  4 South Portland,ME 43.64147 -70.24088         2.5 25577 12.08  2117.3
##  5         Auburn,ME 44.09785 -70.23117         5.7 22948 65.70   349.3
##  6      Biddeford,ME 43.49258 -70.45338         3.9 21362 30.40   702.7
##  7        Sanford,ME 43.43916 -70.77428         4.9 20938 48.74   429.6
##  8      Brunswick,ME 43.91402 -69.96700         4.9 20645 47.28   436.7
##  9    Scarborough,ME 43.59020 -70.33449         4.9 20023 47.72   419.6
## 10           Saco,ME 43.50092 -70.44283         4.4 19213 38.89   494.0
## # ... with 19 more rows
## 
## Attaching package: 'dplyr'
## The following objects are masked from 'package:twitteR':
## 
##     id, location
## The following objects are masked from 'package:stats':
## 
##     filter, lag
## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union

After looping through the cities of Maine and downloading the twitter messages we have created a dataframe with twitter messages as well as the geo coordinates associated with them.

##                                                                                                                                text
## 1 RT @WCSH6: Charles Manson, one of nation's most infamous mass killers, dead at 83 https://t.co/BBS1XHHR6N https://t.co/LZkA9xQ28v
## 2                      New Release &amp; Excerpt: Dennis &amp; Greer by Molly Gould https://t.co/RIaSRsRaUO https://t.co/vErMMSga19
## 3                                                                      not putting punisher in the defenders was a colossal mistake
## 4            Check out MOSSIMO Fold Over Waist Southwest Print Stretch Long Maxi Size Medium #Mossimo #Maxi https://t.co/Yp66tUAh02
## 5                                     Spying Scandal Cartoons and Comics - funny pictures from CartoonStock https://t.co/VumLnxCUtc
## 6                                                                            When did Nasir al-Wuhayshi die https://t.co/HQaMrXHxxb
##   favorited favoriteCount replyToSN             created truncated
## 1     FALSE             0      <NA> 2017-11-20 06:34:05     FALSE
## 2     FALSE             0      <NA> 2017-11-20 06:34:04     FALSE
## 3     FALSE             0      <NA> 2017-11-20 06:33:31     FALSE
## 4     FALSE             0      <NA> 2017-11-20 06:33:26     FALSE
## 5     FALSE             0      <NA> 2017-11-20 06:33:17     FALSE
## 6     FALSE             0      <NA> 2017-11-20 06:33:16     FALSE
##   replyToSID                 id replyToUID
## 1       <NA> 932497254946082816       <NA>
## 2       <NA> 932497253515763712       <NA>
## 3       <NA> 932497114353078272       <NA>
## 4       <NA> 932497091674411009       <NA>
## 5       <NA> 932497056744161280       <NA>
## 6       <NA> 932497048774909952       <NA>
##                                                                          statusSource
## 1  <a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>
## 2                            <a href="https://dlvrit.com/" rel="nofollow">dlvr.it</a>
## 3 <a href="https://about.twitter.com/products/tweetdeck" rel="nofollow">TweetDeck</a>
## 4                <a href="https://www.socialoomph.com" rel="nofollow">SocialOomph</a>
## 5             <a href="https://mobile.twitter.com" rel="nofollow">Mobile Web (M2)</a>
## 6             <a href="https://mobile.twitter.com" rel="nofollow">Mobile Web (M2)</a>
##        screenName retweetCount isRetweet retweeted longitude latitude
## 1     DWLODKOWSKI            1      TRUE     FALSE      <NA>     <NA>
## 2     AntrimCycle            0     FALSE     FALSE      <NA>     <NA>
## 3       TreyAdell            0     FALSE     FALSE      <NA>     <NA>
## 4    Portland_Buz            0     FALSE     FALSE      <NA>     <NA>
## 5 RegulatoryDrone            0     FALSE     FALSE      <NA>     <NA>
## 6    islamoinform            0     FALSE     FALSE      <NA>     <NA>
##       lats     longs       towns
## 1 43.66147 -70.25533 Portland,ME
## 2 43.66147 -70.25533 Portland,ME
## 3 43.66147 -70.25533 Portland,ME
## 4 43.66147 -70.25533 Portland,ME
## 5 43.66147 -70.25533 Portland,ME
## 6 43.66147 -70.25533 Portland,ME

We process the twitter text to make them more easily analyzable for sentiment relationships.

## # A tibble: 6 x 19
##   favorited favoriteCount      replyToSN             created truncated
##       <lgl>         <dbl>          <chr>              <dttm>     <lgl>
## 1     FALSE             0 1followernodad 2017-11-20 04:07:20     FALSE
## 2     FALSE             0 1followernodad 2017-11-20 04:07:20     FALSE
## 3     FALSE             0 1followernodad 2017-11-20 04:07:20     FALSE
## 4     FALSE             0 1followernodad 2017-11-20 04:07:20     FALSE
## 5     FALSE             0 1followernodad 2017-11-20 04:07:20     FALSE
## 6     FALSE             0 1followernodad 2017-11-20 04:07:20     FALSE
## # ... with 14 more variables: replyToSID <chr>, id <chr>,
## #   replyToUID <chr>, statusSource <chr>, screenName <chr>,
## #   retweetCount <dbl>, isRetweet <lgl>, retweeted <lgl>, longitude <chr>,
## #   latitude <chr>, lats <dbl>, longs <dbl>, towns <chr>, word <chr>
## # A tibble: 6 x 2
##        word sentiment
##       <chr>     <chr>
## 1    abacus     trust
## 2   abandon      fear
## 3   abandon  negative
## 4   abandon   sadness
## 5 abandoned     anger
## 6 abandoned      fear
## # A tibble: 6 x 20
##   favorited favoriteCount      replyToSN             created truncated
##       <lgl>         <dbl>          <chr>              <dttm>     <lgl>
## 1     FALSE             0 1followernodad 2017-11-20 04:07:20     FALSE
## 2     FALSE             0 1followernodad 2017-11-20 04:07:20     FALSE
## 3     FALSE             0 1followernodad 2017-11-20 04:07:20     FALSE
## 4     FALSE             0       agiberti 2017-11-20 04:47:36     FALSE
## 5     FALSE             0       agiberti 2017-11-20 04:47:36     FALSE
## 6     FALSE             0       agiberti 2017-11-20 05:30:56     FALSE
## # ... with 15 more variables: replyToSID <chr>, id <chr>,
## #   replyToUID <chr>, statusSource <chr>, screenName <chr>,
## #   retweetCount <dbl>, isRetweet <lgl>, retweeted <lgl>, longitude <chr>,
## #   latitude <chr>, lats <dbl>, longs <dbl>, towns <chr>, word <chr>,
## #   sentiment <chr>

Now that we have the words and sentiments matched up, lets look at the sentiment counts

## # A tibble: 29 x 2
##           towns     n
##           <chr> <int>
##  1    Bangor,ME   208
##  2    Brewer,ME   208
##  3 Brunswick,ME   165
##  4   Topsham,ME   164
##  5 Biddeford,ME   159
##  6      Saco,ME   147
##  7  Lewiston,ME   146
##  8 Westbrook,ME   145
##  9   Windham,ME   144
## 10    Auburn,ME   142
## # ... with 19 more rows

Now let’s take a look at how these cities compare from a positive/negative sentiment perspective

Let’s join all these sentimental frequencies together to compare and contrast town sentiments

##                    Town Total neg pos trust anticipation joy sadness anger
## 1             Auburn,ME   142  19  29    22           20  17       9     6
## 2            Augusta,ME   116  19  20     6           11   8      13    11
## 3             Bangor,ME   208  31  34    22           23  15      19    19
## 4          Biddeford,ME   159  12  45    26           14  25       7    10
## 5             Brewer,ME   208  31  34    22           23  15      19    19
## 6          Brunswick,ME   165  31  29     9           15   8      16    16
## 7     Cape Elizabeth,ME   113  20  18    14           11   8      11     7
## 8           Falmouth,ME   101  14  20    16           11  10       5     8
## 9             Gorham,ME   122  17  13    17           15  11       9     9
## 10         Kennebunk,ME    46   8  12     5            6   3       1     2
## 11           Kittery,ME   103  20  23    11           10   7       4    10
## 12          Lewiston,ME   146  20  29    24           21  16       9     7
## 13            Lisbon,ME   102  13  25    16            7   4       9     7
## 14 Old Orchard Beach,ME    84  20  10     3            9   8      10     9
## 15             Orono,ME    55  10  10     8            9   6       5     2
## 16          Portland,ME   108  30  22    12            2   2      11    10
## 17      Presque Isle,ME   127  18  25    17           17  15       4    13
## 18              Saco,ME   147  18  37    25            9  17       8    11
## 19           Sanford,ME   118  26  32    18            3   4      14     6
## 20       Scarborough,ME   138  20  23    12           12  15      12    12
## 21    South Portland,ME   131  20  25    10           12  18      14    10
## 22          Standish,ME   130   7  28    23           25  19       5     1
## 23           Topsham,ME   164  31  29     9           15   8      15    16
## 24        Waterville,ME   118  14  30    21           18  16       3     4
## 25             Wells,ME   139  23  24    13           17  16       7    13
## 26         Westbrook,ME   145  18  23    18           18  17      13    12
## 27           Windham,ME   144  17  27    15           17  20      11     9
## 28          Yarmouth,ME   101  18  22    12            7   9      11     4
## 29              York,ME   138  28  20    11            6  10      16    12
##    fear disgust surprise
## 1     6       4       10
## 2    16       5        7
## 3    22      14        9
## 4     9       6        5
## 5    22      14        9
## 6    18      16        7
## 7     8      12        4
## 8     8       4        5
## 9    16       9        6
## 10    3       3        3
## 11   10       5        3
## 12    6       4       10
## 13    9       8        4
## 14    5       7        3
## 15    1       4       NA
## 16   13       5        1
## 17    5       6        7
## 18    8       7        7
## 19    9       5        1
## 20   14      12        6
## 21    9       8        5
## 22    2      NA       20
## 23   18      16        7
## 24    3       4        5
## 25   12      10        4
## 26   12      11        3
## 27   11      10        7
## 28    4       6        8
## 29   10      12       13
##                    Town Total neg pos trust anticipation joy sadness anger
## 1             Auburn,ME   142  19  29    22           20  17       9     6
## 2            Augusta,ME   116  19  20     6           11   8      13    11
## 3             Bangor,ME   208  31  34    22           23  15      19    19
## 4          Biddeford,ME   159  12  45    26           14  25       7    10
## 5             Brewer,ME   208  31  34    22           23  15      19    19
## 6          Brunswick,ME   165  31  29     9           15   8      16    16
## 7     Cape Elizabeth,ME   113  20  18    14           11   8      11     7
## 8           Falmouth,ME   101  14  20    16           11  10       5     8
## 9             Gorham,ME   122  17  13    17           15  11       9     9
## 10         Kennebunk,ME    46   8  12     5            6   3       1     2
## 11           Kittery,ME   103  20  23    11           10   7       4    10
## 12          Lewiston,ME   146  20  29    24           21  16       9     7
## 13            Lisbon,ME   102  13  25    16            7   4       9     7
## 14 Old Orchard Beach,ME    84  20  10     3            9   8      10     9
## 15             Orono,ME    55  10  10     8            9   6       5     2
## 16          Portland,ME   108  30  22    12            2   2      11    10
## 17      Presque Isle,ME   127  18  25    17           17  15       4    13
## 18              Saco,ME   147  18  37    25            9  17       8    11
## 19           Sanford,ME   118  26  32    18            3   4      14     6
## 20       Scarborough,ME   138  20  23    12           12  15      12    12
## 21    South Portland,ME   131  20  25    10           12  18      14    10
## 22          Standish,ME   130   7  28    23           25  19       5     1
## 23           Topsham,ME   164  31  29     9           15   8      15    16
## 24        Waterville,ME   118  14  30    21           18  16       3     4
## 25             Wells,ME   139  23  24    13           17  16       7    13
## 26         Westbrook,ME   145  18  23    18           18  17      13    12
## 27           Windham,ME   144  17  27    15           17  20      11     9
## 28          Yarmouth,ME   101  18  22    12            7   9      11     4
## 29              York,ME   138  28  20    11            6  10      16    12
##    fear disgust surprise  NegToPos JoyToAnger AnticipationToDisgust
## 1     6       4       10 0.6551724  2.8333333             5.0000000
## 2    16       5        7 0.9500000  0.7272727             2.2000000
## 3    22      14        9 0.9117647  0.7894737             1.6428571
## 4     9       6        5 0.2666667  2.5000000             2.3333333
## 5    22      14        9 0.9117647  0.7894737             1.6428571
## 6    18      16        7 1.0689655  0.5000000             0.9375000
## 7     8      12        4 1.1111111  1.1428571             0.9166667
## 8     8       4        5 0.7000000  1.2500000             2.7500000
## 9    16       9        6 1.3076923  1.2222222             1.6666667
## 10    3       3        3 0.6666667  1.5000000             2.0000000
## 11   10       5        3 0.8695652  0.7000000             2.0000000
## 12    6       4       10 0.6896552  2.2857143             5.2500000
## 13    9       8        4 0.5200000  0.5714286             0.8750000
## 14    5       7        3 2.0000000  0.8888889             1.2857143
## 15    1       4       NA 1.0000000  3.0000000             2.2500000
## 16   13       5        1 1.3636364  0.2000000             0.4000000
## 17    5       6        7 0.7200000  1.1538462             2.8333333
## 18    8       7        7 0.4864865  1.5454545             1.2857143
## 19    9       5        1 0.8125000  0.6666667             0.6000000
## 20   14      12        6 0.8695652  1.2500000             1.0000000
## 21    9       8        5 0.8000000  1.8000000             1.5000000
## 22    2      NA       20 0.2500000 19.0000000                    NA
## 23   18      16        7 1.0689655  0.5000000             0.9375000
## 24    3       4        5 0.4666667  4.0000000             4.5000000
## 25   12      10        4 0.9583333  1.2307692             1.7000000
## 26   12      11        3 0.7826087  1.4166667             1.6363636
## 27   11      10        7 0.6296296  2.2222222             1.7000000
## 28    4       6        8 0.8181818  2.2500000             1.1666667
## 29   10      12       13 1.4000000  0.8333333             0.5000000

Our first comparison looks at the ratio of negative to positive sentiment. Here any value of NegToPos >1 means the town’s twitterers exhibit more negative sentiment than positive and of course anything below 1, means there is more positive sentiment than negative sentiment. Old Orchard Beach tops this category having twice as many positive sentiment words and negative sentiment, whereas Kennebunk with a negative to psitive ratio of .608 is at the bottom of this category.

##                    Town Total neg pos  NegToPos
## 1  Old Orchard Beach,ME    84  20  10 2.0000000
## 2               York,ME   138  28  20 1.4000000
## 3           Portland,ME   108  30  22 1.3636364
## 4             Gorham,ME   122  17  13 1.3076923
## 5     Cape Elizabeth,ME   113  20  18 1.1111111
## 6          Brunswick,ME   165  31  29 1.0689655
## 7            Topsham,ME   164  31  29 1.0689655
## 8              Orono,ME    55  10  10 1.0000000
## 9              Wells,ME   139  23  24 0.9583333
## 10           Augusta,ME   116  19  20 0.9500000
## 11            Bangor,ME   208  31  34 0.9117647
## 12            Brewer,ME   208  31  34 0.9117647
## 13           Kittery,ME   103  20  23 0.8695652
## 14       Scarborough,ME   138  20  23 0.8695652
## 15          Yarmouth,ME   101  18  22 0.8181818
## 16           Sanford,ME   118  26  32 0.8125000
## 17    South Portland,ME   131  20  25 0.8000000
## 18         Westbrook,ME   145  18  23 0.7826087
## 19      Presque Isle,ME   127  18  25 0.7200000
## 20          Falmouth,ME   101  14  20 0.7000000
## 21          Lewiston,ME   146  20  29 0.6896552
## 22         Kennebunk,ME    46   8  12 0.6666667
## 23            Auburn,ME   142  19  29 0.6551724
## 24           Windham,ME   144  17  27 0.6296296
## 25            Lisbon,ME   102  13  25 0.5200000
## 26              Saco,ME   147  18  37 0.4864865
## 27        Waterville,ME   118  14  30 0.4666667
## 28         Biddeford,ME   159  12  45 0.2666667
## 29          Standish,ME   130   7  28 0.2500000

Our next comparison looks at the ratio of joy to anger sentiment. Here any value of JoyToAnger >1 means the town’s twitterers exhibit more joyful sentiment than anger and of course anything below 1, means there is more angry sentiment than joyful sentiment. Standish apparently is heading into the holiday season with the right attitude. Maybe a larger sample size might temper the cheer and joy but they are certainly well ahead of FAlmouth with a JoyToAnger ratio that would make even the Grinch envious.

##                    Town Total joy anger JoyToAnger
## 1           Standish,ME   130  19     1 19.0000000
## 2         Waterville,ME   118  16     4  4.0000000
## 3              Orono,ME    55   6     2  3.0000000
## 4             Auburn,ME   142  17     6  2.8333333
## 5          Biddeford,ME   159  25    10  2.5000000
## 6           Lewiston,ME   146  16     7  2.2857143
## 7           Yarmouth,ME   101   9     4  2.2500000
## 8            Windham,ME   144  20     9  2.2222222
## 9     South Portland,ME   131  18    10  1.8000000
## 10              Saco,ME   147  17    11  1.5454545
## 11         Kennebunk,ME    46   3     2  1.5000000
## 12         Westbrook,ME   145  17    12  1.4166667
## 13          Falmouth,ME   101  10     8  1.2500000
## 14       Scarborough,ME   138  15    12  1.2500000
## 15             Wells,ME   139  16    13  1.2307692
## 16            Gorham,ME   122  11     9  1.2222222
## 17      Presque Isle,ME   127  15    13  1.1538462
## 18    Cape Elizabeth,ME   113   8     7  1.1428571
## 19 Old Orchard Beach,ME    84   8     9  0.8888889
## 20              York,ME   138  10    12  0.8333333
## 21            Bangor,ME   208  15    19  0.7894737
## 22            Brewer,ME   208  15    19  0.7894737
## 23           Augusta,ME   116   8    11  0.7272727
## 24           Kittery,ME   103   7    10  0.7000000
## 25           Sanford,ME   118   4     6  0.6666667
## 26            Lisbon,ME   102   4     7  0.5714286
## 27         Brunswick,ME   165   8    16  0.5000000
## 28           Topsham,ME   164   8    16  0.5000000
## 29          Portland,ME   108   2    10  0.2000000

Our last comparison looks at the ratio of anticipation to disgust sentiment. Here any value of AnticipationToDisgust >1 means the town’s twitterers exhibit more the sentiment of anticpation than that of disgust and of course anything below 1, means there is more disgust than anticipation sentiment. And although Sanford is very excited about something, it still doesn’t come near Standish which with a frequency of 27 for anticipation and zero for disgust didn’t even rank due to the magic of zero denominators. Of course, Standish was also feeling the joy with a JoyToAnger ratio of 20, so it might be worth a trip just to see what is happening in Standish before the holidays. Unforunately, Falmouth, which we noted earlier was not feeling the joy of the holiday season, apparently has very little to anticipate with an AnticipationToDisgust ratio at the bottom of the rankings with only .625 compared to Standishes, well, ??? !

##                    Town Total anticipation disgust AnticipationToDisgust
## 1           Lewiston,ME   146           21       4             5.2500000
## 2             Auburn,ME   142           20       4             5.0000000
## 3         Waterville,ME   118           18       4             4.5000000
## 4       Presque Isle,ME   127           17       6             2.8333333
## 5           Falmouth,ME   101           11       4             2.7500000
## 6          Biddeford,ME   159           14       6             2.3333333
## 7              Orono,ME    55            9       4             2.2500000
## 8            Augusta,ME   116           11       5             2.2000000
## 9          Kennebunk,ME    46            6       3             2.0000000
## 10           Kittery,ME   103           10       5             2.0000000
## 11             Wells,ME   139           17      10             1.7000000
## 12           Windham,ME   144           17      10             1.7000000
## 13            Gorham,ME   122           15       9             1.6666667
## 14            Bangor,ME   208           23      14             1.6428571
## 15            Brewer,ME   208           23      14             1.6428571
## 16         Westbrook,ME   145           18      11             1.6363636
## 17    South Portland,ME   131           12       8             1.5000000
## 18 Old Orchard Beach,ME    84            9       7             1.2857143
## 19              Saco,ME   147            9       7             1.2857143
## 20          Yarmouth,ME   101            7       6             1.1666667
## 21       Scarborough,ME   138           12      12             1.0000000
## 22         Brunswick,ME   165           15      16             0.9375000
## 23           Topsham,ME   164           15      16             0.9375000
## 24    Cape Elizabeth,ME   113           11      12             0.9166667
## 25            Lisbon,ME   102            7       8             0.8750000
## 26           Sanford,ME   118            3       5             0.6000000
## 27              York,ME   138            6      12             0.5000000
## 28          Portland,ME   108            2       5             0.4000000
## 29          Standish,ME   130           25      NA                    NA