Distinguishing personal from organisational tweets: proposed algorithm

This note sets out an algorithm for distinguishing tweets from individuals from organisational or corporate tweets.

It uses natural language processing to parse and annotate the tweets and named entity recognition (NER) to label tweets with person or organisation tags.

The algorithm is as follow:

1. Download relevant tweets
2. Join user name and user description fields
3. Parse the description field using spacy_parse with the lemma, tag, entity and nounphrase arguments set to TRUE.
4. Create a new field pasting the entity to the token
5. Group by doc_id and collapse back into sentences
6. Add a flag where lemma = -PRON- and token in I, me, mine, am (first person singular)
7. Add a classification field = person if sentence contains a [PERSON] entity tag, or if flag = 1, else = “org”

Code as follows - using descriptions of people who retweet @PublicHealthBot.

Get tweets

q <- "@PublicHealthBot"

results <- get_timeline(q, n = 18000, retry_on_ratelimit = TRUE)

Join user name and retweet description text and add doc id field

results1 <- results %>%
  mutate(desc1 = paste(retweet_name, retweet_description), 
         row = row_number(), doc_id = paste0("text", row)
  )

Reduce variables; parse text

## slim down variables 

results2 <- results1 %>%
  select(created_at, text, row, doc_id, doc_id, desc1) 

results2 %>%
  head() %>%
  knitr::kable()

#PublicHealth #ArtificialIntelligence #ML #HealthTech https://t.co/xX5lIVUVUj 4 text4 Steph S. ♻️ #Innovation #Disruption | Marketing Manager | @Accenture Health & Public Service | Photography, Emergency Preparedness/Response Volunteer | Opinions my own.
2020-01-02 20:03:24 Public health should strive to attract professionals with broadly applicable data-science skills." 5 text5 ClinEpiDB A global Clinical Epidemiology data resource from collaborators at University of Pennsylvania, University of Georgia & University of Liverpool
2020-01-02 16:22:32 A computer algorithm has been shown to be as effective as human radiologists in spotting breast cancer from x-ray images. https://t.co/PNAlhVNsNc via @imperialcollege #GlobalHealth 6 text6 USAID STAR Project @USAID-supported #globalhealth project, implemented by @phidotorg. We engage talented professionals in capacity-building projects at GH orgs. https://t.co/cZvg3KKB8N

## parse 

parsed <- spacy_parse(results2$desc1, tag = TRUE, entity = TRUE, nounphrase = TRUE)

## Found 'spacy_condaenv'. spacyr will use this environment

## successfully initialized (spaCy Version: 2.0.16, language model: en)

## (python options: type = "condaenv", value = "spacy_condaenv")

head(parsed)

##   doc_id sentence_id token_id      token      lemma   pos tag entity nounphrase
## 1  text1           1        1    IDDjobs     iddjob  NOUN NNS  ORG_B   beg_root
## 2  text1           1        2          :          : PUNCT   :                  
## 3  text1           1        3 infectious infectious   ADJ  JJ               beg
## 4  text1           1        4    disease    disease  NOUN  NN               mid
## 5  text1           1        5   dynamics    dynamic  NOUN NNS               mid
## 6  text1           1        6       jobs        job  NOUN NNS          end_root
##   whitespace
## 1      FALSE
## 2       TRUE
## 3       TRUE
## 4       TRUE
## 5       TRUE
## 6       TRUE

Paste entity to token

parsed1 <- parsed %>% 
  mutate( pspn = ifelse(lemma == "-PRON-" & token %in% c("I", "my", "am", "mine"),1, 
                        ifelse(str_detect(lemma, "[Vv]iew|[Pp]ersonal"), 1, 0)) , 
          annotated = ifelse(nchar(entity) > 0 , paste(token, "[", entity, "]"), token))

head(parsed1)

##   doc_id sentence_id token_id      token      lemma   pos tag entity nounphrase
## 1  text1           1        1    IDDjobs     iddjob  NOUN NNS  ORG_B   beg_root
## 2  text1           1        2          :          : PUNCT   :                  
## 3  text1           1        3 infectious infectious   ADJ  JJ               beg
## 4  text1           1        4    disease    disease  NOUN  NN               mid
## 5  text1           1        5   dynamics    dynamic  NOUN NNS               mid
## 6  text1           1        6       jobs        job  NOUN NNS          end_root
##   whitespace pspn         annotated
## 1      FALSE    0 IDDjobs [ ORG_B ]
## 2       TRUE    0                 :
## 3       TRUE    0        infectious
## 4       TRUE    0           disease
## 5       TRUE    0          dynamics
## 6       TRUE    0              jobs

Collapse into sentences

sentences <- parsed1 %>% 
  group_by(doc_id, sentence_id) %>%
  mutate(annotated_sentence = paste(annotated, collapse = " ")) %>%
  ungroup() %>%
  select(doc_id, annotated_sentence, pspn) %>%
  distinct() %>%
  group_by(doc_id) %>%
  mutate(sumpsp = sum(pspn), 
         annotated_description = paste(annotated_sentence, collapse = ". ")) %>%
  ungroup() %>%
  select(doc_id, annotated_description, sumpsp) %>%
  distinct()


head(sentences)

## # A tibble: 6 x 3
##   doc_id   annotated_description                                          sumpsp
##   <chr>    <chr>                                                           <dbl>
## 1 text1    "IDDjobs [ ORG_B ] : infectious disease dynamics jobs Find [ …      0
## 2 text10   "rikkert Eenhoorn [ ORG_B ] Corporate [ ORG_I ] Accountmanage…      0
## 3 text100  "Joel [ PERSON_B ] Lindsey [ PERSON_I ] Sr [ PERSON_I ] . Exe…      1
## 4 text1000 "Sven Awege Sven AWEGE. \" Dot Connector \" # healthcare # in…      0
## 5 text1001 "jwindz. Sow the wind and reap the whirlwind"                       0
## 6 text1002 "English [ LANGUAGE_B ] @. ECU. At the [ ORG_B ] ECU [ ORG_I …      0

Categorize

sentences <- sentences %>%
  mutate(person = ifelse(str_detect(annotated_description, "PERSON")|sumpsp > 0, "person", "org"))


sample <- sample_n(sentences, 20)

sample %>%
  knitr::kable()