3.1 Summary statistics

The full data consists of thee diffferent sources: “news”, “blogs” and “twitter”. The news data has 899288 lines, the blogs data has 1010242 lines and twitter data is the longest with 2360148 lines. The basic summaries are as follows.

data <- readRDS("../data/data-prep_20201118.rds")
swfull <- readRDS("../data/sw-prep_20201118.rds")

3.2 Word counts

I created n-grams from a (randomized) subset of 30 percent of the original data. The n-grams were counted by the < ngram function of the ngram package, e.g. two-word counts by <tt ngram(data, n=2) . The ngrams were calculated line-by-line, so no ngrams crossing line boundaries were counted (because these are meaningless). We get the frequency tables of ngrams by using the get.phrasetable() function of the same package. I only saved ngrams with a frequency of at least 2 in order to save memory and because ngrams which occured only once are meaningless in order to predict words.

The most frequent words and word combinations are shown here. Because simple words are hardly very interesting, we calculate the same for the corpus without stopwords:

4.1 primitive algorithm

So far I have tried to create a simple prediction algorithm (see appendix):

• When the user enters a string like “hello and good evening”, for example, it checks whether there are any 5-grams starting with “hello and good evening”.
• If there are, it returns the next word of the most frequent 5-grams (in this case: ).
• If there aren’t, it checks whether there are any 4-grams starting with “and good evening” (so it removes the first word of the string first).
• If there are, it returns the next word of the most frequent 4-grams (in this case: ).
• If there aren’t, … (and so on)
• If no n-grams are found, it just returns the most frequent word.

The algorithm lets the user decide if he or she wants to include stopwords or just want a “meaningful” result.

4.2 Results

The prediction accuracy so far has not been really well. It does not depend so much on the size of the training dffata set (here: 10, 20, 30, 40 or 50 percent of the original data set). It does depend, however, a lot on whether you are looking for a word including stopwords (higher accuracy) or for a meaningful word (lower accuracy).

We can see that the average accuracy of the predictions does not increase for the size of the training data set (x-axis,in percent), neither for words including stopwords (left) nor for words excluding stopwords).

But in the lower figures we can see that the average time for the results increases with larger training data. Hence, it seems best to base the results on a smaller ttraining data set to increase user experience.

Also, the size of ngrams (“steps”) does not influence the prediction, which is rather strange). I have to work on that.

6.1 prediction algorithm

nextword <- function(x, pick=F, n=1, sw=F, permille=1, steps=2 ) {
  
  # preprocess the entered string similar to corpus
  word <- preprocess(
    concatenate(x), 
    case="lower",
    remove.punct=T,
    remove.numbers=T,
    fix.spacing=T)
  
  # separate into single words
  word <- str_split(word, " ", simplify=T)
  if(sw) word <- word[!word %in% tm::stopwords()]
  
  comb <- data.frame()
  
  # words at most 4 words long
  if(length(word) >= (steps-1)) word <- word[(length(word)-(steps-2)):length(word)]
  
  # search for ngrams until match is found
  while(nrow(comb) <= 1 & length(word) > 0) {
    
    if(sw) tabnow <- tab_sw[[permille]][[length(word)+1]] 
    if(!sw) tabnow <- tab_corpus[[permille]][[length(word)+1]] 
    word <- paste(word, collapse=" ")
    
    # where does beginning of string match?
    comb <- tabnow %>% 
      filter(str_detect(ngrams, paste0("^", word, "\\s"))) #%>%
    
    # table or single predicted word?
    if(pick) {
      comb <- comb %>%
    arrange(desc(freq)) %>%
    filter(row_number() <= n) %>% 
    select(ngrams)
      
      comb <- str_split(comb$ngrams, " ", simplify = T)
      
    }
    
    word <- str_split(str_remove(word, "\\s$"), " ", simplify = T)[-1]
  
}


# result
  
  # if match was found
if(length(comb) > 1) {
  
  if(!pick) nextword <- comb
  if(pick) nextword <-  comb[,ncol(comb)]
  return(nextword)
  
  # if not: return most frequent single word
} else { 
  
  if(!sw) nextword <- tab_corpus[[permille]][[1]][1:n,1]
  if(sw)  nextword <- tab_sw[[permille]][[1]][1:n,1]
  
  return(nextword) 
}

}