Since the combination of four words normally appear less than two or three words, but it's more likely to appear if the first three words typed together.
For example, when we type “consumer financial protection” we'll normally expect the next word as “bureau”, but as only for “protection”, the prediction will be “agency”, and the combination of “consumer financial protection bureau” word “agency”
We try to put more weights into the function for four words then three words, and also more than two words to fix that “bureau” will appear sooner as prediction word 1.
bidata %>%
filter(db == "protection")
# A tibble: 3 x 3
# Groups: db [1]
db prediction n
<chr> <chr> <int>
1 protection agency 332
2 protection act 92
3 protection district 70
fourdata %>%
filter(db == "consumer financial protection")
# A tibble: 1 x 3
db prediction n
<chr> <chr> <int>
1 consumer financial protection bureau 55
