The corpus first has all non US-keyboard characters, punction and numbers removed. It is then converted to lower case matched up against a dictionary and has profane words removed. Each text entry is turned into n-grams up to length \( 5 \). For the 1-gram (which is the list of words), and 2-gram we keep all occurances, for all other lengths we drop unique occurances to manage the memory. The tool was able to process \( 85 \)% of the raw data, resulting in approximately 75 MB of storage space required. The tables are stored with the prefix and tail in seperate columns, the count, as well as the score which is the count divided by the count of the frequency of the prefix in the (n-1) gram table.
\[
S\left(w_n \middle\vert w_1^{n-1} \right) =
S\left(w_n \middle\vert w_1, \dots, w_n\right) =
\begin{cases}
\frac{C(w_{i}^n)}{C(w{i+1}^n)},& \text{ if } x > 1\\
\alpha S(w_n \vert w_{2}^{i-1}),& \text{otherwise}
\end{cases}
\]
double check
Details on this slide about how we clean, maybe some details of what we built in n-gram tables.
Details about how we pick which words to score
Details about the scoring algorithm
The Word Cloud feature in the UI and the data table help provide additional insight into the scores.
In a production application we would expect the choice of top three buttons be incorporated into the typing interface as 3 (or more) dynamic hotkeys.
