About the data
The data was archived by heliohost.org and retrieved via Wayback Machine and can be downloaded from this link. About corpus
About the model used
The main corpus was cleaned and proccessed into n-grams tables (unigrams, bigrams and trigrams) for the final model. The model used was the Katz-Backoff that relys on the absolute discounting and backoff methods for estimating the probabilities for words in an unseen context.
\[
\begin{aligned}
\bullet P(w_1|w_1^{n-1}) & \approx \prod_{k=1}^{n}P(w_n|w_{n-N+1}^{n-1})\quad \text{General n-gram equation}\\
\end{aligned}
\\
\bullet P_{Backoff}(w_n|w_{n-N-1}^{n-1}) = \boldsymbol{\{} {\begin{align} P^{discounted}(w_n|w_{n-N+1}^{n-1}),\ \text{if}\qquad C(w_{n-N+1}^{n}) > 0 \\ \\ \alpha(w_{n-N+1}^{n-1})P_{backoff}(w_{n-N+2}^{n-1}),\ \text{otherwise}\end{align}}
\]
