1 Executive summary

This milestone explores the three US English text files supplied for the Coursera Data Science Capstone and turns the findings into a practical plan for a next-word prediction product. The corpus contains blogs, news articles and Twitter posts, giving the model a useful mix of long-form, edited and conversational English.

Main finding: Twitter contributes most of the lines, while blogs contribute the most words. Sampling only by line would therefore over-represent short social posts. The modelling pipeline should use source-stratified sampling and memory-efficient processing.

2 Data and method

The source is the Coursera SwiftKey US English corpus. Lines represent newline-delimited text records. Word totals use whitespace-delimited tokens as a transparent baseline; a production tokenizer may give slightly different totals because of contractions, punctuation, links and emojis.

Basic corpus inventory
Source File Size (MB) Lines Words Average words per line
Blogs en_US.blogs.txt 200.42 899,288 37,334,114 41.5
News en_US.news.txt 196.28 1,010,242 34,372,520 34.0
Twitter en_US.twitter.txt 159.36 2,360,148 30,373,543 12.9

Across all sources there are 4,269,678 lines, 102,080,177 words and approximately 556.06 MB of text.

3 Corpus comparisons

3.1 Important findings

  • Twitter dominates by record count: it supplies 55.3% of all lines, but only 29.8% of all words. Its records average just 12.9 words.
  • Blogs contain the most text: blogs contribute 36.6% of all words and have the longest records at 41.5 words per line.
  • News provides a useful middle ground: its average record length is 34.0 words, between blogs and Twitter.
  • Source balance matters: equal sampling by line would bias the training set toward short, informal language. A stratified sample by source and token volume will preserve style diversity.
  • The full corpus is large enough to require pruning: n-gram tables should discard very rare sequences and retain only the information needed for accurate, fast prediction.

4 Text preparation strategy

The raw text will be processed in reproducible, memory-aware batches:

  1. Read each source in UTF-8 and draw a fixed-seed stratified training sample, with separate validation and test sets.

  2. Convert text to lowercase while retaining apostrophes inside contractions.

  3. Replace or remove URLs, email addresses, user handles, repeated punctuation, numbers and unsupported symbols.

  4. Remove profanity from candidate predictions without deleting surrounding training context.

  5. Tokenise sentences and count unigrams, bigrams, trigrams and four-grams.

  6. Prune rare n-grams, store compact lookup tables and record source-level diagnostics.

A held-out test set will never be used to construct the frequency tables. This keeps the reported accuracy honest.

5 Prediction algorithm and Shiny application plan

5.1 Plain-English algorithm

When a user enters a phrase, the predictor will clean it using the same rules as the training data and inspect the final words. It will first search for phrases that match the last three words. If no reliable match exists, it will back off to the last two words, then the last one. Candidate next words will be ranked using frequency, backoff weight and a small probability discount so that common phrases are preferred without letting one source dominate.

Prediction flow
Step Action
1 Normalise the user’s phrase
2 Extract the last one to three words
3 Search four-gram, trigram and bigram tables
4 Back off and rank candidate next words
5 Return the highest-ranked safe word

5.2 Product design

The Shiny app will contain one text box, a Predict next word button and a clearly highlighted result. The model will load once when the application starts, so each prediction requires only a small table lookup. Optional ranked alternatives can be added after the core one-word output is reliable.

Performance will be evaluated on:

  • Top-one accuracy: percentage of held-out phrases where the first prediction equals the observed next word.
  • Coverage: percentage of inputs for which the model returns a prediction.
  • Latency: median time from clicking the button to displaying the result.
  • Model size: compressed lookup-table size used by the deployed app.

6 Risks, limitations and next steps

The corpus reflects public web writing rather than every type of English. It may contain bias, slang, personal information and offensive language. Source balancing, profanity filtering and careful validation will reduce these risks, but they cannot remove them completely.

The next steps are to run the cleaning pipeline on a stratified sample, build and prune the n-gram tables, tune the backoff weights on validation data, measure top-one accuracy, and deploy the smallest accurate model in Shiny.

7 Conclusion

The exploratory analysis shows that the three sources differ sharply in record length and contribution. A source-balanced, backoff n-gram model is therefore a practical design: it is explainable, fast enough for an interactive Shiny application and small enough to deploy after pruning.