Milestone Report: Exploratory Analysis for a Predictive Text App

1. The three data sources

We were given three large collections of real English text: blogs, news, and Twitter posts. Below is a summary of each file’s size.

Source Lines Words Size (MB)
Blogs 899,288 37,334,131 210.2
News 1,010,242 34,372,530 205.8
Twitter 2,360,148 30,373,543 167.1
Total 4,269,678 102,080,204 583.1

2. Overall size of the three text sources

3. How long is a typical line

Using a random sample of 30,000 lines from each source, we counted words per line:
Source Average words/line Median words/line Longest line (words)
Blogs 41.5 28 6630
News 33.9 31 360
Twitter 13.0 12 34

4.What words and prases shows more often

After removing extremely common “glue” words (the, and, to, etc.), here are the words that stand out most in each source:

5.How big does our “dictionary” need to be?

A key design question for the app: do we need to remember every word ever used, or can a smaller list of common words cover most of what people actually type? We sorted words by frequency and measured how much of everyday text a growing dictionary would capture.

Source Unique words (sample) Words needed for 50% coverage Words needed for 90% coverage
Blogs 51,475 106 6,034
News 49,886 192 7,488
Twitter 28,336 126 4,815

6.Data quality notes

Profanity is present. In our sample, profane language appeared in about 1.2% of Blog lines, 0.1% of News lines, and 3.2% of Twitter lines. As instructed by our partner, we will filter this out so the app never predicts or displays offensive words. The text is “messy” in normal, expected ways — contractions, typos, inconsistent capitalization, numbers, hashtags, and abbreviations (especially on Twitter). Light cleanup (lower-casing, removing stray punctuation) will happen before modeling. Each source has a distinct voice. A model trained only on News would likely predict poorly for casual texting, and vice versa — so our training data should blend all three sources rather than relying on just one.

7. Plan for the prediction algorithm

The technique we’ll use is called an n-gram model. Rather than trying to understand meaning, it learns statistically which words tend to follow which other words, based on millions of real examples. For instance, after “I went to the”, it has seen that words like “store”, “gym”, or “doctor” commonly come next, and ranks them accordingly.

Build phrase tables from 1-word, 2-word, 3-word, and 4-word sequences across all three sources combined. Rank likely next words by matching whatever the user has typed against the closest phrase we’ve seen before. Handle phrases we’ve never seen. No data set covers every possible sentence, so we’ll use a standard “backing off” technique — falling back to a shorter phrase match, with smoothing so even rare or unseen combinations get a reasonable (never zero) chance. Trim for speed and size. Using the coverage findings above, we’ll drop extremely rare phrases so the final model is small and fast enough to respond instantly inside a web app.

8. Plan for the Shiny app

The final product will be a simple, interactive web page with:

A text box where the user types a phrase. A “Predict” button that instantly shows the top 3 most likely next words. A lightweight, mobile-friendly design so it loads and responds quickly, mirroring the experience of a phone keyboard.

9. What we’d like feedback on

Whether prioritizing speed (a smaller, faster model) over squeezing out a little more accuracy is the right tradeoff for this app’s intended use. Whether there are specific writing styles (formal vs. casual) that matter most for the target users. Any additional data sources worth incorporating later (e.g., domain-specific text).