The WordFlow engine is a high-performance natural language processing tool designed to predict the next word in a sequence.

• The Challenge: Processing massive text corpora while maintaining a small enough memory footprint for web deployment.
• The Solution: A pruned 4-gram back-off model optimized for the M4 architecture and Shiny Server environments.
• The Product: A reactive web interface that provides instant suggestions as the user types.

Experience the App: https://vivpro.shinyapps.io/NextWord/

The engine utilizes a Katz-style Back-off model to balance context and probability.

1. Quadgram Search: The model first attempts to match the last three words of input.
2. Back-off Logic: If no match exists, it sequentially “backs off” to Trigrams (2 words) and Bigrams (1 word).
3. Deep Search Filtering: To improve accuracy, the model filters out high-frequency “grammar filler” words (stopwords) during the search process, allowing meaningful nouns and verbs to surface.

Building a responsive app required significant data engineering to stay within the 1GB RAM limit of the Shiny server.

• Corpus: Compiled from millions of lines of Twitter, Blog, and News data.
• Cleaning: Automated scripts removed profanity, punctuation, and non-ASCII characters.
• Top-1 Pruning: To ensure speed, the model was “pruned” to only store the single most likely prediction for every unique context.
• Efficiency: This reduced the final model size by over 80% without sacrificing the primary “Top-1” accuracy.

The WordFlow engine was benchmarked for speed and responsiveness.

• Latency: Average prediction runtime is 0.04 seconds, providing a “zero-lag” user experience.
• Resource Usage: The total size of the predictive .rds files is under 50MB, ensuring rapid startup times on the web.
• Accuracy: The back-off hierarchy ensures that the model always provides a statistically sound suggestion, even for rare phrases.

The Shiny application was designed with a “Mobile-First” philosophy using the Cosmo theme.

1. Input: Type any phrase into the text entry box.
2. Real-time Processing: The engine detects changes and updates the prediction instantly.
3. Output: The predicted word is displayed in large, high-contrast text for easy reading.

Technical Stack: Developed in R using data.table for high-speed lookups and shiny for the reactive interface.