Data Science Capstone Project - Text Prediction App

author: Jaeyuel Park

date: 05/02/2025

autosize: true

Text Prediction Challenge

• A sophisticated text prediction web application was developed
• Text data was sourced from Twitter, News sites, and Blogs
• Language model training utilized observed word combinations (n-grams)
• Model evaluation performed on independent testing dataset
• Interactive Shiny web app implements real-time prediction
• Higher n-gram models showed improved prediction accuracy
• Performance optimization balanced accuracy with computational efficiency
• Memory management strategies implemented for large-scale data processing

Language Model Development

• Language modeling based on n-gram frequency analysis
• N-grams capture sequential word patterns in text
• Text preprocessing and n-gram extraction pipeline
• Base words and following words frequency analysis
• Maximum Likelihood Estimation (MLE) calculation
• Optimized lookup tables for Shiny application
• Efficient model integration with web interface

Data Processing and Model Training

Text Preprocessing Pipeline

• Linguistic Standardization

  • Contraction normalization
  • Abbreviation expansion
  • Numerical expression handling
  • Special character normalization

• Social Media Processing

  • Twitter-specific text handling
  • Hashtag and mention processing
  • URL and emoji handling

• Text Cleaning

  • Redundant whitespace removal
  • Punctuation normalization
  • Case standardization

N-gram Development

• Sequential Processing

  • Sentence boundary detection
  • Word tokenization
  • N-gram extraction (n = 2 to 5)
  • Frequency analysis
  • Statistical pruning
  • Probability estimation

Web Application Implementation

Features

• Interactive text input interface
• Real-time text preprocessing
• Dynamic n-gram pattern extraction
• Multi-level model consultation
• Probability-based word prediction

LINK: https://pjy1993.shinyapps.io/datascience/