2023-07-01

Quick explanation

The code is extremely simple, it just:

  1. Reads the .txt files

  2. Samples the files up to 1%

  3. Converts the files and preprocess them using the tm package

  4. Saves the function to local to be loaded by the server.R

Code

library(tm)
library(NLP)
library(RWeka)
library(plotly)
library(dplyr)
library(stringr)
library(quanteda)
library(data.table)
setwd("C:/Users/andre/OneDrive - unizar.es/Data Science/RStudio/Data products/DataScience_Coursera_Repo/Final Capstone")

getCorpus <- function(){
  #Load Data
  con <- file("en_US.news.txt", open="r")
  news_text <- readLines(con); close(con)
  
  con <- file("en_US.blogs.txt", open="r")
  blogs_text <- readLines(con); close(con) 
  
  con <- file("en_US.twitter.txt", open="r")
  twit_text <- readLines(con); close(con)
  rm(con)
  
  #Sampling
  set.seed(2510)
  blogs_text <- sample(blogs_text, size = 500)
  news_text <- sample(news_text, size = 500)
  twit_text <- sample(twit_text, size = 500)
  
  # Union corpora
  corpora <- c(news_text,blogs_text,twit_text)
  corpora <-  iconv(corpora, to ="utf-8")
  corpora <- VectorSource(corpora)
  corpora <- VCorpus(corpora)
  
  corpora <- preprocess(corpora)
  corpora
}

preprocess <- function(text){
  toSpace <- content_transformer(function(x, pattern) gsub(pattern, " ", x))
  text <- tm_map(text, toSpace, "/|@|//|$|:|:)|*|&|!|?|_|-|#|")
  text <- tm_map(text, removeNumbers)
  text <- tm_map(text, content_transformer(tolower))
  text <- tm_map(text, removePunctuation)
  text <- tm_map(text, stemDocument)
  text <- tm_map(text, stripWhitespace)
  text
}

saveRDS(getCorpus(),'corpus.RData')

How to use it

  • Use the Slider to select the proper N-gram
  • Write your sentence to be completed
  • Click Predict and wait some seconds. It takes a while, but stay tuned!
  • You are done!! I hope you enjoyed it

References