NLP

Summar: The goal of this project is just to display that you’ve gotten used to working with the data and that you are on track to create your prediction algorithm. Please submit a report on R Pubs (http://rpubs.com/) that explains your exploratory analysis and your goals for the eventual app and algorithm. This document should be concise and explain only the major features of the data you have identified and briefly summarize your plans for creating the prediction algorithm and Shiny app in a way that would be understandable to a non-data scientist manager. You should make use of tables and plots to illustrate important summaries of the data set. The motivation for this project is to: 1. Demonstrate that you’ve downloaded the data and have successfully loaded it in.2. Create a basic report of summary statistics about the data sets.3. Report any interesting findings that you amassed so far.4. Get feedback on your plans for creating a prediction algorithm and Shiny app.

Open the file conection Set seed for reproducible reseach Load the data intp T,B,N For each of the dataset, take a sample of 10000. After that, close the connection

A brief data summary. As we can see, the upper limit of twitter is 140.

summary(sapply(T, nchar))

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    2.00   37.00   64.00   68.68  100.00  140.00

summary(sapply(B, nchar))

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##       1      47     156     230     329   40833

summary(sapply(N, nchar))

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##     1.0   110.0   185.0   201.2   268.0 11384.0

This is to calculate the final exploratory data analysis dataframe

size_t <- file.size("Final/en_US/en_US.twitter.txt") / 1024^2 
size_b <- file.size("Final/en_US/en_US.blogs.txt") / 1024^2 
size_n <- file.size("Final/en_US/en_US.news.txt") / 1024^2 

EDA = data.frame(
        "file size (MB)" = c(size_t, size_b, size_n),
        "line counts" = c(length(T), length(B), length(N)),
        "word counts" = c(wordcount(T), wordcount(B), wordcount(N)),
        "element counts" = c(sum(nchar(T)),sum(nchar(B)),sum(nchar(N))),
        row.names=c("twitter", "blog", "news"))

EDA

##         file.size..MB. line.counts word.counts element.counts
## twitter       159.3641     2360148    30373543      162096031
## blog          200.4242      899288    37334131      206824505
## news          196.2775     1010242    34372530      203223159

To visualize the table

par(mfrow=c(1,2))
barplot(EDA$line.counts,names.arg = c("twitter","blog","news"), 
        xlab = "type of dataset", ylab = "line counts")
barplot(EDA$word.counts,names.arg = c("twitter","blog","news"), 
        xlab = "type of dataset", ylab = "word counts")

par(mfrow=c(1,2))
barplot(EDA$element.counts,names.arg = c("twitter","blog","news"), 
        xlab = "type of dataset", ylab = "element counts")
barplot(EDA$file.size..MB.,names.arg = c("twitter","blog","news"), 
        xlab = "type of dataset", ylab = "file size")