Group 2 - CIS-8392 Project Proposal - Stack Exchange Text Mining & Predictive Analytics

Saurabh Binani
Kashyap Dave
Kun Zhou
Jay Patel

3/25/2021

Data

Data is extracted using the Stack Exchange API for Stack Exchange which needs to be setup
API to be used is https://api.stackexchange.com/docs
The data includes details like answers, badges, comments, events, info, posts, privileges, questions, revisions, search, tags, users etc.

Identifying (by prediction) the topics (categories/stack exchange websites) wherein the most engagement will occur in the future
Predicting the reputation of users by badges in future (gold badge/silver badge/bronze badge)

Data Gathering
Data cleaning and Pre-processing:
- Tokenization
- Normalization (lower casing, stemming, lemmatization)
- Removing stopwords
- Term Frequency (tf-idf)
Configuring the URL sources: install.packages(c(“repurrrsive”, “jsonlite”, “tidyverse”))
Understanding the domain structure: Q&A -> json -> NLP(tidytext, stringr, lubridate)
Test crawl: result <- stream_in(file(https://api.stackexchange.com/docs”))
Crawl restrictions: time period, a certain domain (stack overflow, photography etc.)
Applying regression modeling techniques and performing visualization

Use text mining on answers on various websites of stack exchanges to identify the type of category in which discussions happen
Visualize and understand by means of bar plot, pie chart, wordcloud, correlation matrix the patterns on usage of all the stack exchanges child websites
Predict the future usage of stack exchange websites by applying regression modeling techniques to predict the number of users and popularity (by posts, questions, answers)
Predict the reputation of users by analyzing historical data and recommendation for any kind of reward process to keep the stack exchange community engaged