We were asked to use data to answer the question, "Which are the most valued data science skills?"

As a team we used Github and Slack as our method of team collaboration. Within Github, we forked, edited and commited. Then we utilized the blame view to review line-by-line revision history.

process map

We obtained data from Kaggle.com

Jeff Hale obtained data from online job listing sites such as LinkedIn, Indeed, SimplyHired and Monster in the US in October 2018 using Python. When observing this data he noted how many times a keyword was mentioned by post throughout the different platforms.

library(tidyverse)
library(knitr)
library(kableExtra)
library(tm)
library(wordcloud)
library(memoise)
library(SnowballC)
library(RColorBrewer)
library(RCurl)
library(XML)
library(treemap)

We read data from the CSV file which was uploaded to Github.

url <- "https://raw.githubusercontent.com/miachen410/DATA607Project3/master/DataSkills.csv"
data_skills <-read.csv(url, stringsAsFactors = FALSE)
kable(data_skills) %>% kable_styling(bootstrap_options = "striped", font_size = 7)

First we looked at the structure of the dataset.

str(data_skills)

## 'data.frame':    30 obs. of  5 variables:
##  $ Keyword    : chr  "machine learning" "analysis" "statistics" "computer science" ...
##  $ LinkedIn   : chr  "5,701" "5,168" "4,893" "4,517" ...
##  $ Indeed     : chr  "3,439" "3,500" "2,992" "2,739" ...
##  $ SimplyHired: chr  "2,561" "2,668" "2,308" "2,093" ...
##  $ Monster    : chr  "2,340" "3,306" "2,399" "1,900" ...

We removed the commas in the numbers and changed the data types from character to numeric for the following columns: LinkedIn, Indeed, SimplyHired and Monster.

data_skills$LinkedIn <- str_replace_all(data_skills$LinkedIn, ",", "") %>% as.numeric()
data_skills$Indeed <- str_replace_all(data_skills$Indeed, ",", "") %>% as.numeric()
data_skills$SimplyHired <- str_replace_all(data_skills$SimplyHired, ",", "") %>% as.numeric()
data_skills$Monster <- str_replace_all(data_skills$Monster, ",", "") %>% as.numeric()
str(data_skills)

## 'data.frame':    30 obs. of  5 variables:
##  $ Keyword    : chr  "machine learning" "analysis" "statistics" "computer science" ...
##  $ LinkedIn   : num  5701 5168 4893 4517 3404 ...
##  $ Indeed     : num  3439 3500 2992 2739 2344 ...
##  $ SimplyHired: num  2561 2668 2308 2093 1791 ...
##  $ Monster    : num  2340 3306 2399 1900 2053 ...

We got rid of the rows we didn't need by subsetting and eliminating those in which LinkedIn was NA; we also excluded the "Total" row which was not a data science skill.

data_skills_subset <- subset(data_skills, !is.na(LinkedIn)) %>% subset(!Keyword == "Total")
kable(data_skills_subset) %>% kable_styling(bootstrap_options = "striped", font_size = 7)

We mutuated the data frame to generate a new column Total_Mention. Total_Mention was calculated by adding all numbers from the 4 jobboards for each skill.

data_skills_2 <- data_skills_subset %>% mutate(Total_Mention = LinkedIn + Indeed + SimplyHired + Monster) 
kable(data_skills_2) %>% kable_styling(bootstrap_options = "striped", font_size = 7)

We added rows "AI" and "Artificial Intelligence" then subtracted the overlapping skills. We assigned the values to "AI + Artificial Intelligence".

data_skills_2[18,2:6] <- data_skills_2[16,2:6] + data_skills_2[17,2:6] - data_skills_2[18,2:6]

We added rows "NLP" and "Natural Language Processing" then subtracted the overlapping skills. We assigned the values to "NLP + Natural Language Processing".

data_skills_2[21,2:6] <- data_skills_2[19,2:6] + data_skills_2[20,2:6] - data_skills_2[21,2:6]

We then removed the unnecessary rows "AI", "Artificial Intelligence", "NLP" and "Natural Language Processing".

data_skills_tidy <- data_skills_2[- c(16, 17, 19, 20), ]

We mutuated the data frame to generate another new column Percentage. Percentage was calculated by dividing the total number of each skill by the overall total of all skills.

data_skills_tidy <- data_skills_tidy %>% mutate(Percentage = Total_Mention/sum(Total_Mention))

Using the ggplot2 package, we created a bar plot that shows the total frequency of each data science skill mentioned in the jobboards, ranked from highest to lowest.

Using the treemap package, we create a treemap to show the percentage of each data science skill in respect to total.

Using the wordcloud package, we decided to visually determine the most popular words in our data set. We used Data Science Tutorial YouTube Channel YouTube.com as a reference.

We observed the top five keyword/skills mentioned by post throughout the different platforms were Analysis, Machine Learning, Statistics, Computer Science and Communication. These keywords/skills ranked differently across platforms.

In LinkedIn, the rank was as follows: Machine Learning, Analysis, Statistics, Computer Science and Communication. In Indeed and Simply Hired, the rank was as follows: Analysis, Machine Learning, Statistics, Computer Science and Communication. In Monster, the rank was as follows: Analysis, Statistics, Machine Learning, Communication and Computer Science.