Workspace preparation

Create vector with all needed libraries.

 load_packages <- c(
                    "knitr",
                    "RMySQL",
                    "tidyverse",
                    "tidyr",
                    "dplyr",
                    "stringr",
                    "plotly",
                    "htmlTable",
                    "stringr",
                    "prettydoc",
                    "shinythemes",
                    "treemap",
                    "data.tree",
                    "janitor",
                    "ggplot2",
                    "ggthemes",
                    "stats"
                  )

Organization and Communication

As a team, we had a brainstorm meetup in which some roles and lines of work were defined. Following are the most important agreements.

Data

After some preliminary research and analysis, we decided to collect data from current data science job postings. We identified several web sites that offered raw information on position, location, company, salary and skills. In the end selected Paysa (https://www.paysa.com) because it offered the most comprehesive set of variables.

Limitations

Please note that this data has been collected from a single source and relates to job postings extant on March 14, 2017. No assumption should be made for past or future data science skills or any other conclusion we might end up for this project. A different sample from the same source will likely produce different results.

Collection

We were unable to find data in a table, csv file or other structured format. In addition, the Paysa proprietors declined our request to provide sample data for the project. We attempted to scrape the data, but Paysa has designed its web pages to prevent scraping. Because we did not require a huge sample of data for this process, we cut and pasted Paysa data into a text file that we then cleaned, organized and refined. We imported that raw text data into a SQL database for permanent storage, then exported it to R to conduct our analysys. The import-export process also was conducted via R. In total, we collected 390 job postings that listed 95 overall skills. Because some of those skills were the same but named differently, we collapsed the list into 35 unique skills and a miscellaneous group of skills that appeared fewer than 10 times in the data.

We named our raw text file paysa.txt and uploaded it to GitHub.

Data Preparation

Once we had our paysa.txt file with our desired information, we proceeded to extract valuable information from it and created a data frame. The code was shared among us to continue further cleaning and tidying.

url <- "https://raw.githubusercontent.com/dilipganesan/D607-Group-Project/patch-1/scripts/paysa.txt"

mystring = read_file(url, locale=default_locale())

## Data Scientist Job Results, showing 6K recent job postings
## Update Saved SearchNotification Settings
## 0%
## MATCH
## Head of SBG Data Science Engineering Logo
## Head of SBG Data Science Engineering
## Intuit
## EXPECTED
## $338K
## MARKET SALARY
## Base Salary$253K
## Annual Bonus$86K
## Signing Bonus$24K
## Annual Equity$0
## APPLY NOW
## Head of SBG Data Science Engineering at Intuit in Mountain View, CA
## How you match this job:
## Add more skills  to see how you match this job
## You can learn valuable new skills like: Distributed Systems, Big Data, Algorithms, Data Science, Strategy, Databases and more.
##   Jobs at Intuit      Jobs in Mountain View, CA
## 0%
## MATCH
## Principal Lead Data Scientist Logo
## Principal Lead Data Scientist
## Akamai Technologies
## EXPECTED
## $317K
## MARKET SALARY
## Base Salary$204K
## Annual Bonus$27K
## Annual Equity$86K
## Signing B

MySQL

Next, we stored this initial data set on a remote MySQL server. We provided clear instructions for all team members about how to read tables into R.

The remote server setup is at MySQL URL mydvtech.com. We chose a commercial site because in our day-to-day work we will be reading and storing company data in company portals. This was good practice or all. Step-by-step instructions for connecting using MySQL server and cPanel were created in a manual distributed to the team.

http://rpubs.com/dvillalobos/confMySQLcPanel

writeMySQLTable <- function(my.data = NULL, myLocalTableName = NULL){
  
  # Creating a schema if it doesn't exist by employing RMySQL() in R
  
  mydbconnection <- dbConnect(MySQL(), 
                  user = myLocalUser,
                  password = myLocalPassword,
                  host = myLocalHost)
  
  MySQLcode <- paste0("CREATE SCHEMA IF NOT EXISTS ",myLocalMySQLSchema,";",sep="")
  dbSendQuery(mydbconnection, MySQLcode)

  # Write our data frame into MySQL
  mydbconnection <- dbConnect(MySQL(), 
                  user = myLocalUser,
                  password = myLocalPassword,
                  host = myLocalHost,
                  dbname = myLocalMySQLSchema)
  
  myLocalTableName <- tolower(myLocalTableName)
  MySQLcode <- paste0("DROP TABLE IF EXISTS ",myLocalTableName,";",sep="")
  
  dbSendQuery(mydbconnection, MySQLcode)
  dbWriteTable(mydbconnection, name= myLocalTableName , value= my.data) 

  # Closing connection with local Schema
  dbDisconnect(mydbconnection)

  # Close all other open connections we might have
  lapply( dbListConnections( dbDriver( drv = "MySQL")), dbDisconnect)
}

readMySQLTable <- function(myLocalTableName = NULL){
  
  # Connecting to a schema by employing RMySQL() in R
  mydbconnection <- dbConnect(MySQL(), 
                  user = myLocalUser,
                  password = myLocalPassword,
                  host = myLocalHost,
                  dbname = myLocalMySQLSchema)

  # Check to see if our table exists? and read our data
  myLocalTableName <- tolower(myLocalTableName)
  if (dbExistsTable(mydbconnection, name = myLocalTableName)  == TRUE){
    slookup <- dbReadTable(mydbconnection, name = myLocalTableName)
  }

  # Closing connection with local Schema
  dbDisconnect(mydbconnection)

  #To close all open connections
  lapply( dbListConnections( dbDriver( drv = "MySQL")), dbDisconnect)
  
  return(slookup)
}

Tidying and transformation

We divided our tidying and transformation into several tasks as follows..

New cleaned table results.

Continuing with our clean up, we created new variable names, fixed dollar values and separated city and state.

Tidy table

Below is our tidy table with a total of 2217 skills. (In a subsequent step, we collapse repetitive skills (i.e., “C” and “C++”.)

Skills

From our gathered data we have a total of 72 skills.

Top 10 most desired skills by employers, by count of raw skill names.

Compensation (Exploratory Analysis)

The Paysa data lists Base Salary, Annual Bonus, Signing Bonus and a total called Expected Salary, providing a way to another way to measure value beyond raw counts.

Tree Analysis

Another way of Analysing the above data is by performing a tree analysis. This tree shows Total Salary for jobs associated with Machine Learning skills in Washington and California.

Following are the highest-paid jobs in the data.

Total Salary

Table

Table: Companies and job offerings with the top 10 Total Salaries

Position	Company	City	State	Type	Salary	Rank
Head of SBG Data Science Engineering	Intuit	Mountain View	CA	Expected Salary	338000	1
Principal Data Science Manager	Microsoft	Redmond	WA	Expected Salary	322000	2
Principal Lead Data Scientist	Akamai Technologies	Santa Clara	CA	Expected Salary	317000	3
Data Science and Analytics Lead, Global Revenue Acceleration	Google	Mountain View	CA	Expected Salary	305000	4
Director, Data Scientist	Dropbox	San Francisco	CA	Expected Salary	289000	5
Principal Data Scientist	Microsoft	Bellevue	WA	Expected Salary	289000	6
Principal Data Scientist	Microsoft	Redmond	WA	Expected Salary	287000	7
Director	Unknown	Vienna	VA	Expected Salary	285000	8
Sr. Principal Data Scientist (A/B Platform)	Coupang	Palo Alto	CA	Expected Salary	281000	9
Manager of Data Science	Yelp	San Francisco	CA	Expected Salary	278000	10

Chart

Top paid skills

The below combination of desired skills by employers will generate the top Total Salary of $338000.

Table: Top paid skills by top paid Total Salaries

Top Paid Skills	Listed in top most desired skills by employers
Distributed Systems	FALSE
Big Data	TRUE
Algorithms	TRUE
Data Science	TRUE
Strategy	FALSE
Databases	FALSE

The below tree will present the top two highest-paid salaries and respective skills.

Signing Bonus

Table

Table: Companies and job offerings with the top 10 Signing Bonus

Position	Company	City	State	Type	Salary	Rank
Principal Data Science Engineer	OpenTable	San Francisco	CA	Signing Salary	43000	1
Data Scientist, Population and Survey Sciences	Facebook	Menlo Park	CA	Signing Salary	42000	2
Data Scientist, Infrastructure	Facebook	Menlo Park	CA	Signing Salary	42000	3
Data Scientist- Consumer Insights	Facebook	Menlo Park	CA	Signing Salary	42000	4
Data Visualization Scientist	Facebook	Menlo Park	CA	Signing Salary	42000	5
Data Scientist	Facebook	Menlo Park	CA	Signing Salary	42000	6
Data Scientist, Auction & Delivery	Facebook	Menlo Park	CA	Signing Salary	42000	7
Software Development Mgr - Advertising Analytics and Data Science	Amazon	Seattle	WA	Signing Salary	40000	8
Infrastructure Data Scientist & Strategy Analyst	Facebook	Menlo Park	CA	Signing Salary	40000	9
Senior Data Scientist	Amazon	Seattle	WA	Signing Salary	39000	10

Chart

Top paid skills

The below combination of desired skills by employers will generate the top Signing Bonus of $43000.

Table: Top paid skills by top paid Signing Bonus

Top Paid Skills	Listed in top most desired skills by employers
Product Management	FALSE
Hadoop	TRUE
Software Design	FALSE
Information Retrieval	FALSE
Machine Learning	TRUE
ETL	FALSE

The below tree will present the top two highest-paid signing bonus and respective skills.

Annual Bonus

Table

Table: Companies and job offerings with the top 10 Annual Bonus

Position	Company	City	State	Type	Salary	Rank
Head of SBG Data Science Engineering	Intuit	Mountain View	CA	Annual Salary	86000	1
Principal Data Science Manager	Microsoft	Redmond	WA	Annual Salary	59000	2
Principal Data Scientist	Microsoft	Redmond	WA	Annual Salary	53000	3
Principal Data Scientist	Microsoft	Bellevue	WA	Annual Salary	49000	4
Principal Data Scientist Architect	SAP	Palo Alto	CA	Annual Salary	49000	5
Data Science and Analytics Lead, Global Revenue Acceleration	Google	Mountain View	CA	Annual Salary	48000	6
Principal Data Science Engineer	OpenTable	San Francisco	CA	Annual Salary	46000	7
Legal- Firmwide Initiatives Team OLO Data Scientist- VP	JPMorgan Chase	New York	NY	Annual Salary	46000	8
CIB-Rapid Prototyping Data Scientist	Unknown	New York	NY	Annual Salary	46000	9
Director	Unknown	Vienna	VA	Annual Salary	45000	10

Chart

Top paid skills

The below combination of desired skills by employers will generate the top Annual Bonus of $86000.

Table: Top paid skills by top paid Annual Bonus

Top Paid Skills	Listed in top most desired skills by employers
Distributed Systems	FALSE
Big Data	TRUE
Algorithms	TRUE
Data Science	TRUE
Strategy	FALSE
Databases	FALSE

The below tree will present the top two highest paid salaries and respective skills.

Base Salary

Table

Table: Companies and job offerings with the top 10 Base Salaries

Position	Company	City	State	Type	Salary	Rank
Head of Data Science and Engineering	Amazon	Seattle	WA	Base Salary	265000	1
Head of SBG Data Science Engineering	Intuit	Mountain View	CA	Base Salary	253000	2
Director	Unknown	Vienna	VA	Base Salary	240000	3
Principal Lead Data Scientist	Akamai Technologies	Santa Clara	CA	Base Salary	204000	4
Head of Data Science, Liquidity	First Republic Bank	San Francisco	CA	Base Salary	202000	5
Principal Data Scientist	Microsoft	Bellevue	WA	Base Salary	200000	6
Chief Data Scientist, Brilliant Manufacturing Job	GE	Seattle	WA	Base Salary	199000	7
Principal Data Scientist	Microsoft	Redmond	WA	Base Salary	198000	8
Corporate - Firmwide Forecasting & Analysis - Data Scientist/Engineer, Vice President	JPMorgan Chase	New York	NY	Base Salary	196000	9
Data Scientist, State Street Global Exchange, Vice President	State Street Corporation	New York	NY	Base Salary	195000	10

Chart

Top paid skills

The below combination of desired skills by employers will generate the top Base Salary of $265000.

Table: Top paid skills by top paid Base Salaries

Top Paid Skills	Listed in top most desired skills by employers
Product Management	FALSE
Machine Learning	TRUE
Data Science	TRUE
Analytics	FALSE
Statistics	FALSE

The below tree will present the top two highest-paid base salaries and respective skills.

Maps

Highest-Valued Skills Measured by Mean Compensation

Some of the highest-valued skills are not the most common skills. They include Strategy, Leadership, Management and Data Science, a catch-all. ETL – for Extract, Transfer and Load – is a critical area of data warehousing.

This part of the analysis looks at the value of skills based on what employers pay rather the frequency of skills in a job posting. To do this, we compute a mean value for each skill across the database.

For example, the job ‘Principle Lead Data Scientist’ at Akamai is associated with six skills: Hadoop, Data Mining, Machine Learning, Python, Matlab and Ruby. The total compensation this job is $317,000. To value each job, we divide total compensation by 6 to get $52,833. We do similar computation for skills in each job, then calculate the overall mean of those values across all jobs for each skill. We then plot those values to rank skills in descending order.

Relative Value of Skills

Using ANOVA, we can compute how much a particular skill adds or subtracts from the mean Expected Salary for Algorithms, the reference level, all other things equal.

For example, the reference mean compensation for Algorithms is $30,323. Having ETL skills adds $9,283 to that mean; Matlab skills are worth $1,282 less.

The chart summarizes the skill values relative to the Algorithm baseline.

The below chart, display a salary weight composition for each desired skills by employers.