Research Question

Which are the most valued data science skills?

Importing data

Kaggle ML and Data Science Survey, 2017

The survey data included:

1. free-response items
2. multiple choice items

A Day in a Data Scientist’s Life

Variables and their definition

16716 total Kaggle respondents 13610 males (84%) 2778 females (16%) About 5 (~4.8) times as many males as females Age peak at 25 for both males and females Median at about 30

A Day in a Data Scientist’s Life

What does a data scientist do?

A Day in a Data Scientist’s Life

Manipulating data - DS Activities

Kaggle captured data about time spent in different activities:

• Time Gathering Data
• Time Model Building
• Time Production
• Time Visualizing
• Time Finding Insights

A Day in a Data Scientist’s Life

Exploratory Data Analysis (EDA)

• 37.75% of time spent in Data Acquisition
• 19.23% on Model Building (2nd highest)
• only 10.23% in production activities

A Day in a Data Scientist’s Life

A Day in a Data Scientist’s Life

Manipulating data - Learning Platforms and Learners Sentiments

A Day in a Data Scientist’s Life

Exploratory Data Analysis (EDA)

What do Data Scientists Want to Learn?

Does survey takers’ formal education have any relationship to the Machine Learning/Data Science method he or she is most excited about learning in the next year?

Understanding what working professionals want to learn could give us insight into what skills are most valued in the field.

What do Data Scientists Want to Learn?

Variables and their definition

1. FormalEducation: Which level of formal education have you attained?
2. MLMethodNextYearSelect : Which Machine Learning/Data Science method are you most excited about learning in the next year?

These questions were asked to all participants.

What do Data Scientists Want to Learn?

Exploratory Data Analysis (EDA)

Mostly Master’s degrees followed by Bachelor’s then doctoral degrees.

What do Data Scientists Want to Learn?

Let’s look at the different Machine Learning/Data Science methods.

What do Data Scientists Want to Learn?

Distribution of Machine Learning/Data Science methods with formal education.

What do Data Scientists Want to Learn?

Data Science Methods

What are the most frequently used data science (DS) methods by those writing code in DS professions?

Do those relate to formal educational attainment?

In the previous section, we examined what data science methods learners are most excited about and working on.

In this section, we’ll look at which data science methods are the most frequently used and if that has any relationship to educational attainment.

May indicate whether or not certain methods require advanced academic training.

Data Science Methods

Variables and their definition

The variables listed below indicate how often the respondents use each of these data science methods.

One additional variable: Formal Education.

Response options were: Rarely, Sometimes, Often, Most of the time

• WorkMethodsFrequencyA/B
• WorkMethodsFrequencyAssociationRules
• WorkMethodsFrequencyBayesian
• WorkMethodsFrequencyCNNs
• WorkMethodsFrequencyCollaborativeFiltering
• WorkMethodsFrequencyCross-Validation
• WorkMethodsFrequencyDataVisualization
• WorkMethodsFrequencyDecisionTrees
• WorkMethodsFrequencyEnsembleMethods
• WorkMethodsFrequencyEvolutionaryApproaches
• WorkMethodsFrequencyGANs
• WorkMethodsFrequencyGBM
• WorkMethodsFrequencyHMMs
• WorkMethodsFrequencyKNN
• WorkMethodsFrequencyLiftAnalysis
• WorkMethodsFrequencyLogisticRegression
• WorkMethodsFrequencyMLN
• WorkMethodsFrequencyNaiveBayes
• WorkMethodsFrequencyNLP
• WorkMethodsFrequencyNeuralNetworks
• WorkMethodsFrequencyPCA
• WorkMethodsFrequencyPrescriptiveModeling
• WorkMethodsFrequencyRandomForests
• WorkMethodsFrequencyRecommenderSystems
• WorkMethodsFrequencyRNNs
• WorkMethodsFrequencySegmentation
• WorkMethodsFrequencySimulation
• WorkMethodsFrequencySVMs
• WorkMethodsFrequencyTextAnalysis
• WorkMethodsFrequencyTimeSeriesAnalysis

Data Science Methods

Manipulating data

Filtered the dataset down to only those who were:

• classified as code writers
• employed in some capacity working in data science
• endorsed at least one data science skill on the question, “At work, which data science methods do you use?

Endorsed data science methods were aggregated and plotted for frequency.

Top five given a frequency score for frequently of use.

Grouped by formal education to identify the most frequently endorsed data science methods for each group.

May help identify if those writing certain types of code and using certain data analyses are potentially benefitted by pursuing advanced education.

Data Science Methods

Exploratory Data Analysis (EDA)

Frequency with which data science methods were endorsed by a total of 7,773 respondents.

Data Science Methods

Frequency of endorsements for the data science methods

Data Science Methods

“Frequency Score” for the Top Five most endorsed data science methods.

Data Science Methods

We see that in the majority of educational attainment brackets, data visualization remains the most frequently endorsed data science method.

Data Science Methods

Data Science Methods

Data visualization, logistic regression, cross-validation, decision trees, and random forests are not only frequently endorsed methods, but methods that are essential even if only used in small ways.

It seems like across data science code writers, these methods are popular and then for individual data science code writers, they are used frequently.

• Data science methods used by code writers does not seem to differ given the educational level.
  
• Data visualization remains the most frequently endorsed data science method for the majority of educational groups.

This has important implications for students of data science in understanding that certain popular job functions are not only performed by those with advanced degrees.

Learners vs. Employed Data Scientists

Is there a difference between what Learners think are the important skills to learn…

…and what employed Data Scientists say are the skills and tools they use?

Learners vs. Employed Data Scientists

Variables and Manipulating Data - Likert Scales

105 variables in 4 likert scale categories

• Learning Platform Usefulness - learners and data scientists
  • Not Useful   Somewhat useful   Very useful
• Job Skill Importance - learners
  • Unnecessary   Nice to have   Necessary
• Work Tools Frequency - data scientists
  • Rarely   Sometimes   Often   Most of the time
• WorkMethodsFrequency - data scientists
  • Rarely   Sometimes   Often   Most of the time

Plus a few basic demographic fields

Learners vs. Employed Data Scientists

Employed Data Scientists - Demographics

30% go by “Data Scientist”

20% go by “Scientist/Researcher” or “Software Developer/Software Engineer”

Learners vs. Employed Data Scientists

Learners - Demographics

About 73% are college students

55% are “focused on learning mostly data science skills” regardless of academic status.

Learners vs. Employed Data Scientists

Learning Platform Usefulness - Learners

Learners - Top 3 ways to learn data science:

1. Courses
2. Projects
3. College

Friends are the least useful learning platform.

Learners vs. Employed Data Scientists

Learning Platform Usefulness - Employed Data Scientists

• Projects and Courses belong in the top 3, but College is in 5th place
• Much greater importance on Friends…
• 46.5% say Friends are “Very useful”
• 97.5% say Friends are “Somewhat useful” or “Very useful”

Learners vs. Employed Data Scientists

Job Skills Importance to Learners

• 63.6% say Python is “Necessary”
• 39% say Data Visualization is “Necessary”
• 35% say R is “Necessary”

Learners vs. Employed Data Scientists

Work Tools Frequency

• 75.4% use Python either “Often” or “Most of the time”
• 63.6% use R either “Often” or “Most of the time”

Learners vs. Employed Data Scientists

Work Methods Frequency

Data Visualization is at the top of the list.

• 91% use it “Often” or “Most of the time”
• 0% use it “Rarely”

Only 38.8% of Learners said Visualizations were a “Necessary” job skill to learn!

R vs. Python

The most frequently used of the programming languages are R and Python. But do those that use R recommend R or Python? And do those that use Python recommend R or Python? In other words, do those survey takers feel that others should first and foremost study the languages they themselves have taken up, or perhaps with their insight, know to suggest the language of the two they themselves did not learn?

Thus the following questions were explored:

1. What is the distribution of following programming languages Kaggle survey takers used in the past year:

   • R Only
   • Python only
   • Both Python and R
   • Neither Python nor R

2. What is the distribution of programming language recommendations by following programming languages Kaggle survey takers used in the past year:

   • Using R Only
   • Using Python only
   • Using Both Python and R
   • Using Neither Python nor R

R vs. Python

Variables and their definition

There are 2 variables used in this section of the analysis :

1. LanguageRecommendationSelect=(What programming language would you recommend a new data scientist learn first? (Select one option) - Selected Choice)

2. WorkToolsSelect= For work, which data science/analytics tools, technologies, and languages have you used in the past year? (Select all that apply) - Selected Choice

R vs. Python

Exploratory Data Analysis (EDA)

## [1] 16716   229

R vs. Python

## [1] 7955  229

R vs. Python

R vs. Python

R vs. Python

Let’s examine the above graph of LanguageRecommendationSelect

R vs. Python

## [1] 7955    5

R vs. Python

Salary Comparison for Python vs. R

Finally, true to the word “value,” considerations have to be made regarding pay. The compensation received by survey takers for their work in either R or Python needs quantification to discover which language earns a data scientist more overall and in general.

Salary Comparison for Python vs. R

Contributing Variables

Three variables were used:

• WorkToolsSelect: a “select all that apply” variable with a list of various data science tools, technologies, and languages
• CompensationAmount: a numerical value to indicate their annual pay
• CompensationCurrency: a character string that indicated currency of annual pay

There was also the variable “id” that was created for the purpose of this report, acting as a way to identify each individual survey taker, and the variable “work_tools” which was a derivative of WorkToolsSelect, breaking the lists down into their individual components.

Salary Comparison for Python vs. R

Exploration and Review of Compensations

Conclusion

On the contentious debate on which Machine Learning/Data Science methods Data Scientists are most excited about learning in the next year as the most valued Data Science Skills

• Deep Learning is the top most Machine Learning/Data Science method in all categories of formal education followed by Neural Nets except High school graduates, all others wants to learn
• Time Series Analysis as the third Machine Learning/Data Science method. High school graduates want to learn Genetic & Evolutionary Algorithms as their third choice.
• Among doctoral survey takers, Bayesian Methods is the third preference.

Conclusion

On Data Science Methods Used on the Job

• Data Visualization is a remarkably popular data science method. It is the most endorsed by nearly all education attainment levels.
• Cross validation, random forests, logistic regression, and decision trees are also heavily endorsed.
• These are not just short but required or essential tasks–not only do so many of those writing code use data visualization, but they also use it quite frequently
• The data suggest that data science methods do not differ much between formal educational attainment groups.

Conclusion

Learners vs. Employed Data Scientists

• Both Learners and employed Data Scientists agree that Courses, Projects and College are in the top three ways to learn Data Science
• Learners place much less importance on Friends for learning than employed Data Scientists do
• Learners place a higher importance on Python vs. R as compared with employed Data Scientists
• Data Visualization is the top used skill for working Data Scientists even though learners put relatively little importance on it as a Job Skill

Conclusion

On Data Science Activities

• Gathering Data is the main activity where data scientists spend most of their time followed by model building.
• Personal projects and Online Courses appear to be very useful learning platforms.

Conclusion

On the contentious debate on R vs Python as the most valued Data Science Skills

• Half of the sample uses both R and Python
  
• R only to Python only users are in 1:2 ratio
  
• More R users recommended Python than the Python users recommended R
  
• Both users recommendations in Python is more than their recommendation in R
  
• R users are more likely to have a higher base salary, but Python users have the greater potential for wage growth