Introduction

The purpose of this project was to analyze job ads related to program evaluation in order to understand the representation of evaluation skills, based on keywords, as they are reflected on the job market.

This analysis examines a small snapshot of the program evaluation job market. It uses both quantiative and qualitative content analysis to understand current, in-demand skills for program evaluators.

The Many Meanings of Evaluation

In our analysis of “evaluation jobs,” it became clear that evaluation has multiple meanings. In particular, there are three distinct domains of “evaluation”:

  • Program Evaluation - The focus of this project was to search for jobs related to professional program evaluation, which can be defined as a systematic “exploration of a program’s merits, including its effectiveness, quality, and value” (Fink, 2015, p. 20).
    • Closely related to higher education program evaluation are jobs in assessment, which is a systematic approach to assessing and evaluating student learning or a related phenomenon.
  • Monitoring and Evaluation (M&E) - An overwhelming amount of job advertisements on EvaluationJobs.org are for monitoring and evaluation jobs. This type of evcaluation is concerned with evaluating the performance of international development and government projects. While these jobs may include some aspects of program evaluation, such as data collection and analysis or other evaluation methods, they also rely heavily on assessing finances, budgets, supplies, equipment, personnel and other areas unrelated to program evaluation. For more information, please see the World Health Organization’s description of M&E jobs
  • Academic Evaluation - These jobs are typically housed in a higher education institution’s registrar office and are concerned primarily with academic transcript evaluation.

Methodology

The following analysis of program evaluation job ads looks at a random sample of purposefully selected advertisements from two websites:

Web Scraping

This project utilized two webscraping packages in R. Rselenium was used to scrape job ads from EvaluationJobs.org. The Rselenium package was used because EvaluationJobs.org loads its jobs dynamically via javascript, which typical webscraping tools cannot process. RSelenium acts as a virtual browser, loading all dynamic elements before scraping begins. rvest was then used to process text from the scraped pages.

HigherEdJobs.org has a very strict security policy that does not allow web scraping. Therefore, job advertisements were searched for using a “Job Agent” and jobs relevant to program evaluation were manually saved. The package `rvest~ was then used to scrape the saved jobs. The search term used for the job agents was as follows: “evaluation OR assessment OR evaluator”.

After scraping all jobs, data was combined and a final job ad database was created for analysis.

Content Analysis

Job ads were initially analyzed following quantiative content analysis methods and text mining approaches using the tidytext package (see Text Mining with R). To facilitate analysis, a random sample of job ads were manually coded for keywords related to program evaluation. This analysis led to the creation of a data dictionary that contained:

  • Categories: These represent the broad roles that evaluators undertake:
    • Planning - definition
    • Analyzing - definition
      • Software skills - These are related to analyzing skills are refer specifically to the software and programming languages preferred or required in jobs ads.
    • Data Collecting - definition
    • Reporting - definition
  • Subcategories: These represent the major skill areas under each category. For example, the role of planning requires project management, evaluation design, and often training of others.
  • Skills: These are the specific skills each skill area requires. For example, project management may require skills such as negotiation, management, or supervision

The quantiative content analysis compared job ads against this data dictionary in order to track the frequency of categories and skills in the job ad database.

Job Ads in Analysis

Number of Jobs Analyzed
95
source n percent
HigherEdJobs.org 52 54.74%
EvaluationJobs.org 43 45.26%
Total 95 -

Skills

Most Common Skills

Category Proportion of Ads Number of Jobs Percent of Jobs
Planning 23% 95 100%
Reporting 23% 94 99%
Analyzing 22% 90 95%
Software 21% 88 93%
Data Collecting 11% 46 48%

Planning Skills

Overall Representation by Sub Category

Representation by specific skill

Analysis Skills

Representation by Specific Skill

Data Collection Skills

Representation by Specific Skill

Reporting Skills

Representation by Specific Skill

Software Skills

**********************

Degree

Qualitative Analysis

tbd

Key Takeaways

tbd

Job Ad Explorer

Note 1: Older jobs may be expired and no longer have valid job pages on their source websites. These links will return 404 error or similar pages.

Note 2: Any adds with strange formatting (javascript, HTML) are from EvaluationJobs.org, which procures ads by webscraping and sometimes includes errant code in their page.