2022-11-19

Applied Sentiment Analysis: IT Support Tickets

We’ll explore sentiment analysis for a 2022 IT Service Desk data set containing 6900 IT support tickets/requests.

…And my real-world attempt to address a critical function of that IT Service Desk.

Background and Context

I am the manager for IT support services team for faculty, staff and students at a liberal arts college in New York City.


  • 1000 faculty and administrative staff
  • 2600 undergrad students
  • Generate any where from 8,000-10,000 IT service requests per year
  • The data set we are looking at today is from my college / job

What makes for a good IT Service Desk and a good organization for its employees?

  • The holy grail for a customer support organization (including IT support) is customer satisfaction.

  • The holy grail for worker retention is employee satisfaction.

  • To maximize satisfaction, every interaction matters.


Organizations work best when their customers and their employees are happy.

But happy is hard to measure.

Conventional Ways to Measure IT Service Desk performance

IT support and service organizations are full of vanity metrics

Vanity metrics are metrics that make you look good to others but do not help you understand your own performance in a way that informs future strategies.

  • Average number of tickets assigned / resolved by an analyst

  • Average hold time on call

  • Average call time duration

  • Average time to resolution


The key question to identify these is So What?

To illustrate the point…

Vanity metrics don’t offer much value, don’t lead to decisions or actions & incentivize wrong behaviors in search of hitting the metric.


Customer and Employee Satisfaction Surveys

Customer and employee satisfaction surveys are a common way to try to assess customer and employee happiness.

“How did we do?”

“Rate us a 5”

But…

These surveys tend to suffer from all sorts of response biases.

Enter Sentiment Analysis

  • In the process of exploring sentiment analysis as an alternative to this method.

  • Up until now, we had been focused on improving the survey mechanism.

  • Sentiment analysis allows for a more detailed understanding on ‘sentiment’ at a per ticket/per customer / per employee level.

  • The analysis can be performed across the entire data set - i.e., all tickets in the dataset.

  • The analyst can be done in with tickets that are still open or recently closed.

A few words about the data set..

  • It is from a commercial / enterprise Service Desk ticketing system

  • It contains all tickets opened from Jan 1 2022 - Nov 4, 2022

  • Names have been changed to protect the innocent

  • For reasons confidentiality and data sensitivity, the dialogue columns have been removed from the dataset before a clean dataset was published to github. Only the sentence sentiment scores and word counts are included.

A few words about the Sentiment Analysis - SentimentR

The Customer & Analyst Dialogue

Original Source Data

  1. The original data set was imported from a local file.
  2. The dialogue columns were removed.

Profanity Check

Before the dialogue columns were removed, sentimentr allows for a profanity check

Profanity Results

Abusive language violates the College’s code of conduct and HR employee policies.

In my data set, no profanities was discovered when evaluating individual sentences.

Exploring IT Support Sentiment

## Rows: 119,735
## Columns: 14
## $ number             <chr> "INC0120526", "INC0120526", "INC0120526", "INC01205…
## $ assigned_to        <chr> "John", "John", "John", "John", "John", "John", "Jo…
## $ contact_type       <chr> "Self-service", "Self-service", "Self-service", "Se…
## $ assignment_group   <chr> "Service Desk", "Service Desk", "Service Desk", "Se…
## $ u_subcategory      <chr> "Computer (Desktop/Laptop)", "Computer (Desktop/Lap…
## $ opened_at          <chr> "10/21/22 17:03", "10/21/22 17:03", "10/21/22 17:03…
## $ sys_mod_count      <int> 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,…
## $ reopen_count       <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, …
## $ reassignment_count <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …
## $ resolved_at        <chr> "11/1/22 18:00", "11/1/22 18:00", "11/1/22 18:00", …
## $ element_id         <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, …
## $ sentence_id        <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, …
## $ word_count         <int> 6, 10, 4, 12, 5, 17, 12, 9, 20, 11, 10, 1, 12, 1, 5…
## $ sentiment          <dbl> 0.34701105, 0.00000000, 0.25000000, 0.07216878, 0.3…

Distribution of Sentiment Across the IT Organization: Overall Positive

Distribution of Sentiment by IT Group: Overall Positive

Sentiment Range: A Rollercoaster

Opportunity: Neutral Sentiment

Sentiment for Service Desk Analysts

Nora’s Negative Sentiment Tickets

Is it Nora? Is it the customer? Is the technology?

(When joined to the original data), a list of actionable tickets for followup.


Sentiment Analysis Opens Up Avenues of Exploration

  • By Analyst / IT Group

  • By Customer / Customer Group (Projects, Special Initiatives)

  • By Academic or Administrative Dept (HR, History, etc)

  • By Support Category (Hardware, Software, Password, etc)

  • By Seasonality/Time (Term Start, Midterms, Finals, etc)

Conclusions (and Takeaways from my experience)

  • At least in this use case, sentiment analysis is a superior way to identify areas for improvement – but further vetting is needed.

  • Domain expertise is important to identify areas of focus which may warrant further analysis & which areas do not.

  • Sentiment analysis allowed us to detect negative sentiments over shorter intervals of time and while they were occurring.