We’ll explore sentiment analysis for a 2022 IT Service Desk data set containing 6900 IT support tickets/requests.
…And my real-world attempt to address a critical function of that IT Service Desk.
2022-11-19
We’ll explore sentiment analysis for a 2022 IT Service Desk data set containing 6900 IT support tickets/requests.
…And my real-world attempt to address a critical function of that IT Service Desk.
I am the manager for IT support services team for faculty, staff and students at a liberal arts college in New York City.
The holy grail for a customer support organization (including IT support) is customer satisfaction.
The holy grail for worker retention is employee satisfaction.
To maximize satisfaction, every interaction matters.
Organizations work best when their customers and their employees are happy.
But happy is hard to measure.
IT support and service organizations are full of vanity metrics
Vanity metrics are metrics that make you look good to others but do not help you understand your own performance in a way that informs future strategies.
Average number of tickets assigned / resolved by an analyst
Average hold time on call
Average call time duration
Average time to resolution
The key question to identify these is So What?
Vanity metrics don’t offer much value, don’t lead to decisions or actions & incentivize wrong behaviors in search of hitting the metric.
Customer and employee satisfaction surveys are a common way to try to assess customer and employee happiness.
“How did we do?”
“Rate us a 5”
These surveys tend to suffer from all sorts of response biases.
In the process of exploring sentiment analysis as an alternative to this method.
Up until now, we had been focused on improving the survey mechanism.
Sentiment analysis allows for a more detailed understanding on ‘sentiment’ at a per ticket/per customer / per employee level.
The analysis can be performed across the entire data set - i.e., all tickets in the dataset.
The analyst can be done in with tickets that are still open or recently closed.
It is from a commercial / enterprise Service Desk ticketing system
It contains all tickets opened from Jan 1 2022 - Nov 4, 2022
Names have been changed to protect the innocent
For reasons confidentiality and data sensitivity, the dialogue columns have been removed from the dataset before a clean dataset was published to github. Only the sentence sentiment scores and word counts are included.
Before the dialogue columns were removed, sentimentr allows for a profanity check
Abusive language violates the College’s code of conduct and HR employee policies.
In my data set, no profanities was discovered when evaluating individual sentences.
## Rows: 119,735 ## Columns: 14 ## $ number <chr> "INC0120526", "INC0120526", "INC0120526", "INC01205… ## $ assigned_to <chr> "John", "John", "John", "John", "John", "John", "Jo… ## $ contact_type <chr> "Self-service", "Self-service", "Self-service", "Se… ## $ assignment_group <chr> "Service Desk", "Service Desk", "Service Desk", "Se… ## $ u_subcategory <chr> "Computer (Desktop/Laptop)", "Computer (Desktop/Lap… ## $ opened_at <chr> "10/21/22 17:03", "10/21/22 17:03", "10/21/22 17:03… ## $ sys_mod_count <int> 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,… ## $ reopen_count <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, … ## $ reassignment_count <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, … ## $ resolved_at <chr> "11/1/22 18:00", "11/1/22 18:00", "11/1/22 18:00", … ## $ element_id <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, … ## $ sentence_id <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, … ## $ word_count <int> 6, 10, 4, 12, 5, 17, 12, 9, 20, 11, 10, 1, 12, 1, 5… ## $ sentiment <dbl> 0.34701105, 0.00000000, 0.25000000, 0.07216878, 0.3…
Is it Nora? Is it the customer? Is the technology?
(When joined to the original data), a list of actionable tickets for followup.
By Analyst / IT Group
By Customer / Customer Group (Projects, Special Initiatives)
By Academic or Administrative Dept (HR, History, etc)
By Support Category (Hardware, Software, Password, etc)
By Seasonality/Time (Term Start, Midterms, Finals, etc)
At least in this use case, sentiment analysis is a superior way to identify areas for improvement – but further vetting is needed.
Domain expertise is important to identify areas of focus which may warrant further analysis & which areas do not.
Sentiment analysis allowed us to detect negative sentiments over shorter intervals of time and while they were occurring.