The idea is to use “#firstsevenjobs” and “#first7jobs” tweets as example of text analysis, starting from querying Twitter API with the rtweet package, then cleaning the tweets a bit, and then using the Monkeylearn package to classify the jobs in an industry.
“#firstsevenjobs” and “#first7jobs” tweets initial goal was to provide a short description of the 7 first activities they were paid for. Of course the hashtag was also used by spammers, for making jokes, and for advertising for analyses in R, so not all the tweets contain the 7 descriptions. However, we are confident quite a lot of them do.
library("rtweet")
library("dplyr")
library("monkeylearn")
library("stringr")If following the instructions given in the repository of the rtweet package, one does not have to re-input their own tokens each time.
first7jobs <- search_tweets(q = "#firstsevenjobs", count = 1000, type = "recent")
first7jobs <- rbind(first7jobs,
search_tweets(q = "firstsevenjobs", count = 1000, type = "recent"))
first7jobs <- unique(first7jobs)
first7jobs <- filter(first7jobs, lang == "en")
first7jobs## # A tibble: 740 x 26
## created_at status_id retweet_count favorite_count
## <time> <chr> <int> <int>
## 1 <NA> 763039745026818048 0 0
## 2 <NA> 763039713188077568 0 0
## 3 <NA> 763039642887200768 0 0
## 4 <NA> 763039575191068673 1 0
## 5 <NA> 763039484141244418 0 0
## 6 <NA> 763039417669918720 0 0
## 7 <NA> 763039417221124096 0 0
## 8 <NA> 763039405263257600 0 0
## 9 <NA> 763039381863198721 0 0
## 10 <NA> 763039354033991681 916 0
## # ... with 730 more rows, and 22 more variables: text <chr>,
## # in_reply_to_status_id <chr>, in_reply_to_user_id <chr>,
## # is_quote_status <lgl>, quoted_status_id <chr>, lang <chr>,
## # user_id <chr>, user_mentions <list>, hashtags <list>, urls <list>,
## # is_retweet <lgl>, retweet_status_id <chr>, place_name <chr>,
## # country <chr>, long1 <dbl>, long2 <dbl>, long3 <dbl>, long4 <dbl>,
## # lat1 <dbl>, lat2 <dbl>, lat3 <dbl>, lat4 <dbl>The goal is to get jobs on their own, which is hard since depending on the tweets the separation was done differently. Moreover with this code I don’t totally get rid of 1) spam tweets 2) parts of the “#firstsevenjobs” tweet that are not job descriptions.
The parsing is tricky because each tweet can have different separations between description, e.g. use back slashes, or use commas and back slashes for a mixed job description (waitress/dishwasher, second job, third job, etc.) so I’ll try to guess which character is the separator by counting the number of back slashes, commas, semi commas, etc. and selecting the most present one in each tweet.
If someone used spaces, then I won’t be able to parse the tweet. Hopefully it was a rare occurence (it’s quite hard to read the tweet as an human being when there are only spaces as separators!).
separators <- c("\\?", "\\.", "\\!", "\\;",
"\\,", "\\\n", "\\/", "[1-7]")
first7jobs <- first7jobs %>%
select(status_id, text) %>%
group_by(status_id) %>%
# count the potential separators
mutate(no_interrogation = str_count(text, "\\?")) %>%
mutate(no_point = str_count(text, "\\.")) %>%
mutate(no_exclamation = str_count(text, "\\!")) %>%
mutate(no_semicolumn = str_count(text, "\\;")) %>%
mutate(no_comma = str_count(text, "\\,")) %>%
mutate(no_newline = str_count(text, "\\\n")) %>%
mutate(no_backslash = str_count(text, "\\/")) %>%
mutate(no_number = str_count(text, "[1-7]")) %>%
mutate(separator = separators[order(c(no_interrogation,
no_point,
no_exclamation,
no_semicolumn,
no_comma,
no_newline,
no_backslash,
no_number),
decreasing = TRUE)[1]]) %>%
# no numbers
mutate(text = gsub("[1-9].", "", text)) %>%
# no parenthesis
mutate(text = gsub("[\\(\\)]", "", text)) %>%
# no username
mutate(text = gsub("\\@.*", "", text)) %>%
# no RT part
filter(!grepl("RT \\@", text)) %>%
# don't keep the polluting hashtags
filter(!grepl("\\#NameofFirstPet", text)) %>%
mutate(text = gsub("\\#firstsevenjobs", "", text)) %>%
mutate(text = gsub("\\#firstobs", "", text)) %>%
filter(!grepl("\\#MothersMaidenName", text)) %>%
filter(!grepl("\\#HighSchoolMascot", text)) %>%
# no polluting sentence "Can we get these trending too"
filter(!grepl("Can we get these trending too", text)) %>%
# no hyphen
mutate(text = gsub("-", " ", text)) %>%
mutate(text = gsub(" ", " ", text)) %>%
mutate(text = gsub("- ", "", text)) %>%
# no empty
filter(text != "") %>%
filter(text != " ") %>%
filter(text != "RT") %>%
filter(text != "RT ") %>%
# split by these in order to separate the jobs
do(str_split(.$text, .$separator) %>%
unlist %>%
data_frame(wordsgroup = .)) %>%
# no empty
filter(wordsgroup != "") %>%
filter(wordsgroup != " ") %>%
filter(wordsgroup != "RT") %>%
filter(wordsgroup != "RT ") %>%
mutate(rank = 1:n()) %>%
# remove remaining new lines
mutate(wordsgroup = gsub("\\\n","",wordsgroup)) %>%
ungroup()
first7jobs## # A tibble: 2,234 x 3
## status_id wordsgroup rank
## <chr> <chr> <int>
## 1 763011044423892992 Later also at electronic warehouse 1
## 2 763011044423892992 floor factory 2
## 3 763011044423892992 glass factory 3
## 4 763011044423892992 loading of trucks 4
## 5 763011044423892992 telemarketer 5
## 6 763011044423892992 logistics 6
## 7 763011044423892992 plastic factory 7
## 8 763011084588576768 Ice cream scooper 1
## 9 763011084588576768 Retail sales associate 2
## 10 763011084588576768 intern 3
## # ... with 2,224 more rowsWe use Monkeylearn’s Job Roles Classifier.
The results are not perfect, but often we recognize the industry we would have chosen.
request <- c(first7jobs$wordsgroup)
# classify with Job Roles Classifier
# https://app.monkeylearn.com/main/classifiers/cl_i7vMzUB7/
output <- monkeylearn_classify(request,
classifier_id = "cl_i7vMzUB7")
dim(output)## [1] 2234 4# Output has same length as request, so here no need to join by MD5.
output_with_words <- cbind(output, first7jobs)
# filter high probabilities
output_with_words_high <- filter(output_with_words,
probability > 0.75)
# arrange by probability
output_with_words_high <- arrange(output_with_words_high,
desc(probability))
output_with_words_high %>% head() %>% knitr::kable()| category_id | probability | label | text_md5 | status_id | wordsgroup | rank |
|---|---|---|---|---|---|---|
| 1495359 | 1 | Facilities / General Labor | 7823a5836b82ba06a8112c2d725a7cac | 763011044423892992 | Later also at electronic warehouse | 1 |
| 1495365 | 1 | Manufacturing / Production / Construction / Logistics | ee35f4684dd4229717917228da888f2d | 763011044423892992 | logistics | 6 |
| 1495373 | 1 | Retail | 4f638f5213d65b363ea16dbc851093e2 | 763011084588576768 | Retail sales associate | 2 |
| 1495355 | 1 | Banking / Loan / Insurance | 392b3b6e70663b9e03f0b662508ee5ad | 763011084588576768 | investment banker | 5 |
| 1495359 | 1 | Facilities / General Labor | f845a8195a686c7d620b201c657f6f92 | 763011139764523009 | Vacuum Cleaner Salesman | 3 |
| 1495356 | 1 | Software Development / IT | dc25c8a747ec5343b80ba0d3d598ef77 | 763011139764523009 | Data Center Engineer | 6 |
How are each label represented in the data?
# sum by label
summary_label <- output_with_words_high %>%
group_by(label) %>%
summarize(sum = n(),
median_rank = median(rank),
words = toString(sort(unique(wordsgroup)))) %>%
arrange(desc(sum))
summary_label %>% knitr::kable()| label | sum | median_rank | words |
|---|---|---|---|
| Restaurant / Food Services | 128 | 3.0 | A/R clerk #FirstSevenjobs, baker, Baker’s Burgers, bakery clerk , Banquet server, barista, Barista, bartender, boh food service, clerk, Cook, cashier, drive thru at Wendy’s w/Tejhra, Deidre Pilot Shift manager, photo tech at, Dalton Bookstore clerk Ranch …, deli, deli clerk, dishwasher, Dishwasher, fast food counter help, fast food worker, file clerk, Food Prep, fry cook, invoice clerk, kings bakery, pizza baker and shop assistant., pizza chef, pizza deliverer, Pizza maker, prep cook, restaurant hostess, restaurant manager, Royal food taster, salad cook, server, wawa clerk, " cafe cook, #FirstSevenJobs = babysitter + baker, #FirstSevenJobs Dishwasher, ; Barista; Spec, |
| Retail | 104 | 3.0 | Cashier, Lawnboy, Grocery bagger, Landscaper, https://t, Church daycare Movie theater Referee Retail store Rape crisis counselor Waitress Production assistant, babysitter deli cashier , book store clerk, camera store salesman, candy store, cashier, Cashier, Cashier , cashier/ sandwich maker, convenience store, convenience store clerk, convience store clerk, department store cashier and bridal coordinator, grocery cashier, grocery clerk, grocery store cashier, ice cream store cleaner, music store clerk, old navy store associate, panera cashier at local strip mall, pet store clerk, Pizza Ranch Grocery Store Horrible Bartender Golf Coach Art Museum Concordia Events Landmark Center, retail, Retail, Retail Associate, retail at #WDW, retail intern, retail sales, retail sales assoc, Retail sales associate, retailer, schnuck’s cashier, sox world cashier, store cashier, Store clerk, Store Hand, strite ride cashier & promoter, Target cashier, toy store, #FirstSevenJobs Cashier , Babysitter Camp Counselor UConn Tour Guide Fabric Store Sales Mascotopia MacLaren Strollers Nanny #thenteacher, Carpark Cashier, cashier, Cashier, cashier , Cashier , cashier made it shifts, cashier pool, cashier store, Cashier/ hostess, caving into peer pressure: cashier, clothing store clerk, , College Computer Store, Computer store clerk, delia’s cashier, department store clerk, DollarTree Cashier, drug store clerk, entrepreneur + dry cleaning cashier + interlibrary loan assist. + tutor + temp, fastfood cashier, groceries go getter, Grocery bag boy, grocery cashier, Grocery cashier, grocery shop assistant, Grocery store cashier, Hardware store, hardware store cashier, KFC Cashier, McDonald’s Cashier, McDonalds cashier, menswear store, More outdoor retail, Outdoor retail, Pick N Save cashier, pizza cashier, PM janitor at convenience store, retail, Retail Associate, Retail store asst., Shop Rite cashier, sign store asst, , Store assistant, Store clerk, Store Clerk, store sales clerk, Video Store Clerk, Walmart Cashier |
| Art/Design / Entertainment | 64 | 5.0 | “sandwich artist, ad designer, artist, arts&crafts assistant, designer cookie maker, fashion designer, graphic aide, graphic artist’s secretary, graphic designer , makeup artist, Newspaper boy Fruit & Veg seller, Shelf stacker, Designer, Teacher, Art Director, Creative Director, painter, Photographer, produce deliverer, retail product design, sandwich artist, Waitress Intern CBS Designer IF Studios Freelance Designer NBC Designer HBO, Creative Director E! , WMVP Producer, |
| Education | 60 | 5.0 | box office slave kids theater director hot topic ulta flute teacher camp counselor box office |
| Sales / Customer Care | 55 | 5.0 | Door door sales Shop assistant Waiter Cafe manager Army Officer Strategy consultant Cofounder & CEO, Telemarketer TeleSales Advertising Sales InsideSales FieldSales IT Projects Sales Software Sales, B sales, bicycle sales, cafe service, clothing sales, cust. service, Customer Service, Customer service rep, Dot com customer svc, Email Customer Service Rep, furniture sales, janitorial sales, MTE ticket sales Renforth Pharmacy Princess of Wales Theatre Usher Pantages Theatre Box Office Princess of W…, office supply sales, pharma sales. , sales, sales intern, Sales Trainer, sales/men’s clothes, sales/women’s clothes, Cell phone sales, Clothing Sales, Customer service clerk, customer service rep, Customer Service Rep, Digital sales exec., Direct Sales , Fashion sales assistant, hardware sales, Home theater sales, Hot Topic Sales Associate, Ins. Agency Adult, Internal Sales, Mag sales, Member Services, Phone sales consultant, Plastic Sales, Printer Ribbon Sales Telemarketing, Sales, Sales & advertisement , Sales Assistant, Sales assistant clothes, Sales AsstCustomer Serv AsstPromoterWaitressHome TutorProject Coorhttps://t.co/yDYOLyJm Asst Manager , Sales in a shoeshop, Sales specialist, sales trainer, Ski ticket sales, Student media ad sales, Support representative, Tech Sales Support, Tire sales, Totally forgot telemaketing sales &, TV Ad Sales Planning , YMCA membership sales |
| Writing / Editing / Publishing | 50 | 6.0 | asst features editor, Babysitter LittleCaesar’s cashier/cleaner/pizza maker Grocery store checker Amex clerk Writer Copy editor, Continuity Editor, editor, Editor, Editor at college lit mag, editor. , editorial assistant…, editorial asst, Freelance Writer, Radio reporter, reporter, Researcher/Editor , rootkit writer , Staff reporter, Sub Editor, Tech Writer, TV reporter, Writer, writer. Wait, , Proposal Coord, , Spanish Tutor, Editorial Assist, : printing assistant, ad copywriter, But always, while in the midst of those other jobs: writer/poet/editor, copy editor, Editor, Editorial director, Editorial/writing intern, features reporter, News Writer, Newspaper Editor, Newspaper reporter, Political editor Copy chief, Printing pressman, Reporter, Sports copy editor, Technical writer, TV News reporter, Video Editor |
| Administrative | 48 | 4.5 | Administrative Assistant, administrative asst , Assist, assistant, Assistant, Assistant Program Officer, Campus Dispatcher, Comms Assistant, course assistant, daycare assistant, gym assistant, hair salon receptionist, Herbarium assistant, Learning Assist, Magician’s assistant, museum assistant, office assistant, receptionist, Receptionist at SuperCuts, Receptionist at vet’s office , receptionist., |
| Non-profit / Volunteering | 39 | 4.0 | camp counselor, Camp Counselor, campus pastor |
| Facilities / General Labor | 38 | 3.0 | Cleaner, dry cleaners, facilities, housekeeper, industrial cleaner, janitor, Janitor, Antioch Baptist Church of Corona, Later also at electronic warehouse, natural foods bagger, towmotor operator warehouse, Vacuum Cleaner Salesman, warehouse supervisor, #FirstSevenJobs: Janitor; Class, |
| Travel / Transportation | 37 | 4.0 | > mag delivery boy, advertising delivery service, Concrete crew grunt, Delivery boy, IUP grounds crew, Jimmy Johns Driver, Newspaper Delivery, Newspaper delivery boy, pizza delivery dude, Truck driver, Zamboni driver, #FirstSevenJobs HCED president Bus Boy Porter Used Car Detailer Delivery Truck Driver Purdue mailroom stock room Junk Mail PO, : Newspaper delivery boy Contra Costa Times, [CA] Burger King burger flipper B, |
| Science / Research | 34 | 5.0 | Babysitter, Retail, Research Assist, Bread Scientist, faculty research assistant, fin aid asst, lab technician, mad scientist, Receptionist/research asst., recreation aide, Research analyst, research assistant, research fellow , undergrad researcher, Aide to college veep, Digital Literacy Research Associate, Door to door mktg researcher, Hospital aid, Lab research , Laboratory assistant, Leukaemia research scientist , Organic chemist, PathfinderVillage Aid, professor, researcher, writer, Race riot researcher, research assist, research assistant, Research asst, Research chemist, Research Mgr, Research/comms intern , soc researcher, undergraduate researcher |
| Software Development / IT | 31 | 5.0 | Application Developer, Data Center Engineer, data entry, Data Entry, Data Entry Clerk, Drug store assistant, Factory worker, Data engineer, paperboy construction pizza delivery tech helpdesk warehouse software engineer engineer manager, software developer, Software Engineer #FirstSevenJobs, software qa, software qa manager, Web Developer , Data engineer, data entry, Data entry, Data Entry, Data entry , Data scientist , entrepreneur selling weather data, roofing co. data entry, Senior Analyst, Software developer, Software Developer, software engineer, Software engineer, Software engineer/Dev ops, Software tester, Student Ambassador DarabniTV Cashier T shirt Factory Artist Tamatem Web developer RedTroops , System Admin, Web Development |
| Marketing / Advertising / PR | 29 | 6.0 | church media staff, community relations coordinator, higher ed marketing , market studies, Marketing, marketing director, Marketing Intern, retail social media, Social Media Marketer, #FirstSevenJobs is creating a social media storm, , Marketing, |
| Sports / Fitness | 26 | 4.0 | ballet instructor, Hollister Barista t grade indoor soccer coach Barista Paul Mitchell hair model Hooters Paper route, : lemonade standmowing lawnshanding out flyerstutoring dooroor CalPIRGCutco salesdebate coach, |
| Manufacturing / Production / Construction / Logistics | 25 | 4.0 | assembly technician, Construction, construction work, construction worker, Construction worker, craft supplies seller, freelance video production, Hay Baler Construction Laborer Grocier IM Sports Ref Sylvan Teacher Grad TA H, logistics, plumbing parts supply clerk, So happy where I am now as a #freiherr #lifehacker living off from years of music production, #FirstSevenJobs Construction Schlotzky’s Deli Meter Reader Businessman Business, man, Assembly line worker, Assembly production assistant, Construction, Construction laborer, EDP card machine operator , Electrical supply, Hallmark plant night shift er, plant librarian, Plant stand, Production assistant , Production Asst., Server Assembly, Swimming pool construction |
| Medical / Healthcare | 23 | 6.0 | Babysitter Roguing beans Caring for elderly neighbor Nurses Aid Hotel maid Arby’s Paint Store clerk, Clinical Nurse Manager, guardian pharmacy, Medical Scribe, medical staff, Nurse, Nursing Aide, Pharmacy Tech, resident, test patient, |
| Accounting / Finance | 22 | 4.0 | accounting asst, Accounts clerk, albums buyer, Auditor, Bank clerk EFL Teacher Dispatch rider Accountant Data control Geek Go to #firststevenjobs , bill collector, Budget analyst, errand boy for mother to buy odd bawang, Financial Advisor, financial advisor to bernie madoff, financial aid office again https://t.co/tvAQXuMace, General worker Account clerk Assistant account Audit clerk CSR UT agent Self employed , In house Tax Advisor., intern at A&A tax, Mizzou financial aid, Tax Consultant, Account manager, Accountant for lawfirm, Accounting clerk, Financial sales, Free newspaper finance, Radio Buying Intern |
| Security / Law Enforcement | 21 | 4.0 | Firefighter Garbage man Steve, Radio Shack Security Engineering Intern Security Freelance Translator Localization Editor Moderator, Box Office, Investigator, office asst, Private Security Sector, security guard, treasury officer, : Concessionist; Waitress; Office Temp; Security Guard; Actor; Journalist; IT Consultant, Admissions officer, bomb disposal officer, box office staff, debt recovery officer, Hospital patient transporter, Office Temp, Prison officer, Social Security interviewer, Stocker, supermarket stocker, Walmart Stocker |
| Engineering (Non-Software) | 20 | 5.0 | R&D Intern, NASA Engineering Intern, Pfizer Cashier at local florist I loved this place!, Lawn mower Engineering intern Childcare worker Comp lab monitor Landscaper Mech Engineer Biz/Eng intern, quality assurance analyst, Co op engineer, DOT engineering, electr, engineer, Engineer, Engineering intern , Medical Device Engineer, Manager, Network admin, Network Admin, Network Engineer, Project engineer NRL, Radio Station Engineer, Railway engineer, sound engineer, Sound engineer, Tobacco field farmhand |
| Product Management / Project Management | 19 | 6.0 | proj manager, Project Manager, |
| Hospitality | 16 | 5.0 | dishwasher/host/buffet attendant, drive thru attendant, gas station attendant, Sports Talk host, Cultural Host in , Flight Attendant, Gas station attendant, Hotel bellman, Kitchen Porter, Parking Attendant, Parking Lot Attendant, Parks Canada Gate Attendant, Pump attendant, Radio Show Host, Snack bar attendant, state park attendant |
| Business Development / Consulting | 15 | 6.0 | Consultant, consultant and now entrepreneur. does that count?, none of your business., Analyst, Analyst again, Analyst x, Business leaders have been revealing their on twitter. What were yours? https:, Business Manager, Development Manager, investment analyst, Operations analyst, Our Policy Analyst’s :, Photography Business Owner, Usability analyst |
| Management | 13 | 6.0 | ad director, Chief of Unicorn Division, director, Executive Director, News director, PR director, Camp Director, Challenger Flight Director, Check out some of the DM team’s #FirstSevenJobs! You can find out more here >> https:, Chorale President, Class President, Nonprofit director, Sales Director |
| Installation / Maintenance / Repair | 10 | 3.0 | car stereo installer, Curtain installer, beach maintenance staff, Bike mechanic, Boat Inspector, car mechanic , Carpet Installer, electrician, Electrician, reception/plumbing help |
| Beauty / Wellness | 8 | 3.0 | Dog staring, Dog walker, hair stylist, tanning salon attendant, dietary aide/dishroom, dog care, Dog walker, Therapist |
| Human Resources | 8 | 6.0 | Forwarding agent Waiter Tutor Clerk HR Executive Unit Trust Consultant Entrepreneur, Advertising Coordinator, Camp coordinator, Government HR, humane society employee, Headhunter/Recruiter, Recruiter, Recruitment admin |
| Banking / Loan / Insurance | 7 | 4.0 | bank teller, Bank teller, investment banker, |
| Architecture / Drafting | 6 | 3.0 | landscape nursery, Airport Surveyor, Architecture intern/admin, Environmental Activist, landscaper, Landscaping |
| Skilled Trade | 6 | 3.0 | glass factory, stained glass artisan, apprentice carpenter, auto glass sweeper, Carpenter |
| Legal | 5 | 5.0 | Contracts Manager, law clerk to federal judge , Law clerk, Legal Secretary, State of Michigan Governors Law Office B*tch |
| Real Estate | 5 | 7.0 | Broker , Estate Manager, Given all the stuff, its important to note the REAL jobs we all have staying alive staying healthy & staying motivated, portfolio manager, Real estate appraiser |
Categories by rank
# sum by label
output_with_words_high %>%
group_by(label, rank) %>%
summarize(n = n()) %>%
ungroup() %>%
group_by(label)%>%
filter(n == max(n)) %>%
knitr::kable()| label | rank | n |
|---|---|---|
| Accounting / Finance | 7 | 5 |
| Administrative | 5 | 10 |
| Architecture / Drafting | 2 | 3 |
| Art/Design / Entertainment | 5 | 17 |
| Banking / Loan / Insurance | 5 | 2 |
| Beauty / Wellness | 3 | 4 |
| Business Development / Consulting | 7 | 5 |
| Education | 6 | 14 |
| Education | 7 | 14 |
| Engineering (Non-Software) | 1 | 4 |
| Engineering (Non-Software) | 5 | 4 |
| Engineering (Non-Software) | 7 | 4 |
| Facilities / General Labor | 1 | 10 |
| Hospitality | 3 | 4 |
| Hospitality | 5 | 4 |
| Human Resources | 6 | 3 |
| Installation / Maintenance / Repair | 3 | 7 |
| Legal | 5 | 3 |
| Management | 7 | 5 |
| Manufacturing / Production / Construction / Logistics | 1 | 6 |
| Marketing / Advertising / PR | 6 | 9 |
| Medical / Healthcare | 6 | 8 |
| Non-profit / Volunteering | 4 | 8 |
| Product Management / Project Management | 7 | 6 |
| Real Estate | 7 | 3 |
| Restaurant / Food Services | 2 | 29 |
| Retail | 2 | 25 |
| Sales / Customer Care | 5 | 15 |
| Science / Research | 5 | 9 |
| Security / Law Enforcement | 3 | 5 |
| Skilled Trade | 3 | 3 |
| Software Development / IT | 6 | 6 |
| Sports / Fitness | 4 | 7 |
| Travel / Transportation | 1 | 10 |
| Writing / Editing / Publishing | 7 | 17 |