Overview
This is a basic intro to automated coding in R.
Setup
We will need to download some functions from this Github Gist Repo. We can source all the functions using the devtools::source_gist function.
# install devtools
install.packages("devtools")
# source the helper R file
devtools::source_gist("https://gist.github.com/Shea-Fyffe/a28368a2011583b048d0cb6bb5fbae79",
filename = "utils.R")Now we can source all the other functions needed using the helper function source_gist_files
# source gist files
source_gist_files()Importing and Cleaning Data
First, text data must be imported into R, Then it can be formatted using format_data.
raw_data <- read.csv("data.csv", stringsAsFactors = FALSE)knitr::kable(head(raw_data))| uid | raw_text |
|---|---|
| 100A | Depending on the color, I would be very irritated. I would honestly freak out and scream and yell if it was a color I didn’t like especially. If it was a color I like, I still would be upset because I wasn’t involved in the conversation. I wouldn’t want anyone touching my things even when it’s my friend. I think it’s completely rude to invade my space. I would also know that it was done with good intentions, but it’s still not okay. I try to be kind to people and I try to be understanding, but this would be an instance that I would blow up most likely. I would try everything possible to be kind about it but I know i would lose it. |
| 100C | I would look around to find a possible cause. If there are no obvious reasons, I would call 911 because I don’t know who to call, but if it’s a gas leak, 911 can get the fire department there very quickly and efficiently. They can easily find out if it is a dangerous cause and if it is, prevent a larger issue from happening. I would also try to find anyone in the building and tell them it may be a good idea to go outside for a little bit until someone can examine the cause of the smell. I would find everyone I could and then go outside myself and stay far away from the building in the very unlikely event that it were to blow up for any reason. |
| 100E | I would feel very awkward. I would probably try to find a quiet area to hide. If I couldn’t, I would go and wait outside. I probably wouldn’t have even gone to the party in real life because I think it’s rude to go to someone’s party that you don’t even know. If that person invites you it still isn’t the best time to meet someone, you’d be better off meeting them at a cafe or something beforehand. If I were really there I would wait outside and vape until my friend arrives. If she is more than an hour late I would go back home. |
| 100N | I would think a few things. I would think that I either overstepped my boundaries, or that it was a very sensitive topic for the friend. They may not want to respond to me because I’m not close enough with them. If I am close, it may be something they aren’t ready to talk about. I would give the person time, but if they still don’t respond, I would respond again with something along the lines of, “If you don’t want to talk about it, I understand. If you need someone to talk to, let me know. If you want to talk about it in person we can set up a coffee date. I care about you and I’m sorry if I overstepped my bounds, you can tell me.” |
| 100O | I would go to Florida, even though its only a few states away from PA, my son is there. He was under the custody of his father and he was abusive to me so I left him. At the time our son was with CYS, but they gave him custody despite his record. As of now, he is in CYS custody in Florida and I can’t visit him. I would do anything to be able to visit him in person. Id spend all the money on food for us and buying everything he could possibly need for him and want. I’d buy him so many toys and clothing. Id want to save it so when I have him I can still spoil him. This trip would mean more to me than anything in the world. Nothing in the world matters without him. I’m just glad he’s finally out of the situation he was in with his father due to CYS negligence, knowing his past and taking him away in the first place due to his aggression and my “inability to protect him” because I was the one being abused. |
| 101A | I would cringe at first and hope it’s a color I can live with for awhile without hurting my friend’s feelings and without having to repaint the room myself. I would make sure none of my things in the room had paint on them and I would make sure the room was aired out before I decided to sleep in there. I would be a little annoyed and upset that it was done without asking me first but I wouldn’t say anything about it. I would be upset that my things were moved or that someone was in my room at all without asking but I would understand the reason whether or not I agreed with it. |
Text can then be cleaned in an assortment of ways using clean_text.
raw_data$clean_text <- clean_text(raw_data$raw_text)knitr::kable(head(raw_data))| uid | raw_text | clean_text |
|---|---|---|
| 100A | Depending on the color, I would be very irritated. I would honestly freak out and scream and yell if it was a color I didn’t like especially. If it was a color I like, I still would be upset because I wasn’t involved in the conversation. I wouldn’t want anyone touching my things even when it’s my friend. I think it’s completely rude to invade my space. I would also know that it was done with good intentions, but it’s still not okay. I try to be kind to people and I try to be understanding, but this would be an instance that I would blow up most likely. I would try everything possible to be kind about it but I know i would lose it. | Depending on the color I would be very irritated I would honestly freak out and scream and yell if it was a color I didn’t like especially If it was a color I like I still would be upset because I wasn’t involved in the conversation I wouldn’t want anyone touching my things even when it’s my friend I think it’s completely rude to invade my space I would also know that it was done with good intentions but it’s still not okay I try to be kind to people and I try to be understanding but this would be an instance that I would blow up most likely I would try everything possible to be kind about it but I know i would lose it |
| 100C | I would look around to find a possible cause. If there are no obvious reasons, I would call 911 because I don’t know who to call, but if it’s a gas leak, 911 can get the fire department there very quickly and efficiently. They can easily find out if it is a dangerous cause and if it is, prevent a larger issue from happening. I would also try to find anyone in the building and tell them it may be a good idea to go outside for a little bit until someone can examine the cause of the smell. I would find everyone I could and then go outside myself and stay far away from the building in the very unlikely event that it were to blow up for any reason. | I would look around to find a possible cause If there are no obvious reasons I would call because I don’t know who to call but if it’s a gas leak can get the fire department there very quickly and efficiently They can easily find out if it is a dangerous cause and if it is prevent a larger issue from happening I would also try to find anyone in the building and tell them it may be a good idea to go outside for a little bit until someone can examine the cause of the smell I would find everyone I could and then go outside myself and stay far away from the building in the very unlikely event that it were to blow up for any reason |
| 100E | I would feel very awkward. I would probably try to find a quiet area to hide. If I couldn’t, I would go and wait outside. I probably wouldn’t have even gone to the party in real life because I think it’s rude to go to someone’s party that you don’t even know. If that person invites you it still isn’t the best time to meet someone, you’d be better off meeting them at a cafe or something beforehand. If I were really there I would wait outside and vape until my friend arrives. If she is more than an hour late I would go back home. | I would feel very awkward I would probably try to find a quiet area to hide If I couldn’t I would go and wait outside I probably wouldn’t have even gone to the party in real life because I think it’s rude to go to someone’s party that you don’t even know If that person invites you it still isn’t the best time to meet someone you’d be better off meeting them at a cafe or something beforehand If I were really there I would wait outside and vape until my friend arrives If she is more than an hour late I would go back home |
| 100N | I would think a few things. I would think that I either overstepped my boundaries, or that it was a very sensitive topic for the friend. They may not want to respond to me because I’m not close enough with them. If I am close, it may be something they aren’t ready to talk about. I would give the person time, but if they still don’t respond, I would respond again with something along the lines of, “If you don’t want to talk about it, I understand. If you need someone to talk to, let me know. If you want to talk about it in person we can set up a coffee date. I care about you and I’m sorry if I overstepped my bounds, you can tell me.” | I would think a few things I would think that I either overstepped my boundaries or that it was a very sensitive topic for the friend They may not want to respond to me because I’m not close enough with them If I am close it may be something they aren’t ready to talk about I would give the person time but if they still don’t respond I would respond again with something along the lines of If you don’t want to talk about it I understand If you need someone to talk to let me know If you want to talk about it in person we can set up a coffee date I care about you and I’m sorry if I overstepped my bounds you can tell me |
| 100O | I would go to Florida, even though its only a few states away from PA, my son is there. He was under the custody of his father and he was abusive to me so I left him. At the time our son was with CYS, but they gave him custody despite his record. As of now, he is in CYS custody in Florida and I can’t visit him. I would do anything to be able to visit him in person. Id spend all the money on food for us and buying everything he could possibly need for him and want. I’d buy him so many toys and clothing. Id want to save it so when I have him I can still spoil him. This trip would mean more to me than anything in the world. Nothing in the world matters without him. I’m just glad he’s finally out of the situation he was in with his father due to CYS negligence, knowing his past and taking him away in the first place due to his aggression and my “inability to protect him” because I was the one being abused. | I would go to Florida even though its only a few states away from PA my son is there He was under the custody of his father and he was abusive to me so I left him At the time our son was with CYS but they gave him custody despite his record As of now he is in CYS custody in Florida and I can’t visit him I would do anything to be able to visit him in person Id spend all the money on food for us and buying everything he could possibly need for him and want I’d buy him so many toys and clothing Id want to save it so when I have him I can still spoil him This trip would mean more to me than anything in the world Nothing in the world matters without him I’m just glad he’s finally out of the situation he was in with his father due to CYS negligence knowing his past and taking him away in the first place due to his aggression and my inability to protect him because I was the one being abused |
| 101A | I would cringe at first and hope it’s a color I can live with for awhile without hurting my friend’s feelings and without having to repaint the room myself. I would make sure none of my things in the room had paint on them and I would make sure the room was aired out before I decided to sleep in there. I would be a little annoyed and upset that it was done without asking me first but I wouldn’t say anything about it. I would be upset that my things were moved or that someone was in my room at all without asking but I would understand the reason whether or not I agreed with it. | I would cringe at first and hope it’s a color I can live with for awhile without hurting my friend’s feelings and without having to repaint the room myself I would make sure none of my things in the room had paint on them and I would make sure the room was aired out before I decided to sleep in there I would be a little annoyed and upset that it was done without asking me first but I wouldn’t say anything about it I would be upset that my things were moved or that someone was in my room at all without asking but I would understand the reason whether or not I agreed with it |
Normalizing
To ensure greater accuracy in the automated coding process we can normalize responses using normalize_text.
raw_data$clean_text <- normalize_text(raw_data$clean_text)knitr::kable(head(raw_data))| uid | raw_text | clean_text |
|---|---|---|
| 100A | Depending on the color, I would be very irritated. I would honestly freak out and scream and yell if it was a color I didn’t like especially. If it was a color I like, I still would be upset because I wasn’t involved in the conversation. I wouldn’t want anyone touching my things even when it’s my friend. I think it’s completely rude to invade my space. I would also know that it was done with good intentions, but it’s still not okay. I try to be kind to people and I try to be understanding, but this would be an instance that I would blow up most likely. I would try everything possible to be kind about it but I know i would lose it. | depend on the color i would be very irritate i would honestly freak out and scream and yell if it be a color i do not like especially if it be a color i like i still would be upset because i be not involve in the conversation i would not want anyone touch my thing even when it be my friend i think it be completely rude to invade my space i would also know that it be do with good intention but it be still not okay i try to be kind to people and i try to be understand but this would be a instance that i would blow up much likely i would try everything possible to be kind about it but i know i would lose it |
| 100C | I would look around to find a possible cause. If there are no obvious reasons, I would call 911 because I don’t know who to call, but if it’s a gas leak, 911 can get the fire department there very quickly and efficiently. They can easily find out if it is a dangerous cause and if it is, prevent a larger issue from happening. I would also try to find anyone in the building and tell them it may be a good idea to go outside for a little bit until someone can examine the cause of the smell. I would find everyone I could and then go outside myself and stay far away from the building in the very unlikely event that it were to blow up for any reason. | i would look around to find a possible cause if there be no obvious reason i would call because i do not know who to call but if it be a gas leak can get the fire department there very quickly and efficiently they can easily find out if it be a dangerous cause and if it be prevent a large issue from happen i would also try to find anyone in the build and tell them it may be a good idea to go outside for a little bite until someone can examine the cause of the smell i would find everyone i can and then go outside myself and stay far away from the build in the very unlikely event that it be to blow up for any reason |
| 100E | I would feel very awkward. I would probably try to find a quiet area to hide. If I couldn’t, I would go and wait outside. I probably wouldn’t have even gone to the party in real life because I think it’s rude to go to someone’s party that you don’t even know. If that person invites you it still isn’t the best time to meet someone, you’d be better off meeting them at a cafe or something beforehand. If I were really there I would wait outside and vape until my friend arrives. If she is more than an hour late I would go back home. | i would feel very awkward i would probably try to find a quiet area to hide if i can not i would go and wait outside i probably would not have even go to the party in real life because i think it be rude to go to someone have party that you do not even know if that person invite you it still be not the good time to meet someone you have be good off meet them at a cafe or something beforehand if i be really there i would wait outside and vape until my friend arrive if she be much than a hour late i would go back home |
| 100N | I would think a few things. I would think that I either overstepped my boundaries, or that it was a very sensitive topic for the friend. They may not want to respond to me because I’m not close enough with them. If I am close, it may be something they aren’t ready to talk about. I would give the person time, but if they still don’t respond, I would respond again with something along the lines of, “If you don’t want to talk about it, I understand. If you need someone to talk to, let me know. If you want to talk about it in person we can set up a coffee date. I care about you and I’m sorry if I overstepped my bounds, you can tell me.” | i would think a few thing i would think that i either overstep my boundary or that it be a very sensitive topic for the friend they may not want to respond to me because i be not close enough with them if i be close it may be something they be not ready to talk about i would give the person time but if they still do not respond i would respond again with something along the line of if you do not want to talk about it i understand if you need someone to talk to let me know if you want to talk about it in person we can set up a coffee date i care about you and i be sorry if i overstep my bound you can tell me |
| 100O | I would go to Florida, even though its only a few states away from PA, my son is there. He was under the custody of his father and he was abusive to me so I left him. At the time our son was with CYS, but they gave him custody despite his record. As of now, he is in CYS custody in Florida and I can’t visit him. I would do anything to be able to visit him in person. Id spend all the money on food for us and buying everything he could possibly need for him and want. I’d buy him so many toys and clothing. Id want to save it so when I have him I can still spoil him. This trip would mean more to me than anything in the world. Nothing in the world matters without him. I’m just glad he’s finally out of the situation he was in with his father due to CYS negligence, knowing his past and taking him away in the first place due to his aggression and my “inability to protect him” because I was the one being abused. | i would go to florida even though its only a few state away from pa my son be there he be under the custody of his father and he be abusive to me so i leave him at the time our son be with cys but they give him custody despite his record as of now he be in cys custody in florida and i cannot visit him i would do anything to be able to visit him in person id spend all the money on food for us and buy everything he can possibly need for him and want i have buy him so many toy and clothe id want to save it so when i have him i can still spoil him this trip would mean much to me than anything in the world nothing in the world matter without him i be just glad he have finally out of the situation he be in with his father due to cys negligence know his past and take him away in the first place due to his aggression and my inability to protect him because i be the one be abuse |
| 101A | I would cringe at first and hope it’s a color I can live with for awhile without hurting my friend’s feelings and without having to repaint the room myself. I would make sure none of my things in the room had paint on them and I would make sure the room was aired out before I decided to sleep in there. I would be a little annoyed and upset that it was done without asking me first but I wouldn’t say anything about it. I would be upset that my things were moved or that someone was in my room at all without asking but I would understand the reason whether or not I agreed with it. | i would cringe at first and hope it be a color i can live with for awhile without hurt my friend feeling and without have to repaint the room myself i would make sure none of my thing in the room have paint on them and i would make sure the room be air out before i decide to sleep in there i would be a little annoy and upset that it be do without ask me first but i would not say anything about it i would be upset that my thing be move or that someone be in my room at all without ask but i would understand the reason whether or not i agree with it |
Coding
Now that we have our documents encoded we can start coding. The algorithms were made to reduce the workload substantially, so they will provide suggested codes. If they are unsure they will return NA. You can be more or less conservative using the thresh argument (see code_documents_custom).
Codebook
A code book can be developed to augment the coding experience (an example is below)
# our custom codebook
codebook <- list(agreeablness = c("nice", "kind", "warm"), conscientiousness = c("order",
"organize", "on time", "rule"), extraversion = c("talkative", "outgoing", "not shy",
"social", "expressive"), neuroticism = c("worried", "emotional", "sad", "angry",
"mad", "depressed", "anxious"), openness = c("learn", "new", "open", "smart",
"open minded", "idea", "abstract"))
# convert to data.frame
codebook <- data.frame(code = unlist(codebook), theme = rep(names(codebook), sapply(codebook,
length)))Our now codebook looks like this:
| code | theme |
|---|---|
| nice | agreeablness |
| kind | agreeablness |
| warm | agreeablness |
| order | conscientiousness |
| organize | conscientiousness |
| on time | conscientiousness |
Remember: Codebooks used must follow the basic structure:
| code | theme |
|---|---|
| word1 | themelabel1 |
| word2 | themelabel1 |
| word3 | themelabel1 |
| word4 | themelabel2 |
| word5 | themelabel2 |
| word6 | themelabel3 |
Suggesting Codes using codebook
Returning a single suggested code
You can automate manual coding of documents using lists of words for each theme.
# using the default codebook
suggested_code <- code_documents_custom(raw_data$clean_text, code_book = codebook)
table(suggested_code)suggested_code
agreeablness conscientiousness extraversion neuroticism
385 101 87 339
openness
558 # this can be added to your raw_data
raw_data$suggested_code <- suggested_codeReturning a matrix of counts for each category
By setting verbose = TRUE and normalize_count = FALSE you can return a matrix of counts per category.
# using the default codebook
suggested_code_matrix <- code_documents_custom(raw_data$clean_text, code_book = codebook,
verbose = TRUE, normalize_count = FALSE)knitr::kable(head(as.data.frame(suggested_code_matrix), 25))| agreeablness | conscientiousness | extraversion | neuroticism | openness | suggestion |
|---|---|---|---|---|---|
| 2 | 0 | 0 | 0 | 0 | agreeablness |
| 0 | 0 | 0 | 0 | 1 | openness |
| 0 | 0 | 0 | 0 | 0 | NA |
| 0 | 0 | 0 | 0 | 0 | NA |
| 0 | 0 | 0 | 0 | 0 | NA |
| 0 | 0 | 0 | 0 | 0 | NA |
| 0 | 0 | 0 | 0 | 0 | NA |
| 0 | 0 | 0 | 0 | 0 | NA |
| 1 | 0 | 0 | 0 | 0 | agreeablness |
| 1 | 0 | 0 | 0 | 0 | agreeablness |
| 0 | 0 | 0 | 0 | 1 | openness |
| 0 | 0 | 0 | 0 | 0 | NA |
| 0 | 0 | 0 | 0 | 1 | openness |
| 0 | 0 | 0 | 0 | 0 | NA |
| 0 | 0 | 0 | 0 | 0 | NA |
| 0 | 0 | 0 | 2 | 0 | neuroticism |
| 0 | 0 | 0 | 0 | 0 | NA |
| 0 | 0 | 0 | 0 | 0 | NA |
| 0 | 0 | 0 | 0 | 0 | NA |
| 0 | 0 | 0 | 0 | 0 | NA |
| 0 | 0 | 0 | 0 | 0 | NA |
| 0 | 0 | 0 | 0 | 0 | NA |
| 0 | 0 | 1 | 1 | 0 | NA |
| 0 | 0 | 0 | 0 | 0 | NA |
| 1 | 0 | 0 | 0 | 0 | agreeablness |
This can then be binded to your raw data
# add the matrix to your raw data
raw_data_with_codes <- cbind(raw_data, suggested_code_matrix)Outputting Results
We can now write our data to a .csv file
write.csv(raw_data_with_codes, "data-with-codes.csv", row.names = F)Getting Help
Feel free to email Shea Fyffe sfyffe@masonlive.gmu.edu if you have any other questions, concerns, or suggestions.