R Assignment - Basic Data Loading and Transformations
Loading Data into a Data Frame
Very often, we’re tasked with taking data in one form and transforming it for easier downstream analysis. We will spend several weeks in this course on tidying and transformation operations. Some of this work could be done in SQL or R (or Python or…). Here, you are asked to use R -you may use base functions or packages as you like.
Mushrooms Dataset. A famous-if slightly moldy-dataset about mushrooms can be found in the UCI repository here: https://archive.ics.uci.edu/ml/datasets/Mushroom. The fact that this is such a well-known dataset in the data science community makes it a good dataset to use for comparative benchmarking. For example, if someone was working to build a better decision tree algorithm (or other predictive classifier) to analyze categorical data, this dataset could be useful. A typical problem (which is beyond the scope of this assignment!) is to answer the question, “Which other attribute or attributes are the best predictors of whether a particular mushroom is poisonous or edible?”
Your task is to study the dataset and the associated description of the data (i.e. “data dictionary”). You may need to look around a bit, but it’s there! You should take the data, and create a data frame with a subset of the columns in the dataset. You should include the column that indicates edible or poisonous and three or four other columns. You should also add meaningful column names and replace the abbreviations used in the data-for example, in the appropriate column, “e” might become “edible.” Your deliverable is the R code to perform these transformation tasks.
If you are working in a group, you also have the option of replacing the mushroom dataset in the assignment with a different data set that your group members might find more interesting.
Please place your solution in to a single R Markdown (.Rmd) file and publish your solution out to rpubs.com. You should post the .Rmd file in your GitHub repository, and provide the appropriate URLs to your GitHub repository and your rpubs.com file in your assignment link. You should also have the original data file accessible through your code-for example, stored in a GitHub repository and referenced in your code. We’ll look together at some of the most interesting student solutions in next week’s meetup.
url <- 'https://archive.ics.uci.edu/ml/machine-learning-databases/mushroom/agaricus-lepiota.data'
mushrooms <- read.table(url, sep=",", header=FALSE, stringsAsFactors = FALSE)
head(mushrooms)## V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15 V16 V17 V18 V19 V20
## 1 p x s n t p f c n k e e s s w w p w o p
## 2 e x s y t a f c b k e c s s w w p w o p
## 3 e b s w t l f c b n e c s s w w p w o p
## 4 p x y w t p f c n n e e s s w w p w o p
## 5 e x s g f n f w b k t e s s w w p w o e
## 6 e x y y t a f c b n e c s s w w p w o p
## V21 V22 V23
## 1 k s u
## 2 n n g
## 3 n n m
## 4 k s u
## 5 n a g
## 6 k n gI have saved the dictionary from https://archive.ics.uci.edu/ml/machine-learning-databases/bridges/bridges.names in a Dictionary.txt file
The file I have included into GitHub with the following link: https://raw.githubusercontent.com/dvillalobos/MSDA/master/607/Homework/Villalobos-Homework1-dictionary.txt
file <- 'https://raw.githubusercontent.com/dvillalobos/MSDA/master/607/Homework/Villalobos-Homework1-dictionary.txt'
mushroomsdict <- read.table(file, sep="|", header=TRUE, stringsAsFactors = FALSE)
mushroomsdict## Index Attribute
## 1 0 class
## 2 1 cap-shape
## 3 2 cap-surface
## 4 3 cap-color
## 5 4 bruises?
## 6 5 odor
## 7 6 gill-attachment
## 8 7 gill-spacing
## 9 8 gill-size
## 10 9 gill-color
## 11 10 stalk-shape
## 12 11 stalk-root
## 13 12 stalk-surface-above-ring
## 14 13 stalk-surface-below-ring
## 15 14 stalk-color-above-ring
## 16 15 stalk-color-below-ring
## 17 16 veil-type
## 18 17 veil-color
## 19 18 ring-number
## 20 19 ring-type
## 21 20 spore-print-color
## 22 21 population
## 23 22 habitat
## Information
## 1 edible=e,poisonous=p
## 2 bell=b,conical=c,convex=x,flat=f,knobbed=k,sunken=s
## 3 fibrous=f,grooves=g,scaly=y,smooth=s
## 4 brown=n,buff=b,cinnamon=c,gray=g,green=r,pink=p,purple=u,red=e,white=w,yellow=y
## 5 bruises=t,no=f
## 6 almond=a,anise=l,creosote=c,fishy=y,foul=f,musty=m,none=n,pungent=p,spicy=s
## 7 attached=a,descending=d,free=f,notched=n
## 8 close=c,crowded=w,distant=d
## 9 broad=b,narrow=n
## 10 black=k,brown=n,buff=b,chocolate=h,gray=g,green=r,orange=o,pink=p,purple=u,red=e,white=w,yellow=y
## 11 enlarging=e,tapering=t
## 12 bulbous=b,club=c,cup=u,equal=e,rhizomorphs=z,rooted=r,missing=?
## 13 fibrous=f,scaly=y,silky=k,smooth=s
## 14 fibrous=f,scaly=y,silky=k,smooth=s
## 15 brown=n,buff=b,cinnamon=c,gray=g,orange=o,pink=p,red=e,white=w,yellow=y
## 16 brown=n,buff=b,cinnamon=c,gray=g,orange=o,pink=p,red=e,white=w,yellow=y
## 17 partial=p,universal=u
## 18 brown=n,orange=o,white=w,yellow=y
## 19 none=n,one=o,two=t
## 20 cobwebby=c,evanescent=e,flaring=f,large=l,none=n,pendant=p,sheathing=s,zone=z
## 21 black=k,brown=n,buff=b,chocolate=h,green=r,orange=o,purple=u,white=w,yellow=y
## 22 abundant=a,clustered=c,numerous=n,scattered=s,several=v,solitary=y
## 23 grasses=g,leaves=l,meadows=m,paths=p,urban=u,waste=w,woods=dcolnames(mushrooms) <- mushroomsdict$Attribute
head(mushrooms)## class cap-shape cap-surface cap-color bruises? odor gill-attachment
## 1 p x s n t p f
## 2 e x s y t a f
## 3 e b s w t l f
## 4 p x y w t p f
## 5 e x s g f n f
## 6 e x y y t a f
## gill-spacing gill-size gill-color stalk-shape stalk-root
## 1 c n k e e
## 2 c b k e c
## 3 c b n e c
## 4 c n n e e
## 5 w b k t e
## 6 c b n e c
## stalk-surface-above-ring stalk-surface-below-ring stalk-color-above-ring
## 1 s s w
## 2 s s w
## 3 s s w
## 4 s s w
## 5 s s w
## 6 s s w
## stalk-color-below-ring veil-type veil-color ring-number ring-type
## 1 w p w o p
## 2 w p w o p
## 3 w p w o p
## 4 w p w o p
## 5 w p w o e
## 6 w p w o p
## spore-print-color population habitat
## 1 k s u
## 2 n n g
## 3 n n m
## 4 k s u
## 5 n a g
## 6 k n g# Automatic transformation for all columns by using a function
transformMushrooms <- function(headercolumn){
# Reading information from dictionary to work on Column
mheaderValues <- mushroomsdict$Information[headercolumn]
# Split Column based on "," separators
mheaderValues <- strsplit(as.character(mheaderValues),',',fixed=TRUE)
# Create data frame for newly created separated list
mheaderValues <- data.frame(mheaderValues)
# Rename the Column with Values taking from the dictionary file in order to keep it as valid data frame
colnames(mheaderValues) <- mushroomsdict$Attribute[headercolumn]
# Separate the values of the Column taking "=" as a separator
mheaderValues <- data.frame(do.call('rbind', strsplit(as.character(mheaderValues[,1]),'=',fixed=TRUE)))
# Rename the values of the two new columns; one with the Original Name from the dictionary file and the secondI use as "Values"
colnames(mheaderValues) <- c(mushroomsdict$Attribute[headercolumn], "Values")
#mheaderValues
# Assign factors
# Important: Need to conver to FACTOR in order to work
mush[,headercolumn] <- factor(mush[,headercolumn], ordered = TRUE)
# Thanks to Georgia, I got the idea for the transformation by using "levels" function
# Need to point out that this transformation only work when the factors have the same orders as our dictionary; otherwise will bring errors in the transformation process
levels(mush[,headercolumn]) <- as.character(mheaderValues[,1])
return(mush)
}
# Transforming all values automatically
# Create New Subset
mush <- subset(mushrooms, select = c(1:dim(mushrooms)[2]))
head(mush)## class cap-shape cap-surface cap-color bruises? odor gill-attachment
## 1 p x s n t p f
## 2 e x s y t a f
## 3 e b s w t l f
## 4 p x y w t p f
## 5 e x s g f n f
## 6 e x y y t a f
## gill-spacing gill-size gill-color stalk-shape stalk-root
## 1 c n k e e
## 2 c b k e c
## 3 c b n e c
## 4 c n n e e
## 5 w b k t e
## 6 c b n e c
## stalk-surface-above-ring stalk-surface-below-ring stalk-color-above-ring
## 1 s s w
## 2 s s w
## 3 s s w
## 4 s s w
## 5 s s w
## 6 s s w
## stalk-color-below-ring veil-type veil-color ring-number ring-type
## 1 w p w o p
## 2 w p w o p
## 3 w p w o p
## 4 w p w o p
## 5 w p w o e
## 6 w p w o p
## spore-print-color population habitat
## 1 k s u
## 2 n n g
## 3 n n m
## 4 k s u
## 5 n a g
## 6 k n g# Calling the transformation. function
mush <- transformMushrooms(1) # Calling to transform column # 1
mush <- transformMushrooms(8) # Calling to transform column # 8
mush <- transformMushrooms(9) # Calling to transform column # 9
head(mush,25)## class cap-shape cap-surface cap-color bruises? odor gill-attachment
## 1 poisonous x s n t p f
## 2 edible x s y t a f
## 3 edible b s w t l f
## 4 poisonous x y w t p f
## 5 edible x s g f n f
## 6 edible x y y t a f
## 7 edible b s w t a f
## 8 edible b y w t l f
## 9 poisonous x y w t p f
## 10 edible b s y t a f
## 11 edible x y y t l f
## 12 edible x y y t a f
## 13 edible b s y t a f
## 14 poisonous x y w t p f
## 15 edible x f n f n f
## 16 edible s f g f n f
## 17 edible f f w f n f
## 18 poisonous x s n t p f
## 19 poisonous x y w t p f
## 20 poisonous x s n t p f
## 21 edible b s y t a f
## 22 poisonous x y n t p f
## 23 edible b y y t l f
## 24 edible b y w t a f
## 25 edible b s w t l f
## gill-spacing gill-size gill-color stalk-shape stalk-root
## 1 close narrow k e e
## 2 close broad k e c
## 3 close broad n e c
## 4 close narrow n e e
## 5 crowded broad k t e
## 6 close broad n e c
## 7 close broad g e c
## 8 close broad n e c
## 9 close narrow p e e
## 10 close broad g e c
## 11 close broad g e c
## 12 close broad n e c
## 13 close broad w e c
## 14 close narrow k e e
## 15 crowded broad n t e
## 16 close narrow k e e
## 17 crowded broad k t e
## 18 close narrow n e e
## 19 close narrow n e e
## 20 close narrow k e e
## 21 close broad k e c
## 22 close narrow n e e
## 23 close broad k e c
## 24 close broad w e c
## 25 close broad g e c
## stalk-surface-above-ring stalk-surface-below-ring
## 1 s s
## 2 s s
## 3 s s
## 4 s s
## 5 s s
## 6 s s
## 7 s s
## 8 s s
## 9 s s
## 10 s s
## 11 s s
## 12 s s
## 13 s s
## 14 s s
## 15 s f
## 16 s s
## 17 s s
## 18 s s
## 19 s s
## 20 s s
## 21 s s
## 22 s s
## 23 s s
## 24 s s
## 25 s s
## stalk-color-above-ring stalk-color-below-ring veil-type veil-color
## 1 w w p w
## 2 w w p w
## 3 w w p w
## 4 w w p w
## 5 w w p w
## 6 w w p w
## 7 w w p w
## 8 w w p w
## 9 w w p w
## 10 w w p w
## 11 w w p w
## 12 w w p w
## 13 w w p w
## 14 w w p w
## 15 w w p w
## 16 w w p w
## 17 w w p w
## 18 w w p w
## 19 w w p w
## 20 w w p w
## 21 w w p w
## 22 w w p w
## 23 w w p w
## 24 w w p w
## 25 w w p w
## ring-number ring-type spore-print-color population habitat
## 1 o p k s u
## 2 o p n n g
## 3 o p n n m
## 4 o p k s u
## 5 o e n a g
## 6 o p k n g
## 7 o p k n m
## 8 o p n s m
## 9 o p k v g
## 10 o p k s m
## 11 o p n n g
## 12 o p k s m
## 13 o p n s g
## 14 o p n v u
## 15 o e k a g
## 16 o p n y u
## 17 o e n a g
## 18 o p k s g
## 19 o p n s u
## 20 o p n s u
## 21 o p n s m
## 22 o p n v g
## 23 o p n s m
## 24 o p n n m
## 25 o p k s m# New Subset to reduce the number of columns.
mush <- subset(mush, select = c(1,8,9))
head(mush,25)## class gill-spacing gill-size
## 1 poisonous close narrow
## 2 edible close broad
## 3 edible close broad
## 4 poisonous close narrow
## 5 edible crowded broad
## 6 edible close broad
## 7 edible close broad
## 8 edible close broad
## 9 poisonous close narrow
## 10 edible close broad
## 11 edible close broad
## 12 edible close broad
## 13 edible close broad
## 14 poisonous close narrow
## 15 edible crowded broad
## 16 edible close narrow
## 17 edible crowded broad
## 18 poisonous close narrow
## 19 poisonous close narrow
## 20 poisonous close narrow
## 21 edible close broad
## 22 poisonous close narrow
## 23 edible close broad
## 24 edible close broad
## 25 edible close broad