1 Introduction

This is an R Markdown document. Markdown is a simple formatting syntax for authoring HTML, PDF, and MS Word documents. For more details on using R Markdown see http://rmarkdown.rstudio.com.

When you click the Knit button a document will be generated that includes both content as well as the output of any embedded R code chunks within the document. You can embed an R code chunk like this:

# Summary is a built in function used to generate a statistic summary of a dataset.

# Here we need to get summary of cars dataset.

summary(cars)
##      speed           dist       
##  Min.   : 4.0   Min.   :  2.00  
##  1st Qu.:12.0   1st Qu.: 26.00  
##  Median :15.0   Median : 36.00  
##  Mean   :15.4   Mean   : 42.98  
##  3rd Qu.:19.0   3rd Qu.: 56.00  
##  Max.   :25.0   Max.   :120.00

Including Plots

You can also embed plots, for example:

# Create temperature data
temperature <- seq(0, 360, by = 20)

# Create pressure data
pressure <- c(
  0, 0, 0, 0, 0, 0, 0, 0,
  5, 10, 18, 32, 55, 95, 155,
  245, 375, 560, 805
)
# Generate scatter plot
plot(
  temperature,
  pressure,
  main = "Temperature vs Pressure",
  xlab = "temperature",
  ylab = "pressure",
  pch = 1
)

Note that the echo = FALSE parameter was added to the code chunk to prevent printing of the R code that generated the plot.

Understanding the plot() function

temperature: Data for the X-axis

pressure : Data for the Y-axis

main: Graph title

xlab: X-axis label

ylab: X-asis label

pch = 1 : Displays hollow circular points

You can customize the graph using:

pch = 19: Solid circles.

col = “blue”: Blue points.

cex = 1.2: Increases point size.

Important definitions in data science

Statistics

Modelling

Computation

Calculus

Today’s class,we will study

Installation of R

Installation of R studio

Calculator in r

Data importation in R

Data description using numerical measures and graphs.

2 Data structure (Vectors, Factors, Lists, Data frame, matrix and arrays)

2.1 Vector: The foundational R data structure

Numeric Example 1

x <- c(0.5, 0.6) 
age<-c(20,35,32,29)
summary(age)
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   20.00   26.75   30.50   29.00   32.75   35.00

Example 2

age<-c(35,24,18,24)
mean(age)
## [1] 25.25

Logical

x <-c(TRUE, FALSE)
x <- c(T, F)

Character

x <- c("a","b","c")
class <- c('M','F','F','M')

Integer

x <- 9:29

Complex

x <- c(1+0i, 2+4i)

Matrix and operation within matrices

(y<- matrix(1:6, nrow = 2, ncol = 3))
##      [,1] [,2] [,3]
## [1,]    1    3    5
## [2,]    2    4    6
y
##      [,1] [,2] [,3]
## [1,]    1    3    5
## [2,]    2    4    6
(z <-matrix(1:6, nrow=3, ncol=2))
##      [,1] [,2]
## [1,]    1    4
## [2,]    2    5
## [3,]    3    6
y%*%z
##      [,1] [,2]
## [1,]   22   49
## [2,]   28   64

2.2 List

(x<- list(1, "a", TRUE, 1 + 4i))
## [[1]]
## [1] 1
## 
## [[2]]
## [1] "a"
## 
## [[3]]
## [1] TRUE
## 
## [[4]]
## [1] 1+4i
x
## [[1]]
## [1] 1
## 
## [[2]]
## [1] "a"
## 
## [[3]]
## [1] TRUE
## 
## [[4]]
## [1] 1+4i

2.3 Factor

Factors are useful when your data represents categories

Yes / No

Male / Female

Pass / Fail

Low / Medium / High

x<-factor( c("yes","yes","no", "yes", "no") ) 
x
## [1] yes yes no  yes no 
## Levels: no yes

You can also check the categories using

levels(x)
## [1] "no"  "yes"

count how many times each category occurs

table(x)
## x
##  no yes 
##   2   3

Create a vector with NAs in it

x <- c(1, 2, NA, 10, 3)

Return a logical vector on which elements is na

is.na(x)
## [1] FALSE FALSE  TRUE FALSE FALSE
x <-c(1,2,3,NA, 5)
bad <-is.na(x)
print(bad)
## [1] FALSE FALSE FALSE  TRUE FALSE
x[!bad]
## [1] 1 2 3 5
x <-c(1,2,3,"NA", 5)
bad <-is.na(x)
print(bad)
## [1] FALSE FALSE FALSE FALSE FALSE
x[!bad]
## [1] "1"  "2"  "3"  "NA" "5"

What if there are multiple R objects and you want to take the subset with no missing values of those objects?

x <-c(1,2, NA, 4, NA, 5)
y <-c("a","b",NA, "d", NA, "f")
good <-complete.cases(x, y)
good
## [1]  TRUE  TRUE FALSE  TRUE FALSE  TRUE

2.4 Coercion in R

Coercion means converting a value from one data type to another data type.

For example , R may convert number into characters or logical values into numbers when different type of data are combined

Example 1: Automatic coercion

 # Although most values are numbers, "three" is a character. R converts all values to characters:
x <-c(1,2,"three", 4)
x
## [1] "1"     "2"     "three" "4"

Example 2: Coercion hierarchy

A common coercion order in R is:

logical -> integer -> numeric -> character

y<-c(TRUE, 2, 2.5, "Hello")

# The result becomes a character vector:
y
## [1] "TRUE"  "2"     "2.5"   "Hello"

Example 3: Explicit coercion

You can manually convert data using functions such as:

as.numeric("25")
## [1] 25
as.numeric("Hello")
## Warning: NAs introduced by coercion
## [1] NA

If a vector only contains logical and numbers, R will convert the logical to numbers, Every true becomes a 1, and every FALSE becomes 0

sum(c(TRUE,TRUE,FALSE,FALSE,FALSE))
## [1] 2

2.5 Create data

Create data Using data frame Data frame is more general than matrix How??? Because data frame can contain different modes of data (Numeric, character and so on) Similar to what you can see in SPSS, SAS,…

Let’s create a data frame

studentID <-c("20252MBI001","20252MBI002", "20252MBI003", "20252MBI004","20252MBI005")
math_score <-c(12,17,10,9,NA)
gender<-c("M","F","M","M","F")
it_score<-c(13,18,17,12,19)
score_data<-data.frame(studentID,gender,math_score,it_score)

score_data
##     studentID gender math_score it_score
## 1 20252MBI001      M         12       13
## 2 20252MBI002      F         17       18
## 3 20252MBI003      M         10       17
## 4 20252MBI004      M          9       12
## 5 20252MBI005      F         NA       19
  View(score_data)

2.6 Create data from keyboard

Steps 1. Create data frame (or matrix) with variable names 2. Invoke the text editor in the data objected created at first step

data_class2 <-data.frame(height=numeric(0), weight=numeric(0),bmi=numeric(0))
data_class2 <- edit(data_class2)

3 Data Importation

R has some features that can allow you to import data from different sources (it can be text file, spreadsheet, or database)

  1. Data from excel

  2. Data statistical packages (SAS, SPSS, Stata)

  3. Data from Text files (ASCII, XML, Webscraping)

  4. Data from database management systems (SQL,MySQL, Oracle, Access)

3.1 HW1: Import data from Statistical package and from Database management systems

data_class <- read.table("D:\\Document\\BIG-DATA\\R\\Assignment\\data\\height_weight.txt",
                         header=TRUE,
                         sep=',')
variable.names(data_class)
## [1] "HEIGHT" "WEIGHT"
# Display to 6 rows

head(data_class)
##   HEIGHT WEIGHT
## 1    175     72
## 2    162     56
## 3    180     81
## 4    158     52
## 5    170     68
## 6    165     60
# Display last 6 rows

tail(data_class)
##    HEIGHT WEIGHT
## 30    167     62
## 31    173     73
## 32    156     49
## 33    186     90
## 34    171     66
## 35    169     67
# Display rows from 5 to 20 and all columns from the dataset
# number = 5 mean row to start
# number = 20 mean row to end
#, mean all columns

data_class[5:20, ]
##    HEIGHT WEIGHT
## 5     170     68
## 6     165     60
## 7     185     88
## 8     160     54
## 9     178     76
## 10    168     63
## 11    172     70
## 12    155     50
## 13    182     84
## 14    170     65
## 15    168     66
## 16    163     58
## 17    176     74
## 18    157     51
## 19    188     92
## 20    166     61
data_class[1:5,]    # First 5 rows, all columns
##   HEIGHT WEIGHT
## 1    175     72
## 2    162     56
## 3    180     81
## 4    158     52
## 5    170     68
data_class[10:20,] # Rows 10 to 20, all columns
##    HEIGHT WEIGHT
## 10    168     63
## 11    172     70
## 12    155     50
## 13    182     84
## 14    170     65
## 15    168     66
## 16    163     58
## 17    176     74
## 18    157     51
## 19    188     92
## 20    166     61
data_class[, 1] # all rows and first column(HEIGHT)
##  [1] 175 162 180 158 170 165 185 160 178 168 172 155 182 170 168 163 176 157 188
## [20] 166 174 161 179 169 171 159 183 164 177 167 173 156 186 171 169
data_class[, 2] # all rows and second column(WEIGHT)
##  [1] 72 56 81 52 68 60 88 54 76 63 70 50 84 65 66 58 74 51 92 61 71 55 79 64 69
## [26] 53 86 59 75 62 73 49 90 66 67
data_class[1:5, 1] # first 5 rows and first column(HEIGHT)
## [1] 175 162 180 158 170
data_class[1:5, 2] # first 5 rows and second column(WEIGHT)
## [1] 72 56 81 52 68
data_class[,] # all rows and columns
##    HEIGHT WEIGHT
## 1     175     72
## 2     162     56
## 3     180     81
## 4     158     52
## 5     170     68
## 6     165     60
## 7     185     88
## 8     160     54
## 9     178     76
## 10    168     63
## 11    172     70
## 12    155     50
## 13    182     84
## 14    170     65
## 15    168     66
## 16    163     58
## 17    176     74
## 18    157     51
## 19    188     92
## 20    166     61
## 21    174     71
## 22    161     55
## 23    179     79
## 24    169     64
## 25    171     69
## 26    159     53
## 27    183     86
## 28    164     59
## 29    177     75
## 30    167     62
## 31    173     73
## 32    156     49
## 33    186     90
## 34    171     66
## 35    169     67

Another way to select columns by name

data_class$HEIGHT
##  [1] 175 162 180 158 170 165 185 160 178 168 172 155 182 170 168 163 176 157 188
## [20] 166 174 161 179 169 171 159 183 164 177 167 173 156 186 171 169
summary(data_class$WEIGHT)
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    49.0    58.5    66.0    67.4    74.5    92.0
length(data_class$WEIGHT)
## [1] 35
data_class[,-1] # means select all rows but remove the first column
##  [1] 72 56 81 52 68 60 88 54 76 63 70 50 84 65 66 58 74 51 92 61 71 55 79 64 69
## [26] 53 86 59 75 62 73 49 90 66 67
data_class$WEIGHT
##  [1] 72 56 81 52 68 60 88 54 76 63 70 50 84 65 66 58 74 51 92 61 71 55 79 64 69
## [26] 53 86 59 75 62 73 49 90 66 67

3.2 In case you want to use data set built in R

data() # list of datasets currently available 
data("airquality")
variable.names(airquality)
## [1] "Ozone"   "Solar.R" "Wind"    "Temp"    "Month"   "Day"
str(airquality)
## 'data.frame':    153 obs. of  6 variables:
##  $ Ozone  : int  41 36 12 18 NA 28 23 19 8 NA ...
##  $ Solar.R: int  190 118 149 313 NA NA 299 99 19 194 ...
##  $ Wind   : num  7.4 8 12.6 11.5 14.3 14.9 8.6 13.8 20.1 8.6 ...
##  $ Temp   : int  67 72 74 62 56 66 65 59 61 69 ...
##  $ Month  : int  5 5 5 5 5 5 5 5 5 5 ...
##  $ Day    : int  1 2 3 4 5 6 7 8 9 10 ...
# Display first 6 rows
head(airquality)
##   Ozone Solar.R Wind Temp Month Day
## 1    41     190  7.4   67     5   1
## 2    36     118  8.0   72     5   2
## 3    12     149 12.6   74     5   3
## 4    18     313 11.5   62     5   4
## 5    NA      NA 14.3   56     5   5
## 6    28      NA 14.9   66     5   6
# Display last 6 rows
tail(airquality)
##     Ozone Solar.R Wind Temp Month Day
## 148    14      20 16.6   63     9  25
## 149    30     193  6.9   70     9  26
## 150    NA     145 13.2   77     9  27
## 151    14     191 14.3   75     9  28
## 152    18     131  8.0   76     9  29
## 153    20     223 11.5   68     9  30
# Display statistical summary
summary(airquality)
##      Ozone           Solar.R           Wind             Temp      
##  Min.   :  1.00   Min.   :  7.0   Min.   : 1.700   Min.   :56.00  
##  1st Qu.: 18.00   1st Qu.:115.8   1st Qu.: 7.400   1st Qu.:72.00  
##  Median : 31.50   Median :205.0   Median : 9.700   Median :79.00  
##  Mean   : 42.13   Mean   :185.9   Mean   : 9.958   Mean   :77.88  
##  3rd Qu.: 63.25   3rd Qu.:258.8   3rd Qu.:11.500   3rd Qu.:85.00  
##  Max.   :168.00   Max.   :334.0   Max.   :20.700   Max.   :97.00  
##  NAs    :37       NAs    :7                                       
##      Month            Day      
##  Min.   :5.000   Min.   : 1.0  
##  1st Qu.:6.000   1st Qu.: 8.0  
##  Median :7.000   Median :16.0  
##  Mean   :6.993   Mean   :15.8  
##  3rd Qu.:8.000   3rd Qu.:23.0  
##  Max.   :9.000   Max.   :31.0  
## 
# Display column names
names(airquality)
## [1] "Ozone"   "Solar.R" "Wind"    "Temp"    "Month"   "Day"
# Display number of rows and columns
dim(airquality)
## [1] 153   6
# Count missing values in each column
colSums(is.na(airquality))
##   Ozone Solar.R    Wind    Temp   Month     Day 
##      37       7       0       0       0       0
str(airquality) # Shows dataset structure and variable types
## 'data.frame':    153 obs. of  6 variables:
##  $ Ozone  : int  41 36 12 18 NA 28 23 19 8 NA ...
##  $ Solar.R: int  190 118 149 313 NA NA 299 99 19 194 ...
##  $ Wind   : num  7.4 8 12.6 11.5 14.3 14.9 8.6 13.8 20.1 8.6 ...
##  $ Temp   : int  67 72 74 62 56 66 65 59 61 69 ...
##  $ Month  : int  5 5 5 5 5 5 5 5 5 5 ...
##  $ Day    : int  1 2 3 4 5 6 7 8 9 10 ...
summary(airquality) # answers: What are the statistical characteristics of my dataset?
##      Ozone           Solar.R           Wind             Temp      
##  Min.   :  1.00   Min.   :  7.0   Min.   : 1.700   Min.   :56.00  
##  1st Qu.: 18.00   1st Qu.:115.8   1st Qu.: 7.400   1st Qu.:72.00  
##  Median : 31.50   Median :205.0   Median : 9.700   Median :79.00  
##  Mean   : 42.13   Mean   :185.9   Mean   : 9.958   Mean   :77.88  
##  3rd Qu.: 63.25   3rd Qu.:258.8   3rd Qu.:11.500   3rd Qu.:85.00  
##  Max.   :168.00   Max.   :334.0   Max.   :20.700   Max.   :97.00  
##  NAs    :37       NAs    :7                                       
##      Month            Day      
##  Min.   :5.000   Min.   : 1.0  
##  1st Qu.:6.000   1st Qu.: 8.0  
##  Median :7.000   Median :16.0  
##  Mean   :6.993   Mean   :15.8  
##  3rd Qu.:8.000   3rd Qu.:23.0  
##  Max.   :9.000   Max.   :31.0  
##

4 Exploration data analysis

Examine numerical variable using common summary

data_class
##    HEIGHT WEIGHT
## 1     175     72
## 2     162     56
## 3     180     81
## 4     158     52
## 5     170     68
## 6     165     60
## 7     185     88
## 8     160     54
## 9     178     76
## 10    168     63
## 11    172     70
## 12    155     50
## 13    182     84
## 14    170     65
## 15    168     66
## 16    163     58
## 17    176     74
## 18    157     51
## 19    188     92
## 20    166     61
## 21    174     71
## 22    161     55
## 23    179     79
## 24    169     64
## 25    171     69
## 26    159     53
## 27    183     86
## 28    164     59
## 29    177     75
## 30    167     62
## 31    173     73
## 32    156     49
## 33    186     90
## 34    171     66
## 35    169     67

Summary Statistics

summary(data_class)
##      HEIGHT          WEIGHT    
##  Min.   :155.0   Min.   :49.0  
##  1st Qu.:163.5   1st Qu.:58.5  
##  Median :170.0   Median :66.0  
##  Mean   :170.2   Mean   :67.4  
##  3rd Qu.:176.5   3rd Qu.:74.5  
##  Max.   :188.0   Max.   :92.0

4.1 Understanding apply()

The apply() function is used to perform a function across the rows or columns of a matrix or array.

General syntax:

apply(X, MARGIN, FUN)

X Dataset or matrix MARGIN Specifies rows or columns FUN Function to apply

4.1.1 Understanding 1 and 2

This is very important: 1 = Apply the function across rows. 2 = Apply the function across columns.

Example using your HEIGHT and WEIGHT dataset

Suppose your data looks like this:

HEIGHT WEIGHT

175 72

162 56

180 81

158 52

170 68

Calculate the mean of each row:

data_class_mean <- apply(data_class, 1, mean)
data_class_mean
##  [1] 123.5 109.0 130.5 105.0 119.0 112.5 136.5 107.0 127.0 115.5 121.0 102.5
## [13] 133.0 117.5 117.0 110.5 125.0 104.0 140.0 113.5 122.5 108.0 129.0 116.5
## [25] 120.0 106.0 134.5 111.5 126.0 114.5 123.0 102.5 138.0 118.5 118.0

Calculation:

Row Calculation Mean

1 (175 + 72) / 2 123.5

2 (162 + 56) / 2 109

3 (180 + 81) / 2 130.5

4 (158 + 52) / 2 105

5 (170 + 68) / 2 119

Calculate the mean of each column:

# This calculates:
#- Average HEIGHT.
#- Average WEIGHT.

apply(data_class, 2, mean)
## HEIGHT WEIGHT 
##  170.2   67.4

Other examples of apply()

# Calculate sum of each row
apply(data_class, 1, sum)
##  [1] 247 218 261 210 238 225 273 214 254 231 242 205 266 235 234 221 250 208 280
## [20] 227 245 216 258 233 240 212 269 223 252 229 246 205 276 237 236
# Calculate sum of each column
apply(data_class, 2, sum)
## HEIGHT WEIGHT 
##   5957   2359
# Find maximum value in each column
apply(data_class, 2, max)
## HEIGHT WEIGHT 
##    188     92
# Find minimum value in each column
apply(data_class, 2, min)
## HEIGHT WEIGHT 
##    155     49
# Calculate standard deviation of each column
apply(data_class, 2, sd)
##    HEIGHT    WEIGHT 
##  8.989536 11.973500

Rows : apply(data_class, 1, mean)

Calculates the mean of each row.

Columns : apply(data_class, 2, mean)

Calculates the mean of each column.

c(mean(data_class$HEIGHT),sd(data_class$HEIGHT))
## [1] 170.200000   8.989536
c(mean(data_class$WEIGHT),sd(data_class$WEIGHT))
## [1] 67.4000 11.9735
c(Mean=mean(data_class$HEIGHT),SD=sd(data_class$HEIGHT))
##       Mean         SD 
## 170.200000   8.989536
c(Mean=mean(data_class$WEIGHT),SD=sd(data_class$WEIGHT))
##    Mean      SD 
## 67.4000 11.9735

Variation and covariation using both numerical and graphs

4.2 Relationship between a categorical and a continuous variable

data("iris")
head(iris)
##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 1          5.1         3.5          1.4         0.2  setosa
## 2          4.9         3.0          1.4         0.2  setosa
## 3          4.7         3.2          1.3         0.2  setosa
## 4          4.6         3.1          1.5         0.2  setosa
## 5          5.0         3.6          1.4         0.2  setosa
## 6          5.4         3.9          1.7         0.4  setosa
mean(iris$Petal.Length)
## [1] 3.758
mean(iris$Petal.Length[iris$Species=="setosa"])
## [1] 1.462
mean(iris$Sepal.Width[iris$Species=='virginica'])
## [1] 2.974
# shortcut 
# by(iris$Petal.Length, iris$Species, mean)

4.3 Visualize the differences in continuous variables between categories using Box-and-whisker plot

comparing the quantitative variable Sepal.Length across the categorical variable Species in the built-in iris dataset.

Component Explanation

boxplot() Creates a box-and-whisker plot. iris\(Sepal.Length Quantitative variable (numerical data). iris\)Species Categorical variable (setosa, versicolor, virginica). ~ Separates the quantitative variable from the grouping variable. data = iris Specifies the dataset. main Defines the chart title.

boxplot(iris$Sepal.Length~iris$Species,data=iris, main="Comparison of categorical vs quantitative")

You can make the visualization clearer by adding axis labels and colors.

boxplot(iris$Sepal.Length~iris$Species, data=iris, main="Comparison of categorical vs quantitative",
        xlab = "Iris Speciies", ylab = "Iris Sepal Length cm",
        col=c("lightblue","lightgreen","lightpink"),
        border="black"
)

4.4 Relationship between two categorical variables

ucba<-data.frame(UCBAdmissions)
head(ucba)
##      Admit Gender Dept Freq
## 1 Admitted   Male    A  512
## 2 Rejected   Male    A  313
## 3 Admitted Female    A   89
## 4 Rejected Female    A   19
## 5 Admitted   Male    B  353
## 6 Rejected   Male    B  207
cross <-xtabs(Freq~Gender+Admit, data=ucba)
(cross<-xtabs(Freq~Gender+Admit,data=ucba))
##         Admit
## Gender   Admitted Rejected
##   Male       1198     1493
##   Female      557     1278
## Is there gender bias in UCB graduate admission process?
prop.table(cross,1)
##         Admit
## Gender    Admitted  Rejected
##   Male   0.4451877 0.5548123
##   Female 0.3035422 0.6964578

4.5 Simpson’s paradox

Phenomenon, where a trend that appears in combined groups of data disappears or reverses when broken down into groups.

cross2<-xtabs(Freq~Gender+Admit,data=ucba[ucba$Dept=="F",])
prop.table(cross2,1)
##         Admit
## Gender     Admitted   Rejected
##   Male   0.05898123 0.94101877
##   Female 0.07038123 0.92961877

4.6 Bar Chart

dat <- read.table(text ="ProdA ProdB ProdC ProdD
1 110 50 60 70 
2 120 50 80 65", header= TRUE)

  barplot(as.matrix(dat),beside=TRUE,col=c("Red","green"))

#barplot(as.matrix(dat),beside=TRUE,col=c("gold3","red"))

dat <- read.table(text = "A   B   C   D   E   F    G
1 10 80 30 90 70 60  90
2 20 50 70 50 40 10  40
3 60 80 80 60  60 30 160
4 20 40 70 80 20 10  70", header = TRUE)

barplot(as.matrix(dat))

4.7 Scatter Plot

plot(WEIGHT~HEIGHT,data=data_class)

plot(data_class$HEIGHT,data_class$WEIGHT)

plot(data_class[,1],data_class[,2],main = "Graph")

cor(data_class$HEIGHT,data_class$WEIGHT)
## [1] 0.9941439

4.8 Linear Models

# Create scatter plot
plot(WEIGHT ~ HEIGHT, data = data_class)

# Fit linear regression model
model <- lm(WEIGHT ~ HEIGHT, data = data_class)

# Add regression line
 # abline(model, col = "red", lwd = 2)
data_lm<-lm(HEIGHT~WEIGHT,data=data_class)
summary(data_lm)
## 
## Call:
## lm(formula = HEIGHT ~ WEIGHT, data = data_class)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -2.2128 -0.6190 -0.1839  0.8522  1.8450 
## 
## Coefficients:
##              Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 119.89336    0.96641  124.06   <2e-16 ***
## WEIGHT        0.74639    0.01412   52.85   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.9861 on 33 degrees of freedom
## Multiple R-squared:  0.9883, Adjusted R-squared:  0.988 
## F-statistic:  2793 on 1 and 33 DF,  p-value: < 2.2e-16
plot(data_lm)

hist(data_class$HEIGHT)

hist(data_class$WEIGHT,col = "blue")

4.8.1 Linear with other line

plot(WEIGHT~HEIGHT,data=data_class)
abline(lm(WEIGHT~HEIGHT,data=data_class)$coefficient)
lines(lowess(data_class$HEIGHT,data_class$WEIGHT),col="blue")

cor(data_class$HEIGHT,data_class$WEIGHT)
## [1] 0.9941439
lm(WEIGHT~HEIGHT,data=data_class)
## 
## Call:
## lm(formula = WEIGHT ~ HEIGHT, data = data_class)
## 
## Coefficients:
## (Intercept)       HEIGHT  
##    -157.968        1.324

5 Data management

Create another Variable

attach(data_class)
(BMI<-WEIGHT/(HEIGHT/100)^2) # create BMI variable and calculate BMI
##  [1] 23.51020 21.33821 25.00000 20.83000 23.52941 22.03857 25.71220 21.09375
##  [9] 23.98687 22.32143 23.66144 20.81165 25.35926 22.49135 23.38435 21.82995
## [17] 23.88946 20.69049 26.02988 22.13674 23.45092 21.21832 24.65591 22.40818
## [25] 23.59700 20.96436 25.68007 21.93635 23.93948 22.23099 24.39106 20.13478
## [33] 26.01457 22.57105 23.45856
(BMI<-round(WEIGHT/(HEIGHT/100)^2,digit=2))
##  [1] 23.51 21.34 25.00 20.83 23.53 22.04 25.71 21.09 23.99 22.32 23.66 20.81
## [13] 25.36 22.49 23.38 21.83 23.89 20.69 26.03 22.14 23.45 21.22 24.66 22.41
## [25] 23.60 20.96 25.68 21.94 23.94 22.23 24.39 20.13 26.01 22.57 23.46
head(cbind(data_class,BMI),n=5) # display 5 rows with binding column of BMI
##   HEIGHT WEIGHT   BMI
## 1    175     72 23.51
## 2    162     56 21.34
## 3    180     81 25.00
## 4    158     52 20.83
## 5    170     68 23.53
new_data_class<-cbind(data_class,BMI)
tail(cbind(data_class,BMI),n=10)
##    HEIGHT WEIGHT   BMI
## 26    159     53 20.96
## 27    183     86 25.68
## 28    164     59 21.94
## 29    177     75 23.94
## 30    167     62 22.23
## 31    173     73 24.39
## 32    156     49 20.13
## 33    186     90 26.01
## 34    171     66 22.57
## 35    169     67 23.46

Summary of BMI

summary(BMI)
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   20.13   21.89   23.38   23.04   23.96   26.03
stem(BMI,scale=2)
## 
##   The decimal point is at the |
## 
##   20 | 1
##   20 | 788
##   21 | 0123
##   21 | 89
##   22 | 01234
##   22 | 56
##   23 | 4
##   23 | 55556799
##   24 | 04
##   24 | 7
##   25 | 04
##   25 | 77
##   26 | 00

5.1 Date

today <- Sys.Date()
 format(today, format="%B %d %Y")
## [1] "October 31 2026"
 format(today, format="%A")
## [1] "Saturday"
today <- Sys.Date()
 dob <- as.Date("1982-02-10")
 #today<-as.Date("2024-8-25")
 difftime(today, dob, units="days")
## Time difference of 16334 days
startdate <- as.Date("2004-02-13")
enddate <- as.Date("2011-01-22")
days <- enddate - startdate
days
## Time difference of 2535 days

5.2 Subsetting

Use of $, []. or [[]] Creation of leadership data Interest: How men and women differ in the way they lead their organizations. 5 questions were asked in this study. Example of the question: Do men and women in management position differ in the degree to which they defer to superiors? 1: strongly disagree, 2: disagree, 3:neither agree nor disagree,4: agree, 5: strongly agree

manager <- c(1,2,3,4,5)
country<- c("US","US","UK","UK","UK")
gender<-c("M","F","F","M","F")
age<-c(32,45,25,39,99)
q1<-c(5,3,3,3,2)
q2<-c(4,5,5,3,2)
q3<-c(5,2,5,4,1)
q4<-c(5,5,5,NA,2)
q5<-c(5,5,2,NA,1)
leadership<-data.frame(manager,country, gender, age, q1,q2,q3,q4,q5, stringsAsFactors = TRUE)
leadership<-data.frame(manager,country, gender, age, q1,q2,q3,q4,q5)

str(leadership)
## 'data.frame':    5 obs. of  9 variables:
##  $ manager: num  1 2 3 4 5
##  $ country: chr  "US" "US" "UK" "UK" ...
##  $ gender : chr  "M" "F" "F" "M" ...
##  $ age    : num  32 45 25 39 99
##  $ q1     : num  5 3 3 3 2
##  $ q2     : num  4 5 5 3 2
##  $ q3     : num  5 2 5 4 1
##  $ q4     : num  5 5 5 NA 2
##  $ q5     : num  5 5 2 NA 1
names(leadership)
## [1] "manager" "country" "gender"  "age"     "q1"      "q2"      "q3"     
## [8] "q4"      "q5"

What you can do - combine the score of the five questions -handle the missing values - create a dataset of what you want -create age group or age categories - 99 indicate the value is missing

5.3 Missing value

leadership$age[leadership$age==99]<-NA
View(leadership)
leadership$agecat[leadership$age >75]<-"Elder"
leadership$agecat[leadership$age>=35 & leadership$age<=75]<-"Middle Aged"
leadership$agecat[leadership$age <35]<- "Young"
leadership
##   manager country gender age q1 q2 q3 q4 q5      agecat
## 1       1      US      M  32  5  4  5  5  5       Young
## 2       2      US      F  45  3  5  2  5  5 Middle Aged
## 3       3      UK      F  25  3  5  5  5  2       Young
## 4       4      UK      M  39  3  3  4 NA NA Middle Aged
## 5       5      UK      F  NA  2  2  1  2  1        <NA>

5.3.1 One way to handle missing value

Deleting all observations with missing data ( Listwise deletion) is one of the several methods of handling incomplete datasets. Note: You can also replacing the missing value by the average of the remaining data.

is.na(leadership[,5:9])
##         q1    q2    q3    q4    q5
## [1,] FALSE FALSE FALSE FALSE FALSE
## [2,] FALSE FALSE FALSE FALSE FALSE
## [3,] FALSE FALSE FALSE FALSE FALSE
## [4,] FALSE FALSE FALSE  TRUE  TRUE
## [5,] FALSE FALSE FALSE FALSE FALSE

Removes all rows containing at least one missing value (NA) from the leadership dataset.

newdata<-na.omit(leadership)
newdata
##   manager country gender age q1 q2 q3 q4 q5      agecat
## 1       1      US      M  32  5  4  5  5  5       Young
## 2       2      US      F  45  3  5  2  5  5 Middle Aged
## 3       3      UK      F  25  3  5  5  5  2       Young

5.4 Sorting data

order(leadership$age) # Sort by age 
## [1] 3 1 4 2 5
newdata2<-leadership[order(leadership$age),]

newdata2
##   manager country gender age q1 q2 q3 q4 q5      agecat
## 3       3      UK      F  25  3  5  5  5  2       Young
## 1       1      US      M  32  5  4  5  5  5       Young
## 4       4      UK      M  39  3  3  4 NA NA Middle Aged
## 2       2      US      F  45  3  5  2  5  5 Middle Aged
## 5       5      UK      F  NA  2  2  1  2  1        <NA>

5.5 Merging data sets

To merge two data frame horizontally, you use merge() function. In most cases, two data frames are joined by one or more common key variables. -example 1: merge(dataframeA, dataframeB,by=“ID”) -example 2: merge(dataframeA,dataframeB, by=c(“ID”,“country”))

the second merge the two dataframes by ID and country

To join two data frame( datasets) vertically, use rbind() function: Note that the two data set must have the same variables

5.5.1 Subset (selecting variables,dropping variables,selecting observation )

newdata3<-leadership[,c(5:9)]
myvars<-c("q1","q2","q3","q4","q5")
(newdata3<-leadership[myvars])
##   q1 q2 q3 q4 q5
## 1  5  4  5  5  5
## 2  3  5  2  5  5
## 3  3  5  5  5  2
## 4  3  3  4 NA NA
## 5  2  2  1  2  1
newdata4<-leadership[,c(-1,-2)]
(newdata4<-leadership[,-(1:5)]) # exclude columns from 1 to 5
##   q2 q3 q4 q5      agecat
## 1  4  5  5  5       Young
## 2  5  2  5  5 Middle Aged
## 3  5  5  5  2       Young
## 4  3  4 NA NA Middle Aged
## 5  2  1  2  1        <NA>
newdata5<-leadership[,c(-1,-7)]
leadership[[4]]
## [1] 32 45 25 39 NA
leadership$age
## [1] 32 45 25 39 NA
newdata6<-leadership[c(-1,-3),]
newdata6<-leadership[c(2,4,5),]
newdata7<-subset(leadership,age>=35|age<24,select=c(q1,q2,q3,q4,q5))
attach(leadership)
## The following objects are masked _by_ .GlobalEnv:
## 
##     age, country, gender, manager, q1, q2, q3, q4, q5
#newdata7<-leadership[gender="M" & age>30,]
(newdata7<-leadership[gender=="M" & age>30,])
##   manager country gender age q1 q2 q3 q4 q5      agecat
## 1       1      US      M  32  5  4  5  5  5       Young
## 4       4      UK      M  39  3  3  4 NA NA Middle Aged
detach(leadership)