Outline
Goal: Learn how to use simulation to answer statistical questions!
In science, how do we support or refute a hypothesis?
Instead, we can run a simulation.
Simulation is an important (and big) topic for both statistics and for a variety of other areas where there is a need to introduce randomness.
Sometimes you want to implement a statistical procedure that requires random number generation or sampling (i.e. Markov chain Monte Carlo, the bootstrap, random forests, bagging) and sometimes you want to simulate a system and random number generators can be used to model random inputs.
R comes with a set of pseuodo-random number generators that allow you to simulate from well-known probability distributions like the Normal, Poisson, and binomial.
Some example functions for probability distributions in R.
For each probability distribution there are typically four functions available that start with a “r”, “d”, “p”, and “q”.
The “r” function is the one that actually simulates random numbers from that distribution.
The other functions are prefixed with a
If you’re only interested in simulating random numbers, then you will likely only need the “r” functions and not the others.
Probably the most common probability distribution to work with the is the Normal distribution (also known as the Gaussian). Working with the Normal distributions requires using these four functions.
## [1] 1.6400096 1.0361779 -1.1159705 -1.3423796 0.8128874 1.5461313
## [7] -1.8693206 -0.6347188 -1.1626086 -0.5812986
## [1] 0.4819232 -0.2771778 -0.2084614 0.7621179 -1.1332870 -0.2302228
## [7] 0.5480552 -0.3150096 0.2005406 0.1565243
## [1] -0.6264538 0.1836433 -0.8356286 1.5952808 0.3295078 -0.8204684
## [7] 0.4874291 0.7383247 0.5757814 -0.3053884
## [1] -0.6264538 0.1836433 -0.8356286 1.5952808 0.3295078 -0.8204684
## [7] 0.4874291 0.7383247 0.5757814 -0.3053884
## [1] 18.74709 20.36729 18.32874 23.19056 20.65902 18.35906 20.97486 21.47665
## [9] 21.15156 19.38922
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 18.33 18.91 20.51 20.26 21.11 23.19
It is possible to generate random numbers from other probability distributions like the Poisson. The Poisson distribution is commonly used to model data that come in the form of counts.
rpois(n, lambda)
## [1] 0 1 1 2 0 2 3 1 1 0
## [1] 16 18 19 30 23 16 17 18 18 21
## [1] 1 2 2 3 1 3 4 2 2 0
## [1] 2
## [1] 2.49
We can individually generate data and calculate sample means. But this doesn’t give us information about the variance of these means.
rexp(n, rate = 1)
## [1] 1.669803
## [1] 0.9870225
## [1] 1.026968
## [1] 1.014509
What is the probability of flipping \(5\) coins and getting all heads?
What is the probability of flipping \(5\) coins and getting all heads?
## [1] 0.03125
What is the probability of flipping \(5\) coins and getting all heads?
## [1] 0.0325
## [1] 0.03164
For a set of random variables \(X_1,\ldots,X_n\), each independent with variance \(\sigma^2\), what is the variance of the sample mean?
\[\begin{align*} Var(\bar{X}) &= Var(\frac{1}{n}\sum_{i=1}^n X_i) \\ &= \frac{1}{n^2}\sum_{i=1}^n Var(X_i) \\ &= \frac{n\sigma^2}{n^2} \\ &= \frac{\sigma^2}{n} \end{align*}\]
For a set of random variables \(X_1,\ldots,X_n\), each independent with variance \(\sigma^2\), what is the variance of the sample mean?
We can individually generate data and calculate sample means. But this doesn’t give us information about the variance of these means.
## [1] 1.489288
## [1] 0.8978229
## [1] 1.028234
Simulating random numbers is useful but sometimes we want to simulate values that come from a specific model.
For that we need to specify the model and then simulate from it using the functions described above.
Suppose we want to simulate from the following linear model with intercept 0.5 and slope 2. The variable x might represent an important predictor of the outcome y. Here’s how we could do that in R.
set.seed(1)
x <- rnorm(100, 0, 1) ## Simulate predictor variable
e <- rnorm(100, 0, 2) ## Simulate the error term
y <- 0.5 + 2 * x + e ## Compute the outcome via the model
summary(y)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## -5.4816 -1.1493 0.7582 0.6422 2.3404 6.1534
##
## Call:
## lm(formula = y ~ x)
##
## Residuals:
## Min 1Q Median 3Q Max
## -3.754 -1.228 -0.279 1.079 4.692
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 0.4246 0.1940 2.189 0.031 *
## x 1.9979 0.2155 9.273 4.58e-15 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 1.926 on 98 degrees of freedom
## Multiple R-squared: 0.4674, Adjusted R-squared: 0.4619
## F-statistic: 85.99 on 1 and 98 DF, p-value: 4.583e-15
## [1] 1 1 1 1 1 1 0 0 1 1 0 1 0 0 0 0 0 1 1 0
##
## Shapiro-Wilk normality test
##
## data: scores
## W = 0.95327, p-value = 0.4195
##
## Two Sample t-test
##
## data: scores by groups
## t = -0.074459, df = 18, p-value = 0.9415
## alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
## 95 percent confidence interval:
## -1.834543 1.708957
## sample estimates:
## mean in group 0 mean in group 1
## 10.34651 10.40930
##
## Welch Two Sample t-test
##
## data: scores by groups
## t = -0.071915, df = 14.06, p-value = 0.9437
## alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
## 95 percent confidence interval:
## -1.934786 1.809200
## sample estimates:
## mean in group 0 mean in group 1
## 10.34651 10.40930
We have already seen and used several functions, but you can also create your own! This is incredibly useful when:
function_name: the name you want to give your function,
what you will use to call itfunction(): call this to define a functionparam1, param2: function parameters, what
the user inputs. You can assign default values by setting them equal to
something in the function definitionreturn(): is what your function will return to the
user.## [1] 18
## [1] 18
## [1] 18
## [1] 102
## [1] 3 6 11 18 27
fahr_to_kelvin <- function(fahr) {
# function that converts temperature in degrees Fahrenheit to kelvin
# input: fahr: numeric value representing temp in degrees farh
# output: kelvin: numeric converted temp in kelvin
kelvin <- ((fahr - 32) * (5 / 9)) + 273.15
return(kelvin)
}
fahr_to_kelvin(32)## [1] 273.15
fahr_to_kelvin <- function(fahr) {
# function that converts temperature in degrees Fahrenheit to kelvin
# input: fahr: numeric value representing temp in degrees farh
# output: kelvin: numeric converted temp in kelvin
kelvin <- ((fahr - 32) * (5 / 9)) + 273.15
return(kelvin)
}
fahr_to_kelvin(212)## [1] 373.15
## [1] 555.3722
my_function <- function(x, y) {
w1 <- x + y
w2 <- x * y
return(list(w1, w2))
}
# If we apply the function, we get the following list output:
my_function (x = 5, y = 3) ## [[1]]
## [1] 8
##
## [[2]]
## [1] 15
my_function <- function(x) {
n <- length(x)
mean <- sum (x) / n
sd <- sqrt(sum((x - mean(x))^2) / (n - 1))
return(list(mean, sd))
}
# If we apply the function, we get the following list output:
x <- c(10, 20, 30, 40, 50)
my_function(x) ## [[1]]
## [1] 30
##
## [[2]]
## [1] 15.81139
We can create user-defined functions in R. They are specific to what a user wants and once created they can be used like the built-in functions.
Below is an example of how a function is created and used.
## Create a function to print squares of numbers in sequence.
new.function <- function(a) {
for(i in 1:a) {
b <- i^2
print(b)
}
}
new.function(3)## [1] 1
## [1] 4
## [1] 9
## Create a function to print squares of numbers in sequence.
new.function <- function(a) {
for(i in 1:a) {
b <- i^2
print(b)
}
}
new.function(6)## [1] 1
## [1] 4
## [1] 9
## [1] 16
## [1] 25
## [1] 36
# Create a function without an argument.
new.function <- function() {
for(i in 1:5) {
print(i^2)
}
}
# Call the function without supplying an argument.
new.function()## [1] 1
## [1] 4
## [1] 9
## [1] 16
## [1] 25
# Create a function with arguments.
new.function <- function(a, b, c) {
result <- a * b + c
print(result)
}
# Call the function by position of arguments.
new.function(5, 3, 11)## [1] 26
## [1] 58
quadraticRoots <- function(a, b, c) {
print(paste0("You have chosen the quadratic equation ", a, "x^2 + ", b, "x + ", c, "."))
discriminant <- (b^2) - (4*a*c)
if(discriminant < 0) {
return(paste0("This quadratic equation has no real numbered roots."))
}
else if(discriminant > 0) {
x_int_plus <- (-b + sqrt(discriminant)) / (2*a)
x_int_neg <- (-b - sqrt(discriminant)) / (2*a)
return(paste0("The two x-intercepts for the quadratic equation are ",
format(round(x_int_plus, 5)), " and ",
format(round(x_int_neg, 5)), "."))
}
else #discriminant = 0 case
x_int <- (-b) / (2*a)
return(paste0("The quadratic equation has only one root. This root is ",
x_int))
}
quadraticRoots(1, 0, 5)## [1] "You have chosen the quadratic equation 1x^2 + 0x + 5."
## [1] "This quadratic equation has no real numbered roots."
quadraticRoots <- function(a, b, c) {
print(paste0("You have chosen the quadratic equation ", a, "x^2 + ", b, "x + ", c, "."))
discriminant <- (b^2) - (4*a*c)
if(discriminant < 0) {
return(paste0("This quadratic equation has no real numbered roots."))
}
else if(discriminant > 0) {
x_int_plus <- (-b + sqrt(discriminant)) / (2*a)
x_int_neg <- (-b - sqrt(discriminant)) / (2*a)
return(paste0("The two x-intercepts for the quadratic equation are ",
format(round(x_int_plus, 5)), " and ",
format(round(x_int_neg, 5)), "."))
}
else #discriminant = 0 case
x_int <- (-b) / (2*a)
return(paste0("The quadratic equation has only one root. This root is ",
x_int))
}
quadraticRoots(1, 7, 5)## [1] "You have chosen the quadratic equation 1x^2 + 7x + 5."
## [1] "The two x-intercepts for the quadratic equation are -0.80742 and -6.19258."
quadraticRoots <- function(a, b, c) {
print(paste0("You have chosen the quadratic equation ", a, "x^2 + ", b, "x + ", c, "."))
discriminant <- (b^2) - (4*a*c)
if(discriminant < 0) {
return(paste0("This quadratic equation has no real numbered roots."))
}
else if(discriminant > 0) {
x_int_plus <- (-b + sqrt(discriminant)) / (2*a)
x_int_neg <- (-b - sqrt(discriminant)) / (2*a)
return(paste0("The two x-intercepts for the quadratic equation are ",
format(round(x_int_plus, 5)), " and ",
format(round(x_int_neg, 5)), "."))
}
else #discriminant = 0 case
x_int <- (-b) / (2*a)
return(paste0("The quadratic equation has only one root. This root is ",
x_int))
}
quadraticRoots(2, 4, 2)## [1] "You have chosen the quadratic equation 2x^2 + 4x + 2."
## [1] "The quadratic equation has only one root. This root is -1"
quadraticRoots <- function(a, b, c = 1) {
print(paste0("You have chosen the quadratic equation ", a, "x^2 + ", b, "x + ", c, "."))
discriminant <- (b^2) - (4*a*c)
if(discriminant < 0) {
return(paste0("This quadratic equation has no real numbered roots."))
}
else if(discriminant > 0) {
x_int_plus <- (-b + sqrt(discriminant)) / (2*a)
x_int_neg <- (-b - sqrt(discriminant)) / (2*a)
return(paste0("The two x-intercepts for the quadratic equation are ",
format(round(x_int_plus, 5)), " and ",
format(round(x_int_neg, 5)), "."))
}
else #discriminant = 0 case
x_int <- (-b) / (2*a)
return(paste0("The quadratic equation has only one root. This root is ",
x_int))
}
quadraticRoots(2, 4, 2)## [1] "You have chosen the quadratic equation 2x^2 + 4x + 2."
## [1] "The quadratic equation has only one root. This root is -1"
quadraticRoots <- function(a, b, c = 1) {
print(paste0("You have chosen the quadratic equation ", a, "x^2 + ", b, "x + ", c, "."))
discriminant <- (b^2) - (4*a*c)
if(discriminant < 0) {
return(paste0("This quadratic equation has no real numbered roots."))
}
else if(discriminant > 0) {
x_int_plus <- (-b + sqrt(discriminant)) / (2*a)
x_int_neg <- (-b - sqrt(discriminant)) / (2*a)
return(paste0("The two x-intercepts for the quadratic equation are ",
format(round(x_int_plus, 5)), " and ",
format(round(x_int_neg, 5)), "."))
}
else #discriminant = 0 case
x_int <- (-b) / (2*a)
return(paste0("The quadratic equation has only one root. This root is ",
x_int))
}
quadraticRoots(2, 4)## [1] "You have chosen the quadratic equation 2x^2 + 4x + 1."
## [1] "The two x-intercepts for the quadratic equation are -0.29289 and -1.70711."
A common approach here is the two-sample t-test for difference of means. Can we use this method here? What assumptions does this test require?
Usually:
Observations must be independent of each other.
Each population should be (approximately) normally distributed.
If yes, proceed with the parametric test.
Parametric methods: we assume that sample data comes from a population that we can model from a probability distribution with a finite number of parameters.
If not, proceed with a non-parametric test.
Nonparametric methods: we do not assume an explicit form for the distribution of our data.
Permutation tests use computer simulations to find a p-value instead of relying on the theory of probability distributions like the normal distribution.
To get a p-value, we randomly sample (without replacement) possible permutations of our variable of interest. The p-value is the proportion of samples that have a test statistic larger than that of our observed data.
Here are the steps for a permutation test.
Step 1: calculate the statistic of interest.
Step 2: take all of the results in both groups and simulate randomly reassigning them to the two groups. Do this many times and keep a record of the outcomes for the simulated statistic of interest.
Step 3: the p-value is the percent of simulations with a statistic of interest as extreme or more than the actual value calculated in Step 1.
## weight group
## 1 4.17 ctrl
## 2 5.58 ctrl
## 3 5.18 ctrl
## 4 6.11 ctrl
## 5 4.50 ctrl
## 6 4.61 ctrl
##
## ctrl trt1 trt2
## 10 10 10
t.test(x = PlantGrowth$weight[PlantGrowth$group == "ctrl"],
y = PlantGrowth$weight[PlantGrowth$group == "trt2"])##
## Welch Two Sample t-test
##
## data: PlantGrowth$weight[PlantGrowth$group == "ctrl"] and PlantGrowth$weight[PlantGrowth$group == "trt2"]
## t = -2.134, df = 16.786, p-value = 0.0479
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -0.98287213 -0.00512787
## sample estimates:
## mean of x mean of y
## 5.032 5.526
# remove treatment 1
PlantGrowth <- PlantGrowth %>%
filter(group != "trt1")
obs_diff <- mean(PlantGrowth$weight[PlantGrowth$group == "trt2"]) -
mean(PlantGrowth$weight[PlantGrowth$group == "ctrl"])
obs_diff## [1] 0.494
# store permuted differences
permuted_diff <- numeric()
for (i in 1:10000) {
# for each iteration, scramble group variable
permuted_data <- PlantGrowth %>%
mutate(group = sample(group))
# compute difference
diff = mean(permuted_data$weight[permuted_data$group == "trt2"]) -
mean(permuted_data$weight[permuted_data$group == "ctrl"])
# store difference
permuted_diff[i] <- diff
}
head(permuted_diff)## [1] -0.306 -0.296 0.226 0.374 -0.116 -0.552
## [1] 0.0484
Suppose we have two numerical variables, x and
y, and we want to test whether they are correlated. What
should we do?
We could use some sort of parametric test, or …
set.seed(1111111)
sim_data <- data.frame(x = rnorm(11),
y = c(rnorm(11)))
ggplot(sim_data, aes(x = x, y = y)) +
geom_point()##
## Pearson's product-moment correlation
##
## data: sim_data$x and sim_data$y
## t = -0.32592, df = 9, p-value = 0.7519
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
## -0.6648080 0.5259451
## sample estimates:
## cor
## -0.108006
# store correlations
permuted_cor <- numeric()
for (i in 1:10000) {
permuted_data <-
data.frame(x = sim_data$x,
y = sample(sim_data$y))
permuted_cor[i] <- cor(permuted_data$x,
permuted_data$y)
}
permuted_cor_df <- data.frame(cor = permuted_cor)
head(permuted_cor_df)## cor
## 1 -0.3789125
## 2 -0.6887097
## 3 -0.1764174
## 4 0.2357416
## 5 0.1888525
## 6 0.2586279
## [1] 0.7433
set.seed(302)
sim_data <- data.frame(x = rnorm(30, mean = 0, sd = 1))
sim_data$y <- sim_data$x + rnorm(30, mean = 0, sd = 2)
ggplot(sim_data, aes(x = x, y = y)) +
geom_point()## [1] 0.5432401
##
## Pearson's product-moment correlation
##
## data: sim_data$x and sim_data$y
## t = 3.4238, df = 28, p-value = 0.001921
## alternative hypothesis: true correlation is not equal to 0
## 95 percent confidence interval:
## 0.2274944 0.7556241
## sample estimates:
## cor
## 0.5432401
permuted_cor <- numeric()
for (i in 1:10000) {
permuted_data <-
data.frame(x = sim_data$x,
y = sample(sim_data$y))
permuted_cor[i] <- cor(permuted_data$x,
permuted_data$y)
}
permuted_cor_df <- data.frame(cor = permuted_cor)
head(permuted_cor_df)## cor
## 1 -0.17228945
## 2 -0.22716410
## 3 0.21496325
## 4 0.28050753
## 5 0.06209496
## 6 -0.44889433
## [1] 0.0017
We’ll now learn about loops and some more efficient/syntactically simple loop alternatives
loops are ways of iterating over data
For loops iterate along an input vector, stores the current value of the vector as a variable, and repeatedly evaluates a code chunk until the vector is exhausted.
## [1] 1 1
## [1] 2 4
## [1] 3 9
## [1] 4 16
## [1] 5 25
## [1] 1
## [1] 2
## [1] 3
## [1] 4
## [1] 5
## [1] 6
## [1] 7
## [1] 8
## [1] 1
## [1] 2
## [1] 3
## [1] 4
## [1] 5
## [1] 6
## [1] 7
## [1] 8
## [1] 1
## [1] 4
## [1] 9
## [1] 16
## [1] 25
## [1] 36
## [1] 49
## [1] 64
## [1] 1 4 9 16 25 36 49 64
## [1] 25.5
## [1] 22.58318
## [1] 1 64 484 1600 25 36 49 64
## [1] 290.375
## [1] 551.9097
## [1] 0 1
## [1] 0 1 2
## [1] 0 1 2 3
## [1] 0 1 2 3 4
## [1] 0 1 2 3 4 5
## [1] 1 1
## [1] 1 1 2
## [1] 1 1 2 3
## [1] 1 1 2 3 4
## [1] 1 1 2 3 4 5
## [1] 12 1
## [1] 12 1 2
## [1] 12 1 2 3
## [1] 12 1 2 3 4
## [1] 12 1 2 3 4 5
## [1] "zero"
## [1] "one"
## [1] "two"
## [1] "three"
## [1] "four"
## [1] "five"
phrase <- "Good Night,"
for(word in c("and", "Good", "Luck")) {
phrase <- paste(phrase, word)
print(phrase)
}## [1] "Good Night, and"
## [1] "Good Night, and Good"
## [1] "Good Night, and Good Luck"
index.set <- list(name="Michael", weight=185, is.male=TRUE) # a list
for(i in index.set) {
print(c(i, typeof(i)))
}## [1] "Michael" "character"
## [1] "185" "double"
## [1] "TRUE" "logical"
## [1] 2
## [1] 3
## [1] 3
## [1] 4
## [1] 4
## [1] 5
## [1] 1
## [1] 2
## [1] 2
## [1] 4
## [1] 3
## [1] 6
## [1] 1
## [1] 4
## [1] 2
## [1] 8
## [1] 3
## [1] 12
## [1] 2
## [1] 4
## [1] 12
## [1] 48
## [1] 240
## [1] 240
## [1] 2
## [1] 4
## [1] 12
## [1] 48
## [1] 240
## [1] 1440
## [1] 10080
## [1] 80640
## [1] 725760
## [1] 7257600
## [1] 79833600
## [1] 958003200
## [1] 12454041600
## [1] 174356582400
## [1] 2.615349e+12
## [1] 4.184558e+13
## [1] 7.113749e+14
## [1] 1.280475e+16
## [1] 2.432902e+17
## [1] 4.865804e+18
## [1] 4.865804e+18
## [1] NA NA NA NA NA
## [1] 2 4 6 8 10
## [1] 2 4 6 8 10
## j
## [1,] 10 100 50
## [,1]
## j 10
## 100
## 50
## A
## [1,] 1 1 2 3 4 5 6 7 8 9 10
## [2,] 1 1 4 9 16 25 36 49 64 81 100
## [3,] 1 5 10 15 20 25 30 35 40 45 50
## [,1] [,2] [,3]
## A 1 1 1
## 1 1 5
## 2 4 10
## 3 9 15
## 4 16 20
## 5 25 25
## 6 36 30
## 7 49 35
## 8 64 40
## 9 81 45
## 10 100 50
## [,1] [,2] [,3]
## [1,] 1 2 3
## [2,] 2 4 6
## [3,] 3 6 9
## [4,] 4 8 12
## [,1] [,2] [,3]
## [1,] 1 2 3
## [2,] 1 2 3
## [3,] 1 2 3
## [4,] 1 2 3
## [,1] [,2] [,3]
## [1,] 1 2 3
## [2,] 2 4 6
## [3,] 3 6 9
## [4,] 4 8 12
We won’t really be using while loops in this class
Just be aware that they exist, and that they may become useful to you at some point in your analytics career
## [1] 15
## [1] 20
## [1] 25
## [1] 30
## [1] 30
## [1] 9
## [1] 13
## [1] 17
## [1] 21
## [1] 21
## [1] 5 9
## [1] 5 9 13
## [1] 5 9 13 17
## [1] 5 9 13 17 21
## [1] 21
## [1] 5 9 13 17 21
## [1] -0.9619334
## [1] -0.2925257
## [1] 0.2587882
## [1] 0.2587882