Probability and Statistics: Notes

Math 141

Author

Keira Palmer

Data Visualization

Numerical Graphs:

Histogram

one numerical variable.

geom_histogram()
Side-by-side box plot

one numerical variable and one categorical variable.1

geom_boxplot()
Side-by-side violin plot

one numerical variable and one categorical variable.

geom_violin
Scatter plot

two numerical variables

geom_point
Line graph

two numerical variables[^2] [^2]: For linear regression models, use geom_smooth().

geom_line

Categorical Graphs:

Bar plot

one categorical variable

Segmented bar plot

two categorical variables

Customizing Graphs:

Labelling:

ggplot(df, aes(x = x_example, y = y_example, color, fill)) + 
  geom_point(size = 2, alpha = 0.75, pch = 21, stroke = 0.65) +
  theme_bw() +
  labs(x = "text", 
       y = "text",
       title = "text",
       caption = "text") + 
  scale_color_manual() + 
  scale_fill_manual() +
  facet_grid() + 
  theme(legend.position = "right",
        strip.background = element_rect())

Statistical Analysis

Summary Statistics:

data |> 
  group_by(x) |> 
  summarize(mean = mean(y),
            median = median(y),
            stdev = sd(y),
            iqr = IQR(y),
            n = n())

Data Wrangling

Mutate

Add or modify new or existing variables. Syntax =

mutate(new = old)
Select

Extracts variables from a data frame. Syntax =

select(variables)
Filter

Extracts cases from a data frame. Syntax =

filter(conditional statement)
Arrange

Sort the cases from a data frame. Syntax =

arrange(n OR (desc(n)) OR variable)
Re-coding/Creating Variables

The function case_when() is used inside a function such as mutate to manipulate variables based on conditions

Handling Missing Data

Three kinds of NA removal, na.omit() is aggressive removal, drop_na() is moderate, and na.rm is light. You would append na.rm to the end of something like summarize().

Grouping

Creates separate columns in the output table based on the categories defined in the function

group_by()

Data Collection

Sampling Bias

When the sampled units are systematically different from the non-sampled units on the variables of interest.

Non-response bias

The respondents are systematically different from the non-respondents for the variables of interest.

Random Sampling:

Simple Random Sampling

Each option has an equal chance of being selected.

Cluster Sampling

Random number of all sample clusters are selected to be sampled.

Stratified Random Sampling

Data is divided into subgroups (strata) based on some characteristic and then a random sample is taken from each subgroup.

Systematic Sampling

Random starting point is chosen and then samples are systematically selected based on a number.

Representative Data:

Random sampling is important to ensure a sample is representative of the population

Generalizability

The degree to which you can apply the results of your study to a broader context.

Impact of Size

Small random samples will tend to be more representative than large non-random samples.

How to Analyze Non-random Samples? * Determine if your sampled cases (and respondents) are systematically different from the non-sampled cases (and non-respondents) for the variables you care about and adjust your population of interest.

Descriptive Analysis

Estimating quantities related to the population.

Predictive Analysis

Predict the value of a variable.

Causal Analysis

Determine if changes in a variable cause changes in another variable

Causal Inference: Methods for finding causal relationships even when the data is collected without random assignment. Spurious Relationship: Two variables are associated but not causally related.

Response Variable

The dependent variable that we are observing (y).

Explanatory/Predictor Variables

The independent variables that could explain the response (x).

Confounding Variable

A third variable that is associated with both the explanatory variable and the response variable.

Control

Group that gets no treatment or a standard treatment.

Placebo

A fake treatment given to a group to test whether they experience the desired effect regardless of receiving the treatment or if the treatment is any good.

Blinding

When the subject and/or researchers don’t know the explanatory group assignments. Double blinding describes when both the subject and researchers don’t know the assignments.

Statistical Modeling

Simple Linear Regression:

\[y = \beta_o +\beta_1x + \epsilon \\ \hat{y} = \hat{\beta_o} + \hat{\beta_1}x\]

\[\text{form of the model:}\ y = f(x) + \epsilon\]

Correlation Coefficient

measures the strength and direction of linear relationship between two quantitative variables. Its symbol is \(r\) and it is always between -1 and 1.

cor = cor(x, y)
  • The closer \(r\) is to -1 or 1, the stronger the hypothesized correlation is.
Parameters

Based on the population, if we don’t know the parameters then we don’t have data on the whole population. These would be \(\beta_o\) and \(\beta_1\)

Statistics

Based on the sample data, known, usually estimate a population parameter. These would be \(\hat{\beta_o}\) and \(\hat{\beta_1}\).

Method of Least Squares:

Fitted Value

The estimated value of the i-th case. Equation would be \(\hat{y_i} = \hat{\beta_o} + \hat{\beta_i}x_i\)

Residuals

The observed error term for the i-th case. Equation would be \(e_i = y_i - \hat{y_i}\). The goal is to pick values for \(\hat{\beta_o}\) and \(\hat{\beta_1}\) so that the residuals are small.

Minimizing residuals: \[\sum_{i=1}^{n}e^2_i \to \hat\beta_1 = \frac{\sum_{i=1}^{n}e^2_i(x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n}e^2_i(x_i - \bar{x})^2} \\ or \\ \hat\beta_o = \bar{y} - \hat\beta_1\bar{x} \to where\ \bar{y} = \frac{1}{n}\sum^n_{i=1}y_i\ and \ \bar{x} = \frac{1}{n}\sum^n_{i=1}x_i\]

Constructing in R:

Visualizing data and model:

ggplot(df, aes(x, y)) + 
  geom_point() + 
  geom_smooth(method = "lm", se = FALSE)
lm(y ~ x, data)
get_regression_table()
Intercept

We expect/predict \(y\) to be \(\hat\beta_o\) on average when \(x = 0\).

Slope

For a one-unit increase in \(x\), we expect/predict \(y\) to change by \(\hat\beta_1\) units on average.

Prediction

An estimation from a linear regression model.

Stay cautious to only predict values within the range of x values in the sample. Investigate outliers

new_df <- data.frame(df = conditional)
predict(mod, newdata = new_df)
Interpolation

Predicting values that are below values in the data frame.

Extrapolation

Predicting values that are above values in the data frame.

Assumptions:
Linearity

The relationship between the explanatory variable and response variables must be approximately linear.

geom_point(data, aes(x, y))
OR
ggplot(model) +
  stat_fitted_resid()
Independence

The observations should be independent of one another. Can be checked by considering the context of the data.

ggplot(model) +
  stat_fitted_resid()
Normality

The distribution of residuals should be approximately bell-shaped, uni-modal, symmetric, and centered at 0 at every “slice” of the explanatory variable.

Simple check:
ggplot(mod) + 
  stat_resid_hist()

Better check: 
ggplot(mod) + 
  stat_normal_qq()
Equal Variability

Variance of residuals should be roughly constant across the entire data set. Also called “homoscedasticity”. Models that violate this assumption are sometimes called “heteroscedastic”.

ggplot(mod) + 
  stat_fitted_resid()

ggplot(mod) + 
  stat_resid_hist()
Outliers

Values that fall far from the cloud of points. High leverage points skew data.

ggplot(mod) +
  stat_resid_leverage()

Other method of viewing diagnostic plots:

gglm(mod)

Multiple Linear Regression:

\[y = \beta_o + \beta_1x_1 + \beta_2x_2\ ... \ \beta_ix_i + \epsilon\]

Linear regression is a flexible class of models that allow for:

  • Both quantitative and categorical explanatory variables.
  • Multiple explanatory variables.
  • Curved relationships between the response variable and the explanatory variable.
  • BUT the response variable is quantitative.

Categorical explanatory variables must first be converted to numbers.

Similarities to SLR? * Still use Method of Least Squares to estimate coefficients. * Still use lm() to fit the model and predict() for prediction.

Differences from SLR? * Meaning of coefficients is more complicated and depend on other variables in the model. * Need to decide which variables to include and how (linear term, squared term…)

Visualizing:

ggplot(df, aes(x, y, color = variable)) + 
  geom_point()

Building Model:

same_slope <- lm(y ~ x_1 + x_2, df) # SAME SLOPES

diff_slope <- lm(y ~ x_1 * x_2, df) # DIFFERENT SLOPES

Appropriateness:

Equal Slopes

Forces the relationship between quantitative predictors and the response variable to be the same for each group in the model.

geom_parallel_slopes(se = FALSE)
Varying Slopes

Different slopes allow for different relationships between quantitative predictors and the response variable for each group in the model.

geom_smooth(method = "lm", se = FALSE)

Model Building Guidance

Guiding Principle

Capture the general trend, not the noise.

How to know which model is best? * Measure of quality: \(R^2\) (Coeffient of Determination). It is the percent of total variation in y explained by the model:

\[R^2 = 1 - \frac{\sum(y - \hat{y})^2}{\sum( y - \bar{y})^2}\]

  • Strategy: Compute the \(R^2\) value for each model and pick the one with the highest \(R^2\).

In R:

get_regression_summaries(mod) |> 
  select(r_squared)
  • \(R^2\) only increases as we add more variables.
Adjusted \(R^2\)

New measure of quality, this accounts for the number of explanatory variables in the model (\(p\)). \[R^2_{adjusted} = 1 - \frac{\sum(y - \hat{y})^2}{\sum( y - \bar{y})^2}(\frac{n-1}{n-p-1})\]

Strategy: Compute the \(R^2_{adjusted}\) value for each model and pick the one with the highest \(R^2_{adjusted}\).

Probability

Probability Theory:

It is the study and quantification of uncertainty/randomness in outcomes of repeated experiments.

Random Process

We know what results could happen, but not which will happen.

Can be used to model processes that are complicated, but not truly random, to figure out how they work.

Event

A potential result of some particular random process.

Probability

the probability of a particular event is the proportion of times the event would occur, if we observed the random process an arbitrarily large number of times.

Since probability is defined as a proportion, values will always be between 0 and 1.

Law of Large Numbers

Probability of an event refers to the long-run tendency of the proportion.

Observed proportions can deviate from expected probabilities in a finite number of trials, but as you increase the number of trials, the deviation decreases.

Probability Models:

There are two components in a probability model; 1. A list of all events of a random process. 2. A rule (called the probability function) that assigns each event a probability between 0 and 1, in a consistent manner.

Disjoint Events

Mutually exclusive events, aka, two events where it is impossible for both to occur at the same time.

Addition Rule

Only for disjoint events. For overlapping events, you must subtract the shared overlap (\(A \cap B\)) from the original addition rule.

Theorem 1 (Addition Rule) The probability that at least one event occurs in a pair of disjoint events is the sum of their individual probabilities: \[P(A \lor B ) = P(A) + P(B)\]

Complementary Events

The complement to an event \(A\) (denoted \(A^c\)) is the event that occurs exactly when the original does not.

Theorem 2 (Complement Rule) The probability that the complement of an event occurs is 1 minus the probability of the event: \[P(A^c) = 1-P(A)\]

(Because something either happens or doesn’t happen, the total probability of the sample space is always equal to 1.)

Conditional Probability

Defined as the probability of something occurring, given that another event has already occurred.

Theorem 3 (Conditional Probability Rule) The conditional probability of an event \(A\) given another event \(B\) is \[P(A|B) = \frac{P(A \cap B)}{P(B)}\]

Independence

We say two events are independent if knowing one occurs doesn’t change the probability that another occurs.

Theorem 4 (Criteria for Independence) Two events \(A\) and \(B\) are independent if and only if \[P(A|B) = P(A) \text{ and } P(B|A) = P(B)\]

Multiplication Rule

The probability that two events both occur

Theorem 5 (Multiplication Rule) For independent events \(A\) and \(B\),

\[P(A \cap B) = P(A) \cdot P(B)\]

For dependent events \(A\) and \(B\), \[P(A \cap B) = P(A|B) \cdot P(B)\]

Law of Total Probability

The sum of two probabilities both occurring is equal to the total probability. Useful for finding “overall” probabilities when we only have the “pieces”.

Law of Total Probability

Let \(A\) and \(B\) be events. Then \[P(A) = P(A|B)P(B) + P(A|B^c)P(B^c)\]

Bayes’ Rule

If you know \(P(B|A)\), you can find \(P(A|B)\) using Bayes’ Rule.

Theorem 6 (Bayes’ Rule) Let \(A\) and \(B\) be events. Then, \[P(A|B) = \frac{P(B|A)P(A)}{P(B)}\]

Proof. From the definition of multiplication rule, \[P(A \cap B) = P(A|B) \cdot P(B) \\ P(A \cap B) = P(B|A) \cdot P(A)\] Setting these two equations equal, we find \[P(A|B) \cdot P(B) = P(B|A) \cdot P(A)\]

\[P(A|B) = \frac{P(B|A)P(A)}{P(B)}\]

When extending Bayes’ Theorem, e.g., adding more categorical variables, the numerator is the product of probabilities for each intersection of categorical variables and the denominator is the Law of Total Probability.

Statistical Inference

Sampling Distribution

The distribution of a sample statistic from sample population parameters.

Confidence Interval

Interval of plausible values for a parameter.

  • The confidence level determines the width of the interval and affects how likely it is to contain the population parameter.
Bootstrapping

Estimating a sample statistic from a sample distribution. It isn’t from the direct population but rather a method for estimating proportions from that population.

Bootstrap Distribution

This is the estimated distribution for a sample statistic. The center of the distribution will be the known population sample statistic.

Parameters

Based on the population, these

Infer

Specifying Response (and Explanatory) Variables - specify()

Defines what variables you are interested in and outputs the results as a table.

#if interested in one variable: 
data %>%
  specify(y)

#if interested in two variables:
data %>% 
  specify(y ~ x) 

#or if computing a proportion or difference in proportions, you need to define what the success result is:
data %>% 
  specify(y, success = "")
Declaring the Null Hypothesis - hypothesize()

Assuming either independence or a point estimate, the hypothesize() function defines the null hypothesis based on the given parameter.

#for "independence"
data %>% 
  specify(y ~ x, success = "") %>% 
  hypothesize(null = "independence") 

#or for "point"
data %>% 
  specify(y) %>% 
  hypothesize(null = "point", mu = 40) #replace 40 with whatever value is the point estimate

Generating the Null Distribution - generate()

#SIMULATION-BASED INFERENCE

test_stat <- data %>% 
  specify(y ~ x) %>% 
  fit() or calculate(stat = "blank")

null <- data %>%
  specify(y ~ x) %>% 
  hypothesise(null = "independence/point") %>% 
  generate(reps = 1000, type = "permute/bootstrap") %>% 
  fit() or calculate(stat = "blank")
  
p_value <- null %>% 
  get_p_value(obs_stat = test_stat, direction = "two.sided")

boot_pages <- data %>% 
  specify(y ~ x) %>% 
  hypothesise(null = "independence/point") %>% 
  generate(reps = 1000, type = "bootstrap") %>% 
  fit() or calculate(stat = "blank")

ci_pages <- boot_pages %>% 
  get_ci(point_estimate = test_stat, type = "percentile", level = 0.95)

Footnotes

  1. Overlay points on a boxplot using geom_jitter().↩︎