Day 1

Purpose of the lectures

• Introduction to using statistical methods for quantitative analysis
• The statistical ‘turn’: Using statistical methods to
  • Unearth patterns and Seek generalizations
  • Model behavior in quantitative terms
  • Using model-theoretic approaches along with empirical methods
  • Model-fitting

What we will learn

• Designing our own studies
• Hypothesis testing: One of the many ways of approaching quantitative issues; perhaps the most important way
• What kind of statistical tests need to be performed BEFORE you set out to collect your data
• Perform statistical tests and analyses
• How to present your results
• Data visualization techniques to advance your arguments

Topics

• Central limit function
• Normal, chi-square and F-distribution
• Analysis of Variance(ANOVA)
• Linear and logistic regression
• *Principal component analysis (PCA)
• *Multi-dimensional scaling (MDS)
• *Mixed-model effects

Quantiative aspects of any analysis

• Why quantitative?
• Many qualitative aspects in ways we conduct our analyses in any discipline we are in
• In many cases, the observations we have are not single instances of any event
• Measure how consistently an event and/or its iterations produce uniformity
• Or if there are any conditions that affect this uniformity
• Size of our observations makes it difficult to qualitatively evaluate how the observations we make are drawn from the data we have
• Being able to apply quantitative methods, essentially changes the nature of the questions we can ask

Testing our hypotheses

• The nature of the questions we ask has changed
• We often want to ask questions about mean values
• Is this average chargesheeted_male different from that chargesheeted_female? This is a trick question.
• These all boil down to the question is \(\overline{x}\) (the mean of the data xi) different from ȳ (the mean of the data yi)?
• What can my data tell me about the world?

Hypotheses

• Hypothesis1: Variable A will be larger, bigger, stronger than variable B
• Hypothesis2: Variable B will be larger, bigger, stronger than variable A
• Hypothesis0 (null hypothesis): There will be no difference between Variable A and Variable B
• What statistical test will help us reject the Null hypothesis

Sampling

• Whether qualitative or quantitative
  • Observations should lead to generalizations
  • Our ability to generalize our findings depends on having a representative sample of data
  • good statistical analysis can’t overcome sampling inadequacy
  • The observations that we are exploring consist of complex interactions of potentially many important variables
  • The balancing act that we attempt in research is to stay aware of the complexity, but not let it keep us from seeing the big picture.

Samples and populations

• Making inferences about population from finite samples
• What is the probability (not random chance) that our samples represent the population
• What is the probability that the differences between two variables are significantly different from each other and NOT different by chance

Statistical environment

• R because it is:
  • a powerful statistics package, good at reading data, wide range of statistical tests and techniques, good graphics, very flexible
  • a usable package available for many platforms (PC, Mac, Unix, Linux…. ) programmable user community for support
  • it is noncommercial - distributed under the GNU “copyleft”, maintained by a community of users, upgrades happen because the users need improvements, not because the company needs more money.
  • Where: R project page - http://www.r-project.org/
• How:
  1. Go to the R project page,
  2. click the CRAN link to see the download servers on the Comprehensive R Archive Network,
  3. choose a download server near your location
  4. choose your platform (Windows, Linux, Mac)
• Front end https://posit.co/products/open-source/rstudio/

Describing data

ncrb <- read.csv("NCRB-2021.csv", header=TRUE)
hist(ncrb$Arrested_Total)

head(ncrb)

##   Sl..No.                                          crime_head Arrested_Male
## 1       1                                              Murder          4907
## 2       2           Culpable Homicide not amounting to Murder           385
## 3       3                         Causing Death by Negligence          6261
## 4     3.1 Deaths due to Negligence relating to Road Accidents          5177
## 5   3.1.1                                         Hit and Run          1880
## 6   3.1.2                                     Other Accidents          3297
##   Arrested_Female Arrested_Transgender Arrested_Total Chargesheeted_Male
## 1             178                    2           5087               3950
## 2              12                    0            397                333
## 3             160                    0           6421               6202
## 4             150                    0           5327               5208
## 5              43                    0           1923               1875
## 6             107                    0           3404               3333
##   Chargesheeted._Female Chargesheeted_Transgender Chargesheeted_Total
## 1                   141                         2                4093
## 2                     9                         0                 342
## 3                   154                         0                6356
## 4                   143                         0                5351
## 5                    37                         0                1912
## 6                   106                         0                3439
##   Convicted_Male Convicted_Female Convicted_Transgender Convicted_Total
## 1            446               20                     0             466
## 2             44                0                     0              44
## 3           1118               47                     0            1165
## 4            772               46                     0             818
## 5            338               22                     0             360
## 6            434               24                     0             458
##   Discharged_Male Discharged_Female Discharged_Transgender Discharged_Total
## 1              16                 0                      0               16
## 2               4                 0                      0                4
## 3              61                 0                      0               61
## 4              58                 0                      0               58
## 5               6                 0                      0                6
## 6              52                 0                      0               52
##   Acquitted_Male Acquitted_Female Acquitted_Transgender Acquitted_Total
## 1            592               29                     0             621
## 2             53                0                     0              53
## 3           1184               14                     0            1198
## 4            734               11                     0             745
## 5            264                6                     0             270
## 6            470                5                     0             475

male_data=ncrb$Arrested_Male

female_data=ncrb$Arrested_Female

plot(male_data,female_data, main="Male Female data of arrests", xlab="Total number of males arrested", ylab="Total number of females arrested")

Generating data

x = round(rnorm(36,4.5,2))
x = rnorm(36, 4.5, 2)#notice that this is different from round(rnorm(36,4.5,2)) where we had asked for rounded/integer values
x

##  [1]  3.97140775  5.08689321  5.17729375  8.20023746  3.06957316  5.79466757
##  [7]  4.66753117  7.19614977  6.94237176  1.02516118  3.60091181  7.08618677
## [13]  6.03737347  2.18349699  3.84640617  3.94052865  4.06714608  4.97069518
## [19]  3.27752935  1.15228733 -0.01937894  5.10308874  3.35191410  2.58478757
## [25]  8.60667044  1.08066772  4.42827309  7.71344363  5.07283930  1.41772009
## [31]  4.41576347  4.16155545  2.77593139  3.45994160  4.48977290  6.74695488

hist(x,breaks=30000, xlim = c(0,10))

hist(x, xlim = c(0,10))

#Lets look at another example
x = rnorm(10000, 4.5, 2)
hist(x,breaks=100,freq=FALSE,xlim = c(0,10))
plot(function(x)dnorm(x, mean=4.5, sd=2), 0,10, add=TRUE)

Frequency distribution in R

• How did we generate these plots and distributions
• First we defined a vector using the function, rnorm
• x = rnorm(36, 4.5, 2) #notice that this is different from round(rnorm(36,4.5,2)) where we had asked for rounded/integer values
• We defined a vector, x, with 36 values, a mean of 4.5 and standard deviation of 2. So decimal values would be ok
• Then we made two histograms
  • First with: hist(x,breaks=30000, xlim = c(0,10)) #here we want to plot a histogram where the width of the cells/bins is very small
  • Second with: hist(x, xlim = c(0,10)) #here we want to plot a histogram where the width of the cells/bins is determined by R

Theoretical distributions

• Suppose we could draw from an infinite data set.
• The larger our data set - more detailed a representation of the frequency distribution
• If we keep collecting data for the some event, so that instead of data from x events we have data from 10,000 events
• With a histogram that has 1000 bars in it, we see that ratings near 4.5 are more common than those at the edges of the rating scale
• Adding observations up to infinity and reducing the size of the bars in the histogram of the frequency distribution
• Intervals between bars is vanishingly small - i.e. we end up with a continuous curve, almost
• Plotting the normal distribution curve on the frequency distribution

Adding the normal curve

• Why the excellent fit between the “observed” and the theoretical distributions?

• The data is generated by random selection (rnorm()) of observations from the theoretical normal distribution (dnorm())

• The “normal distribution” is an very useful theoretical function because…

  • Let’s assume that there is an underlying property that we are trying to measure - like amount of processing time
  • Also assume that there is some source of random error that makes it difficult for us to get at this underlying property

• If so, then we can think that - the “true” value of the underlying property we want to measure –

  • Must be at the center of the frequency distribution that we observe in our measurements -And, the distribution (we observe) is caused by error - with (the probability of) bigger errors being less likely than smaller errors

  hist(x,breaks=1000,freq=FALSE,xlim = c(0,10))
  plot(function(x)dnorm(x, mean=4.5, sd=2), 0,10, add=TRUE)

  

Day 2

The normal distribution

• Simply put, the normal distribution is described by the normal curve, or the bell-shaped curve
• It is an exponential function of the mean value (\(μ\) “mew”) and the variance (\(σ\) “sigma”)
• The sum of the area under the curve, fx is 1
• Derived from just two numbers, the mean value and a measure of how variable the data are
• The area under the curving equaling to 1, is also useful to go from frequency distributions to probability densities
• The ‘e’ stands for the Euler’s constant = 2.718
• This is related to hypothesis testing (more later)

Distribution types - Uniform

• Uniform distribution: Every outcome is equally likely. Example: Six sides of a dice - equal likelihood of rolling either side (technically).

plot(function(x)dunif(x,min=-3,max=3), -3,3, main="Uniform")

Distribution - Normal

• Normal - bell-shaped - distribution, measurements gather around a typical value and values become less and less likely as they deviate from the central value

plot(function(x)dnorm(x), -3,3, main="Normal")

Distribution - Skewed right

• Skewed frequency distributions appear when dealing with percentage data
• The mean is no longer ‘central’ to the distribution. Extreme values (from one end of the scale and less from the other) dominate the distribution

plot(function(x)df(x, 3, 100), 0,4, main="Skewed right")

Distribution - J-shaped

• The J-shaped distribution is a special type of skewed distribution where most observations come from the very end of the measurement scale.
• Consider, Uber ratings? Most of us end up giving a 5 star rating. Or even product reviews on Amazon.

plot(function(x)df(x, 1, 10)/3, 0.2,3, main="J-Shaped")

Distribution - Bimodal

• Bimodal distribution is a frequency distribution where clearly two modalities are involved.
• For instance, the persistent pay gap between men and women in athletics and sport

plot(function(x)(dnorm(x, mean=3, sd=1)+dnorm(x, mean=-3, sd=1))/2,-6,6, main="Bimodal")

Distribution - U Shaped

• U shaped distributions result out of polarization where subjects may take drastically one view or the other

plot(function(x)-dnorm(x),-3,3, main="U shaped")

Measures of central tendency

• In the normal distribution, we feel that the central ‘values’ (from our samples) have the highest probability of being part of the population
• What are these?
  • The most frequently occurring value is the mode - the tip of the frequency distribution. -For the skewed distribution here, the mode is at about 0.6
  • The central value, that is in an ordered dataset of the values, the one in the middle is the median; aka, the center of gravity
  • Arithmetic mean, or the sum of values divided by the total number of values, n
• Least squares estimate of central tendency
  1. take the difference between the mean and each value in our data set
  2. square these differences and
  3. add them up, -We will get a value that will be smaller than what we would get if we took the median or any other estimate of the “mid-point” of the data set —

plot(function(x)df(x, 5, 100),0,4, main="Measures of central tendency")
lines(x=c(0.6,0.6),y=c(0,df(0.6,5,100)))
skew.data <-rf(10000,5,100)
lines(
  x=c(mean(skew.data),mean(skew.data)),
  y=c(0,df(mean(skew.data),5,100)))
lines(
  x=c(median(skew.data),median(skew.data)),
  y=c(0,df(median(skew.data),5,100))
)
text(1,0.75,labels="mode")
text(1.3,0.67,labels="median")
text(1.35,0.6,labels="mean")

Weighted means

• Means represent the least squared estimate of the central tendency

• In a jury system, if each juror was asked about their confidence in rating culpability by a weight, w\(_i\)

• This way those ratings with a higher weight would give a better estimate of the central tendency

• The weights represent the confidence each rater has on her particular rating

• Mean \(\overline{x}\)=\(\frac{\sum_{i=0}^{n}x_i}{n}\)

• Weighted mean \(\overline{x}\)=\(\frac{\sum_{i=0}^{n}w_ix_i}{\sum_{i=0}^{n}w_i}\)

• Sum of squared deviations \(d^{2}\)=\(\sum_{i=0}^{n}\)\((x_i-\overline{x})^2\)

The notion of deviation

• While central tendencies, or least squares estimates of central tendencies are interesting
• It is more illuminating to understand the notion of deviation itself
• Why? Well, we want to know more about the distribution and the distribution is made up of deviation from the central tendencies
• We would like to know about dispersion

Measures of dispersion

• Variance

  • We square the deviations before averaging them.
  • We have definitions for variance of a population and for a sample drawn from a larger population

• Notice that sample variance, \(s^2\) is calculated by dividing the sum of the squared deviations by n-1 and not n

• Mean \(\overline{x}\)=\(\frac{\sum_{i=0}^{n}x_i}{n}\)

• Population variance \(\sigma^2\)=\(\frac{\sum(x_i-\mu)^2}{N}\)

• Sample variance \(s^2\)=\(\frac{\sum(x_i-\overline{x})^2}{n-1}\)

Why the n-1

• The mean of the underlying process (population) we don’t know
• The mean of the sample we do, this however contains an error due to statistical noise.
• The effect of the error is reduction in the calculated value of \(s^2\)
• To make up for this, n is replaced by n-1.
• If n is large, the difference doesn’t matter. If n is small, this replacement provides a more accurate estimate of the standard deviation of the underlying process.

Measures of Dispersion

• Standard deviation
• Notice that the variance is the average squared deviation - the differences are squared
• To get to the original unit of deviation we take the square root of the variance; sample and population
• Aka, the RMS (root mean square) sample standard deviation; first square the difference, then take the mean and then square root of that.

Population standard deviation =\(\sqrt\sigma^2\)

Sample standard deviation s=\(\sqrt s^2\)

Relating our data with the normal distribution

• The area under the normal distribution is equal to 1
• We have measures of the central tendency in terms of \(\overline{x}\) (sample mean) and also the sample standard deviation, s
• The normal distribution can be defined for any mean value \(μ\), and any standard deviation σ.
• This distribution is also used to calculate probabilities, where the total area under the curve is equal to 1
• That means that the area under any portion of the curve is equal to some proportion of 1.
• This happens, when the mean of the bell-shaped distribution is 0 and the standard deviation is 1.

Z-score normalization

\(z_i\)=\(\frac{x_i-\overline{x}}{s}\)

• Two things to remember:

  1. The area under the normal distribution curve is 1, therefore we can talk about the probability of finding a value larger than any value of x, smaller than any value of x, or between any two values of x; In other words, we can relate individual scores to the normal distribution

  2. If we assume that our data are normally distributed, then we can approximate our data with a normal distribution - we can state these probabilities for our data given the mean and standard deviation

• Now, we can relate the frequency distribution of our data to the normal distribution because we know the mean and standard deviation of both.
• The key is to be able to express any value in a data set in terms of its distance in standard deviations from the mean. — # Relating data to the normal distribution

#View(ncrb)

x_bar_male = mean(ncrb$Arrested_Male)
sd_male = sd(ncrb$Arrested_Male)

• Let’s say we data where the mean is 2.3 and the standard deviation is 0.7
• This means that a score of 3, is exactly one standard deviation higher than the mean
• A score of 1.6, on the other hand, is exactly one standard deviation lower than the mean
• By doing this, we are expressing our data values, in standard deviation units, and fitting our data on the normal distribution - where the mean is 0 and the standard deviation is 1; this is elementary modeling
• We try to fit our data (mean and standard deviation) to the theoretical normal distribution

z-score normlaization

• One such way of expressing/relating our data to the normal distribution is to acquire z-scores
• Each data value, xi is replaced by the distance between it and the sample mean
• The distance is measured as the number of standard deviations between the data value and the mean
• Due to this “standardizing” of the data, z-scores always have a mean of 0 and a standard deviation of 1, just like the normal distribution.
• Notice, this is also a way of relating our sample data to the population data in meaningful ways; in terms of probability

\(z_i\)=\(\frac{x_i-\overline{x}}{s}\)

Standardization and probability

• Now, with the standardized values we can make probability statements.
• For example, in this figure, the area under the normal curve between -1.96 and 1.96 is 0.95.
• This means that 95% of the values we draw from a normal distribution will be between 1.96 standard deviations below the mean and 1.96 standard deviations above the mean

Central limit theorem

• Sample mean is just an estimate of the population parameter, so if we could measure the error of the sample mean then we could put a confidence value on how well it estimates \(μ\)
• A way to approach this is to consider the sampling distribution of \(\overline{x}\).
• Central limit theorem states that as the number of observations in each sample increases, the distribution of the means drawn from these samples tends toward the normal distribution.
• Consider dice throws for example. The outcomes for a sample of one observation as already know come from a uniform distribution; probability of 1/6,
• The distribution of the means of even two observations can start to look like a normal distribution.
• We have an equal probability in any particular trial of rolling any one of the numbers on the dice - a uniform distribution - but the frequency distribution of the sample mean, even of a small sample size of only 2 rolls, follows the normal distribution.
• Here is the easy explanation: How many ways to average a 6, how many ways to average a 3.5?

Probability density

• The probability of getting exactly some particular value on a continuous measurement scale = 0
• Example - measure the number of arrests divided by convictions
• Typically we measure to a chosen degree of accuracy - to two decimal places.
• In theory, we can produce an arbitrarily precise measurement
• So, on a continuous measurement scale that permits arbitrarily precise values, the probability of finding exactly one particular value, say exactly 25.64, is actually zero
• Because we can always specify some greater degree of precision that will keep our observation from being exactly 25.64 milliseconds - 25.641 ms.
• Cumulative Distribution Function

# create sample data
sample_Data = rnorm(500)

# calculate CDF
CDF <- ecdf(sample_Data )

# draw the cdf plot
plot(CDF)