DATA 605 Final Exam

Problem 1.

Using R, set a random seed equal to 1234 (i.e., set.seed(1234)). Generate a random variable X that has 10,000 continuous random uniform values between 5 and 15.Then generate a random variable Y that has 10,000 random normal values with a mean of 10 and a standard deviation of 2.89.

set.seed(1234)

X <- runif(10000, 5, 15  )
Y <- rnorm(10000, 10, 2.89)

data <- data.frame(X, Y)

x <- median(data$X)
y <- median(data$Y)

Probability.

Calculate as a minimum the below probabilities a through c. Assume the small letter “x” is estimated as the median of the X variable, and the small letter “y” is estimated as the median of the Y variable. Interpret the meaning of all probabilities.

Answers: 5 points a. P(X>x | X>y) b. P(X>x & Y>y) c. P(X<x | X>y)

A. P(X>x | X>y): This expression means that X is greater than x given that X is greater than y.

B. P(X>x & Y>y): The expression means that X and Y take values greater than x and y, respectively.

C. P(X<x | X>y): This expression means that X takes a value less than x given that it is greater than y.

5 points. Investigate whether P(X>x & Y>y)=P(X>x)P(Y>y) by building a table and evaluating the marginal and joint probabilities.

whether_X_gt_x <- data$X > x
whether_Y_gt_y <- data$Y > y
whether_X_gt_x_and_whether_Y_gt_y <- (data$X > x & data$Y > y)

# P(X > x )
p_X_gt_x <- mean(data$X > x)

# P(Y > y)
p_Y_gt_y <- mean(data$Y > y)

# P(X > x & Y > y)
p_X_gt_x_and_Y_gt_y  <- mean(data$X > x & data$Y > y )

df1 <-data.frame(
  X = X,
  Y = Y,
  whether_X_gt_x = whether_X_gt_x,
  whether_Y_gt_y = whether_Y_gt_y ,
  whether_X_gt_x_and_whether_Y_gt_y = whether_X_gt_x_and_whether_Y_gt_y
)
print(head(df1, 10))
##            X         Y whether_X_gt_x whether_Y_gt_y
## 1   6.137034  7.615333          FALSE          FALSE
## 2  11.222994 11.003316           TRUE           TRUE
## 3  11.092747  7.340931           TRUE          FALSE
## 4  11.233794  9.169598           TRUE          FALSE
## 5  13.609154  8.407233           TRUE          FALSE
## 6  11.403106 12.452586           TRUE           TRUE
## 7   5.094958 13.767153          FALSE           TRUE
## 8   7.325505  8.472250          FALSE          FALSE
## 9  11.660838 12.204927           TRUE           TRUE
## 10 10.142511 15.173997           TRUE           TRUE
##    whether_X_gt_x_and_whether_Y_gt_y
## 1                              FALSE
## 2                               TRUE
## 3                              FALSE
## 4                              FALSE
## 5                              FALSE
## 6                               TRUE
## 7                              FALSE
## 8                              FALSE
## 9                               TRUE
## 10                              TRUE
df2 <- data.frame(
  p_X_gt_x = p_X_gt_x,
  p_Y_gt_y = p_Y_gt_y,
  p_X_gt_x_and_Y_gt_y = p_X_gt_x_and_Y_gt_y
)

print(df2)
##   p_X_gt_x p_Y_gt_y p_X_gt_x_and_Y_gt_y
## 1      0.5      0.5              0.2507

Interpretation:

  1. P(X > x): The probability that a randomly chosen value of X is greater than x.
  2. P(Y > y): The probability that a randomly chosen value of Y is greater than y.
  3. P(X > x & Y > y): The joint probability that a randomly chosen value of X is greater than x AND a randomly chosen value of Y is greater than y.

The probablity of the X value being greater than x and Y value being greater than y is equal to 0.2507. This value would be closer to 0.25 if it had more data points. 0.25 is P(X>x)*P(Y>y) since P(X>x)=0.5 and P(Y>y)=0.5

5 points Check to see if independence holds by using Fisher’s Exact Test and the Chi Square Test. What is the difference between the two? Which is most appropriate? Are you surprised at the results? Why or why not?

contingency_table <- table(whether_X_gt_x ,whether_Y_gt_y)

fisher_result <- fisher.test(contingency_table)
print(fisher_result)
## 
##  Fisher's Exact Test for Count Data
## 
## data:  contingency_table
## p-value = 0.7949
## alternative hypothesis: true odds ratio is not equal to 1
## 95 percent confidence interval:
##  0.9342763 1.0946016
## sample estimates:
## odds ratio 
##   1.011264
chi_result <- chisq.test(contingency_table)
print(chi_result)
## 
##  Pearson's Chi-squared test with Yates' continuity correction
## 
## data:  contingency_table
## X-squared = 0.0676, df = 1, p-value = 0.7949

Fishers Exact Test is used when sample sizes are small, while the Chi Square Test is used for larger sample sizes. If the sample size is small, use Fishers Exact Test. For larger sample sizes, use the Chi Square Test. The Fisher’s Exact test gave us more information, so I would say the Fishers Exact test. Yes I’m a bit surprised as this wasn’t the result that I had foreseen, I thought that there was no difference. In both of these tests we see that the p-value is greater than 0.05 thus implying that there is no significant association between the two variables X and Y.

Problem 2

You are to register for Kaggle.com (free) and compete in the Regression with a Crab Age Dataset competition. https://www.kaggle.com/competitions/playground-series-s3e16 I want you to do the following.

5 points. Descriptive and Inferential Statistics. Provide univariate descriptive statistics and appropriate plots for the training data set. Provide a scatterplot matrix for at least two of the independent variables and the dependent variable. Derive a correlation matrix for any three quantitative variables in the dataset. Test the hypotheses that the correlations between each pairwise set of variables is 0 and provide an 80% confidence interval. Discuss the meaning of your analysis. Would you be worried about familywise error? Why or why not?

Introduction to Data

We are going to be looking at the CrabAgePredicition.csv. It gives us the crabs sex, age, length, diameter, weight, shucked weight, and Viscera Weight.

library(readr)
data <- read_csv("/Users/joe/Documents/Programming_workbook/CrabAgePrediction.csv")
## Rows: 3893 Columns: 9
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (1): Sex
## dbl (8): Length, Diameter, Height, Weight, Shucked Weight, Viscera Weight, S...
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
head(data, n =5)
## # A tibble: 5 × 9
##   Sex   Length Diameter Height Weight `Shucked Weight` `Viscera Weight`
##   <chr>  <dbl>    <dbl>  <dbl>  <dbl>            <dbl>            <dbl>
## 1 F      1.44     1.18   0.412  24.6             12.3              5.58
## 2 M      0.888    0.65   0.212   5.40             2.30             1.37
## 3 I      1.04     0.775  0.25    7.95             3.23             1.60
## 4 F      1.18     0.888  0.25   13.5              4.75             2.28
## 5 I      0.888    0.662  0.212   6.90             3.46             1.49
## # ℹ 2 more variables: `Shell Weight` <dbl>, Age <dbl>
hist(data$Age, main = "Histogram of age in Crabs")

The histogram shows a slight tail leading off to the right of the Age.

pairs(data[, c("Age", "Length", "Weight", "Shell Weight","Viscera Weight")], labels = c("Age", "Length", "Weight", "Shell Weight","Viscera Weight"))

Take a look at the scatter plots, we can see a relationship in Length, Weight, Shell Weight. I modest relationship with the Viscera Weight some some data points line up. Surprisingly not really a true sign of a relationship, but a minor one in Length.

Summary Statistics

summary(data)
##      Sex                Length          Diameter          Height      
##  Length:3893        Min.   :0.1875   Min.   :0.1375   Min.   :0.0000  
##  Class :character   1st Qu.:1.1250   1st Qu.:0.8750   1st Qu.:0.2875  
##  Mode  :character   Median :1.3625   Median :1.0625   Median :0.3625  
##                     Mean   :1.3113   Mean   :1.0209   Mean   :0.3494  
##                     3rd Qu.:1.5375   3rd Qu.:1.2000   3rd Qu.:0.4125  
##                     Max.   :2.0375   Max.   :1.6250   Max.   :2.8250  
##      Weight        Shucked Weight     Viscera Weight      Shell Weight     
##  Min.   : 0.0567   Min.   : 0.02835   Min.   : 0.01418   Min.   : 0.04252  
##  1st Qu.:12.6722   1st Qu.: 5.34388   1st Qu.: 2.66485   1st Qu.: 3.71378  
##  Median :22.7930   Median : 9.53961   Median : 4.86194   Median : 6.66213  
##  Mean   :23.5673   Mean   :10.20734   Mean   : 5.13655   Mean   : 6.79584  
##  3rd Qu.:32.7862   3rd Qu.:14.27397   3rd Qu.: 7.20077   3rd Qu.: 9.35534  
##  Max.   :80.1015   Max.   :42.18406   Max.   :21.54562   Max.   :28.49125  
##       Age        
##  Min.   : 1.000  
##  1st Qu.: 8.000  
##  Median :10.000  
##  Mean   : 9.955  
##  3rd Qu.:11.000  
##  Max.   :29.000

Analysis

cor_matrix <- cor(data[, c("Length", "Height", "Weight")])

print(cor_matrix)
##           Length    Height    Weight
## Length 1.0000000 0.8230810 0.9253735
## Height 0.8230810 1.0000000 0.8144051
## Weight 0.9253735 0.8144051 1.0000000
cor_test_result1 <- cor.test(data$Length, data$Height, conf.level = 0.80)
cor_test_result2 <- cor.test(data$Height, data$Weight, conf.level = 0.80)
cor_test_result3 <- cor.test(data$Weight, data$Length, conf.level = 0.80)

# Print the correlation test result
print(cor_test_result1)
## 
##  Pearson's product-moment correlation
## 
## data:  data$Length and data$Height
## t = 90.403, df = 3891, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 80 percent confidence interval:
##  0.8163405 0.8295972
## sample estimates:
##      cor 
## 0.823081
print(cor_test_result2)
## 
##  Pearson's product-moment correlation
## 
## data:  data$Height and data$Weight
## t = 87.543, df = 3891, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 80 percent confidence interval:
##  0.8073691 0.8212096
## sample estimates:
##       cor 
## 0.8144051
print(cor_test_result3)
## 
##  Pearson's product-moment correlation
## 
## data:  data$Weight and data$Length
## t = 152.28, df = 3891, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 80 percent confidence interval:
##  0.9223644 0.9282704
## sample estimates:
##       cor 
## 0.9253735

The Pearsons Product-moment correlation test gives us a minimum of 0.8163 and maximum of 0.8295 between the Length and Height. These numbers a greater than 0 so the correlation between the two is very high.

In Height vs Weight there is still a 0.8073 minimum and a 0.82120 maximum, meaning there is still a high correlation between the two.

Finally, Weight vs Length, we get a 0.92236 minimum and a 0.92827 maximum, meaning there is a high correlation between the two.

5 points. Linear Algebra and Correlation. Invert your correlation matrix from above. (This is known as the precision matrix and contains variance inflation factors on the diagonal.) Multiply the correlation matrix by the precision matrix, and then multiply the precision matrix by the correlation matrix. Conduct LDU decomposition on the matrix.

Correlation Matrix

cor_matrix <- matrix(c(1.0000000, 0.8230810, 0.9253735,
                        0.8230810, 1.0000000, 0.8144051,
                        0.9253735, 0.8144051, 1.0000000), ncol = 3)

precision_matrix <- solve(cor_matrix)

result1 <- cor_matrix %*% precision_matrix

# Multiply the precision matrix by the correlation matrix
result2 <- precision_matrix %*% cor_matrix

LDU Decomposition

eigen_decomposition <- eigen(cor_matrix)

L <- eigen_decomposition$vectors
D <- diag(eigen_decomposition$values)
U <- t(L)

cat("Inverted Correlation Matrix (Precision Matrix):\n")
## Inverted Correlation Matrix (Precision Matrix):
print(precision_matrix)
##           [,1]      [,2]      [,3]
## [1,]  7.730384 -1.594359 -5.855038
## [2,] -1.594359  3.298442 -1.210890
## [3,] -5.855038 -1.210890  7.404252
cat("\nResult of Multiplying Correlation Matrix by Precision Matrix:\n")
## 
## Result of Multiplying Correlation Matrix by Precision Matrix:
print(result1)
##      [,1] [,2] [,3]
## [1,]    1    0    0
## [2,]    0    1    0
## [3,]    0    0    1
cat("\nResult of Multiplying Precision Matrix by Correlation Matrix:\n")
## 
## Result of Multiplying Precision Matrix by Correlation Matrix:
print(result2)
##               [,1] [,2] [,3]
## [1,]  1.000000e+00    0    0
## [2,]  0.000000e+00    1    0
## [3,] -8.881784e-16    0    1
cat("\nLDU Decomposition:\n")
## 
## LDU Decomposition:
cat("\nL (Lower Triangular):\n")
## 
## L (Lower Triangular):
print(L)
##            [,1]       [,2]        [,3]
## [1,] -0.5863851  0.3709049  0.72012645
## [2,] -0.5607737 -0.8274084 -0.03046613
## [3,] -0.5845387  0.4216928 -0.69317365
cat("\nD (Diagonal):\n")
## 
## D (Diagonal):
print(D)
##          [,1]      [,2]       [,3]
## [1,] 2.709591 0.0000000 0.00000000
## [2,] 0.000000 0.2159694 0.00000000
## [3,] 0.000000 0.0000000 0.07443948
cat("\nU (Upper Triangular):\n")
## 
## U (Upper Triangular):
print(U)
##            [,1]        [,2]       [,3]
## [1,] -0.5863851 -0.56077366 -0.5845387
## [2,]  0.3709049 -0.82740844  0.4216928
## [3,]  0.7201265 -0.03046613 -0.6931736

5 points. Calculus-Based Probability & Statistics. Many times, it makes sense to fit a closed form distribution to data. Select a variable in the Kaggle.com training dataset that is skewed to the right, shift it so that the minimum value is absolutely above zero if necessary. Then load the MASS package and run fitdistr to fit an exponential probability density function. (See https://stat.ethz.ch/R-manual/R-devel/library/MASS/html/fitdistr.html ).

Find the optimal value of lambda for this distribution, and then take 1000 samples from this exponential distribution using this value (e.g., rexp(1000, lambda)). Plot a histogram and compare it with a histogram of your original variable. Using the exponential pdf, find the 5th and 95th percentiles using the cumulative distribution function (CDF). Also generate a 95% confidence interval from the empirical data, assuming normality. Finally, provide the empirical 5th percentile and 95th percentile of the data. Discuss.

I chose the Age variable from the crab dataset, as this is skewed to the right. The fit gives a lambda of 0.10045 with standard deviation 0.0016

library(MASS)

fit_result <- fitdistr(data$Age, "exponential")

# Print the result
print(fit_result)
##       rate    
##   0.100454147 
##  (0.001609999)

Histogram

I managed to display both the original the the fitted exponential of the two for Age. The result is that they do not look similar to each other at all. The Age for instance looks more like a bell curve instead of exponential.

hist(data$Age, breaks = 30, col = "skyblue", main = "Histogram", xlab = "Values", ylab = "Frequency", xlim=c(0,60))

lambda_optimal <- fit_result$estimate
samples_exponential <- rexp(1000, rate = lambda_optimal)

hist(samples_exponential, breaks = 30, col = "lightgreen", main = "Generated Exponential Samples", xlab = "Values", ylab = "Frequency")

Here are the 5th and 95th percential of the exponential samples.

percentile_5_exp <- pexp(quantile(samples_exponential, 0.05), rate = lambda_optimal)
percentile_95_exp <- pexp(quantile(samples_exponential, 0.95), rate = lambda_optimal)

print(percentile_5_exp)
##        5% 
## 0.0521892
print(percentile_95_exp)
##       95% 
## 0.9442963

This doesn’t quite make sense to me, judging from the Histograms, I think this looks a little small. I could have done this incorrectly, but the empirical data leads us to a 9.85 - 10.05 in our data.

conf_interval_normality <- t.test(data$Age)$conf.int

print(conf_interval_normality)
## [1]  9.85358 10.05600
## attr(,"conf.level")
## [1] 0.95

This makes a lot more sense, because Age is mostly between 6 and 16

percentile_5_empirical <- quantile(data$Age, 0.05)
percentile_95_empirical <- quantile(data$Age, 0.95)

print(percentile_5_empirical)
## 5% 
##  6
print(percentile_95_empirical)
## 95% 
##  16

10 points. Modeling. Build some type of multiple regression model and submit your model to the competition board. Provide your complete model summary and results with analysis. Report your Kaggle.com user name and score.

Multiple Linear Regression Model

I decided to use multiple regression with only the Length, Height, and Weight to predict Age. In my opinion, Shell Weight and Vicera Weight are related to Weight, and Diameter is related to Length, so I got rid of them in favor of a simpler model.

crab.model <- lm(Age ~ Length + Height + Weight, data = data)

Sumary Analysis

It turns out that Length and Height have the most influence on Age due to their low P-values (2.39e-12 and 2e-16). Weight also has an influence on Height, but only has P-value 0.04.

summary(crab.model)
## 
## Call:
## lm(formula = Age ~ Length + Height + Weight, data = data)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -22.8876  -1.6499  -0.6749   0.8963  16.4001 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept) 2.935656   0.324210   9.055  < 2e-16 ***
## Length      2.737453   0.389274   7.032 2.39e-12 ***
## Height      8.692997   0.727716  11.946  < 2e-16 ***
## Weight      0.016650   0.008239   2.021   0.0434 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 2.624 on 3889 degrees of freedom
## Multiple R-squared:  0.3368, Adjusted R-squared:  0.3363 
## F-statistic: 658.2 on 3 and 3889 DF,  p-value: < 2.2e-16

Here we put the crab.model and fit it to a linear model. It gives us a very horizontal thick line from about 5 to 14. There is about two outliers but they should be eliminiated compared to the rest of the overwhelming data points.

crab.model.fit <- fitted(crab.model)
plot(crab.model.fit)

We have put our linear model into a matrix scatterplot and it gives us some interesting reads. All of the models read positive going up from the left side, however some of them read a lot more then the rest. Weight vs. Height have an extreme positive incline. While Height vs. Length have an almost gradual rise. Age has the most “normal” rise in the slope when plotted against Length and Weight, while Age vs. Height has a very extreme slope as well.

library(car)
## Loading required package: carData
car::scatterplotMatrix(~ Age + Length + Height + Weight, data = data, smooth = FALSE)

Kaggle.com User name and Score.

joegarcia21, im not sure on a score yet, but I posted it on Kaggle.