Using R, set a random seed equal to 1234 (i.e., set.seed(1234)). Generate a random variable X that has 10,000 continuous random uniform values between 5 and 15.Then generate a random variable Y that has 10,000 random normal values with a mean of 10 and a standard deviation of 2.89.
set.seed(1234)
X <- runif(10000, 5, 15 )
Y <- rnorm(10000, 10, 2.89)
data <- data.frame(X, Y)
x <- median(data$X)
y <- median(data$Y)
Calculate as a minimum the below probabilities a through c. Assume the small letter “x” is estimated as the median of the X variable, and the small letter “y” is estimated as the median of the Y variable. Interpret the meaning of all probabilities.
A. P(X>x | X>y): This expression means that X is greater than x given that X is greater than y.
B. P(X>x & Y>y): The expression means that X and Y take values greater than x and y, respectively.
C. P(X<x | X>y): This expression means that X takes a value less than x given that it is greater than y.
whether_X_gt_x <- data$X > x
whether_Y_gt_y <- data$Y > y
whether_X_gt_x_and_whether_Y_gt_y <- (data$X > x & data$Y > y)
# P(X > x )
p_X_gt_x <- mean(data$X > x)
# P(Y > y)
p_Y_gt_y <- mean(data$Y > y)
# P(X > x & Y > y)
p_X_gt_x_and_Y_gt_y <- mean(data$X > x & data$Y > y )
df1 <-data.frame(
X = X,
Y = Y,
whether_X_gt_x = whether_X_gt_x,
whether_Y_gt_y = whether_Y_gt_y ,
whether_X_gt_x_and_whether_Y_gt_y = whether_X_gt_x_and_whether_Y_gt_y
)
print(head(df1, 10))
## X Y whether_X_gt_x whether_Y_gt_y
## 1 6.137034 7.615333 FALSE FALSE
## 2 11.222994 11.003316 TRUE TRUE
## 3 11.092747 7.340931 TRUE FALSE
## 4 11.233794 9.169598 TRUE FALSE
## 5 13.609154 8.407233 TRUE FALSE
## 6 11.403106 12.452586 TRUE TRUE
## 7 5.094958 13.767153 FALSE TRUE
## 8 7.325505 8.472250 FALSE FALSE
## 9 11.660838 12.204927 TRUE TRUE
## 10 10.142511 15.173997 TRUE TRUE
## whether_X_gt_x_and_whether_Y_gt_y
## 1 FALSE
## 2 TRUE
## 3 FALSE
## 4 FALSE
## 5 FALSE
## 6 TRUE
## 7 FALSE
## 8 FALSE
## 9 TRUE
## 10 TRUE
df2 <- data.frame(
p_X_gt_x = p_X_gt_x,
p_Y_gt_y = p_Y_gt_y,
p_X_gt_x_and_Y_gt_y = p_X_gt_x_and_Y_gt_y
)
print(df2)
## p_X_gt_x p_Y_gt_y p_X_gt_x_and_Y_gt_y
## 1 0.5 0.5 0.2507
Interpretation:
The probablity of the X value being greater than x and Y value being greater than y is equal to 0.2507. This value would be closer to 0.25 if it had more data points. 0.25 is P(X>x)*P(Y>y) since P(X>x)=0.5 and P(Y>y)=0.5
contingency_table <- table(whether_X_gt_x ,whether_Y_gt_y)
fisher_result <- fisher.test(contingency_table)
print(fisher_result)
##
## Fisher's Exact Test for Count Data
##
## data: contingency_table
## p-value = 0.7949
## alternative hypothesis: true odds ratio is not equal to 1
## 95 percent confidence interval:
## 0.9342763 1.0946016
## sample estimates:
## odds ratio
## 1.011264
chi_result <- chisq.test(contingency_table)
print(chi_result)
##
## Pearson's Chi-squared test with Yates' continuity correction
##
## data: contingency_table
## X-squared = 0.0676, df = 1, p-value = 0.7949
Fishers Exact Test is used when sample sizes are small, while the Chi Square Test is used for larger sample sizes. If the sample size is small, use Fishers Exact Test. For larger sample sizes, use the Chi Square Test. The Fisher’s Exact test gave us more information, so I would say the Fishers Exact test. Yes I’m a bit surprised as this wasn’t the result that I had foreseen, I thought that there was no difference. In both of these tests we see that the p-value is greater than 0.05 thus implying that there is no significant association between the two variables X and Y.
You are to register for Kaggle.com (free) and compete in the Regression with a Crab Age Dataset competition. https://www.kaggle.com/competitions/playground-series-s3e16 I want you to do the following.
5 points. Descriptive and Inferential Statistics. Provide univariate descriptive statistics and appropriate plots for the training data set. Provide a scatterplot matrix for at least two of the independent variables and the dependent variable. Derive a correlation matrix for any three quantitative variables in the dataset. Test the hypotheses that the correlations between each pairwise set of variables is 0 and provide an 80% confidence interval. Discuss the meaning of your analysis. Would you be worried about familywise error? Why or why not?
We are going to be looking at the CrabAgePredicition.csv. It gives us the crabs sex, age, length, diameter, weight, shucked weight, and Viscera Weight.
library(readr)
data <- read_csv("/Users/joe/Documents/Programming_workbook/CrabAgePrediction.csv")
## Rows: 3893 Columns: 9
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (1): Sex
## dbl (8): Length, Diameter, Height, Weight, Shucked Weight, Viscera Weight, S...
##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
head(data, n =5)
## # A tibble: 5 × 9
## Sex Length Diameter Height Weight `Shucked Weight` `Viscera Weight`
## <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 F 1.44 1.18 0.412 24.6 12.3 5.58
## 2 M 0.888 0.65 0.212 5.40 2.30 1.37
## 3 I 1.04 0.775 0.25 7.95 3.23 1.60
## 4 F 1.18 0.888 0.25 13.5 4.75 2.28
## 5 I 0.888 0.662 0.212 6.90 3.46 1.49
## # ℹ 2 more variables: `Shell Weight` <dbl>, Age <dbl>
hist(data$Age, main = "Histogram of age in Crabs")
The histogram shows a slight tail leading off to the right of the Age.
pairs(data[, c("Age", "Length", "Weight", "Shell Weight","Viscera Weight")], labels = c("Age", "Length", "Weight", "Shell Weight","Viscera Weight"))
Take a look at the scatter plots, we can see a relationship in Length, Weight, Shell Weight. I modest relationship with the Viscera Weight some some data points line up. Surprisingly not really a true sign of a relationship, but a minor one in Length.
summary(data)
## Sex Length Diameter Height
## Length:3893 Min. :0.1875 Min. :0.1375 Min. :0.0000
## Class :character 1st Qu.:1.1250 1st Qu.:0.8750 1st Qu.:0.2875
## Mode :character Median :1.3625 Median :1.0625 Median :0.3625
## Mean :1.3113 Mean :1.0209 Mean :0.3494
## 3rd Qu.:1.5375 3rd Qu.:1.2000 3rd Qu.:0.4125
## Max. :2.0375 Max. :1.6250 Max. :2.8250
## Weight Shucked Weight Viscera Weight Shell Weight
## Min. : 0.0567 Min. : 0.02835 Min. : 0.01418 Min. : 0.04252
## 1st Qu.:12.6722 1st Qu.: 5.34388 1st Qu.: 2.66485 1st Qu.: 3.71378
## Median :22.7930 Median : 9.53961 Median : 4.86194 Median : 6.66213
## Mean :23.5673 Mean :10.20734 Mean : 5.13655 Mean : 6.79584
## 3rd Qu.:32.7862 3rd Qu.:14.27397 3rd Qu.: 7.20077 3rd Qu.: 9.35534
## Max. :80.1015 Max. :42.18406 Max. :21.54562 Max. :28.49125
## Age
## Min. : 1.000
## 1st Qu.: 8.000
## Median :10.000
## Mean : 9.955
## 3rd Qu.:11.000
## Max. :29.000
cor_matrix <- cor(data[, c("Length", "Height", "Weight")])
print(cor_matrix)
## Length Height Weight
## Length 1.0000000 0.8230810 0.9253735
## Height 0.8230810 1.0000000 0.8144051
## Weight 0.9253735 0.8144051 1.0000000
cor_test_result1 <- cor.test(data$Length, data$Height, conf.level = 0.80)
cor_test_result2 <- cor.test(data$Height, data$Weight, conf.level = 0.80)
cor_test_result3 <- cor.test(data$Weight, data$Length, conf.level = 0.80)
# Print the correlation test result
print(cor_test_result1)
##
## Pearson's product-moment correlation
##
## data: data$Length and data$Height
## t = 90.403, df = 3891, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 80 percent confidence interval:
## 0.8163405 0.8295972
## sample estimates:
## cor
## 0.823081
print(cor_test_result2)
##
## Pearson's product-moment correlation
##
## data: data$Height and data$Weight
## t = 87.543, df = 3891, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 80 percent confidence interval:
## 0.8073691 0.8212096
## sample estimates:
## cor
## 0.8144051
print(cor_test_result3)
##
## Pearson's product-moment correlation
##
## data: data$Weight and data$Length
## t = 152.28, df = 3891, p-value < 2.2e-16
## alternative hypothesis: true correlation is not equal to 0
## 80 percent confidence interval:
## 0.9223644 0.9282704
## sample estimates:
## cor
## 0.9253735
The Pearsons Product-moment correlation test gives us a minimum of 0.8163 and maximum of 0.8295 between the Length and Height. These numbers a greater than 0 so the correlation between the two is very high.
In Height vs Weight there is still a 0.8073 minimum and a 0.82120 maximum, meaning there is still a high correlation between the two.
Finally, Weight vs Length, we get a 0.92236 minimum and a 0.92827 maximum, meaning there is a high correlation between the two.
5 points. Linear Algebra and Correlation. Invert your correlation matrix from above. (This is known as the precision matrix and contains variance inflation factors on the diagonal.) Multiply the correlation matrix by the precision matrix, and then multiply the precision matrix by the correlation matrix. Conduct LDU decomposition on the matrix.
cor_matrix <- matrix(c(1.0000000, 0.8230810, 0.9253735,
0.8230810, 1.0000000, 0.8144051,
0.9253735, 0.8144051, 1.0000000), ncol = 3)
precision_matrix <- solve(cor_matrix)
result1 <- cor_matrix %*% precision_matrix
# Multiply the precision matrix by the correlation matrix
result2 <- precision_matrix %*% cor_matrix
eigen_decomposition <- eigen(cor_matrix)
L <- eigen_decomposition$vectors
D <- diag(eigen_decomposition$values)
U <- t(L)
cat("Inverted Correlation Matrix (Precision Matrix):\n")
## Inverted Correlation Matrix (Precision Matrix):
print(precision_matrix)
## [,1] [,2] [,3]
## [1,] 7.730384 -1.594359 -5.855038
## [2,] -1.594359 3.298442 -1.210890
## [3,] -5.855038 -1.210890 7.404252
cat("\nResult of Multiplying Correlation Matrix by Precision Matrix:\n")
##
## Result of Multiplying Correlation Matrix by Precision Matrix:
print(result1)
## [,1] [,2] [,3]
## [1,] 1 0 0
## [2,] 0 1 0
## [3,] 0 0 1
cat("\nResult of Multiplying Precision Matrix by Correlation Matrix:\n")
##
## Result of Multiplying Precision Matrix by Correlation Matrix:
print(result2)
## [,1] [,2] [,3]
## [1,] 1.000000e+00 0 0
## [2,] 0.000000e+00 1 0
## [3,] -8.881784e-16 0 1
cat("\nLDU Decomposition:\n")
##
## LDU Decomposition:
cat("\nL (Lower Triangular):\n")
##
## L (Lower Triangular):
print(L)
## [,1] [,2] [,3]
## [1,] -0.5863851 0.3709049 0.72012645
## [2,] -0.5607737 -0.8274084 -0.03046613
## [3,] -0.5845387 0.4216928 -0.69317365
cat("\nD (Diagonal):\n")
##
## D (Diagonal):
print(D)
## [,1] [,2] [,3]
## [1,] 2.709591 0.0000000 0.00000000
## [2,] 0.000000 0.2159694 0.00000000
## [3,] 0.000000 0.0000000 0.07443948
cat("\nU (Upper Triangular):\n")
##
## U (Upper Triangular):
print(U)
## [,1] [,2] [,3]
## [1,] -0.5863851 -0.56077366 -0.5845387
## [2,] 0.3709049 -0.82740844 0.4216928
## [3,] 0.7201265 -0.03046613 -0.6931736
5 points. Calculus-Based Probability & Statistics. Many times, it makes sense to fit a closed form distribution to data. Select a variable in the Kaggle.com training dataset that is skewed to the right, shift it so that the minimum value is absolutely above zero if necessary. Then load the MASS package and run fitdistr to fit an exponential probability density function. (See https://stat.ethz.ch/R-manual/R-devel/library/MASS/html/fitdistr.html ).
Find the optimal value of lambda for this distribution, and then take 1000 samples from this exponential distribution using this value (e.g., rexp(1000, lambda)). Plot a histogram and compare it with a histogram of your original variable. Using the exponential pdf, find the 5th and 95th percentiles using the cumulative distribution function (CDF). Also generate a 95% confidence interval from the empirical data, assuming normality. Finally, provide the empirical 5th percentile and 95th percentile of the data. Discuss.
I chose the Age variable from the crab dataset, as this is skewed to the right. The fit gives a lambda of 0.10045 with standard deviation 0.0016
library(MASS)
fit_result <- fitdistr(data$Age, "exponential")
# Print the result
print(fit_result)
## rate
## 0.100454147
## (0.001609999)
I managed to display both the original the the fitted exponential of the two for Age. The result is that they do not look similar to each other at all. The Age for instance looks more like a bell curve instead of exponential.
hist(data$Age, breaks = 30, col = "skyblue", main = "Histogram", xlab = "Values", ylab = "Frequency", xlim=c(0,60))
lambda_optimal <- fit_result$estimate
samples_exponential <- rexp(1000, rate = lambda_optimal)
hist(samples_exponential, breaks = 30, col = "lightgreen", main = "Generated Exponential Samples", xlab = "Values", ylab = "Frequency")
Here are the 5th and 95th percential of the exponential samples.
percentile_5_exp <- pexp(quantile(samples_exponential, 0.05), rate = lambda_optimal)
percentile_95_exp <- pexp(quantile(samples_exponential, 0.95), rate = lambda_optimal)
print(percentile_5_exp)
## 5%
## 0.0521892
print(percentile_95_exp)
## 95%
## 0.9442963
This doesn’t quite make sense to me, judging from the Histograms, I think this looks a little small. I could have done this incorrectly, but the empirical data leads us to a 9.85 - 10.05 in our data.
conf_interval_normality <- t.test(data$Age)$conf.int
print(conf_interval_normality)
## [1] 9.85358 10.05600
## attr(,"conf.level")
## [1] 0.95
This makes a lot more sense, because Age is mostly between 6 and 16
percentile_5_empirical <- quantile(data$Age, 0.05)
percentile_95_empirical <- quantile(data$Age, 0.95)
print(percentile_5_empirical)
## 5%
## 6
print(percentile_95_empirical)
## 95%
## 16
10 points. Modeling. Build some type of multiple regression model and submit your model to the competition board. Provide your complete model summary and results with analysis. Report your Kaggle.com user name and score.
I decided to use multiple regression with only the Length, Height, and Weight to predict Age. In my opinion, Shell Weight and Vicera Weight are related to Weight, and Diameter is related to Length, so I got rid of them in favor of a simpler model.
crab.model <- lm(Age ~ Length + Height + Weight, data = data)
It turns out that Length and Height have the most influence on Age due to their low P-values (2.39e-12 and 2e-16). Weight also has an influence on Height, but only has P-value 0.04.
summary(crab.model)
##
## Call:
## lm(formula = Age ~ Length + Height + Weight, data = data)
##
## Residuals:
## Min 1Q Median 3Q Max
## -22.8876 -1.6499 -0.6749 0.8963 16.4001
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 2.935656 0.324210 9.055 < 2e-16 ***
## Length 2.737453 0.389274 7.032 2.39e-12 ***
## Height 8.692997 0.727716 11.946 < 2e-16 ***
## Weight 0.016650 0.008239 2.021 0.0434 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 2.624 on 3889 degrees of freedom
## Multiple R-squared: 0.3368, Adjusted R-squared: 0.3363
## F-statistic: 658.2 on 3 and 3889 DF, p-value: < 2.2e-16
Here we put the crab.model and fit it to a linear model. It gives us a very horizontal thick line from about 5 to 14. There is about two outliers but they should be eliminiated compared to the rest of the overwhelming data points.
crab.model.fit <- fitted(crab.model)
plot(crab.model.fit)
We have put our linear model into a matrix scatterplot and it gives us
some interesting reads. All of the models read positive going up from
the left side, however some of them read a lot more then the rest.
Weight vs. Height have an extreme positive incline. While Height
vs. Length have an almost gradual rise. Age has the most “normal” rise
in the slope when plotted against Length and Weight, while Age
vs. Height has a very extreme slope as well.
library(car)
## Loading required package: carData
car::scatterplotMatrix(~ Age + Length + Height + Weight, data = data, smooth = FALSE)
joegarcia21, im not sure on a score yet, but I posted it on Kaggle.