Introduction

The following report captures the learnings by Team 4 over our semester in IE5342 with Dr. Matis as our professor. The report is separated into 3 parts which chronologically represent our journey through the course. The report starts with the statistical underpinnings of design of experiments through the execution of designed experiments and culminates with a final model equation explaining the key factors that drive Statapult launch distances.

It was our pleasure to take this journey and it is our hope that the readers of this report enjoy the story as well. ~ Team 4

Part 1

The Siege Setup

We performed a designed experiment on the effect of Release Angle on the distance in which a ball is thrown (measured at its furthest distance). Specifically, we studied whether the settings of 175°, 180°, and 185° significantly differ in their mean distance thrown. Since pulling the lever back takes additional work, we investigated whether this made a significant difference on the mean distance thrown. The other factors were set to constant values of the following:

Fire Angle = 90°
Bungee Position = 200mm
Pin Elevation = 200mm
Cup Elevation = 300mm

To test this hypothesis, we used a completely randomized design with an \(\alpha\) = 0.05.

Sample Size

To save resources during the preparation for the siege, we determined how many samples should be collected to detect a mean difference with a medium effect (i.e. 50% of the standard deviation) with a probability of 75%. Figure 1 below shows the Balanced One-way Analysis of Variance Power Calculation.

Figure 1. Determining Number of Samples Required

Figure 1. Determining Number of Samples Required


The results of the sample size calculation showed the need to run 14 launches per Release Angle.

Trial Run Order & Results

We knew that in order to reduce the margin of error in our sampling process, we would randomize our design for collecting catapult data. In order to achieve this we leveraged the built-in capabilities of R through the design.crd function.

The final randomized run order is shown in Table 1, which contains distance measurements:

Table 1. Distances traveled from different angles.
Release Angle Distance Traveled
180 270
185 275
185 281
175 272
180 283
175 269
180 281
185 286
180 266
175 265
185 279
185 282
175 260
185 278
185 281
175 264
180 270
175 267
180 277
180 275
180 267
185 297
180 285
185 278
175 259
180 264
175 268
185 286
185 295
180 263
185 288
175 268
180 250
180 273
175 265
185 273
175 255
175 268
180 268
175 283
175 254
185 275

Hypothesis Testing

We needed to then understand if Release Angle plays a significant role in the distance the ball travels. To complete this task, we ran a One-way ANOVA test.

Hypotheses

The hypotheses for the ANOVA were as follows:

\(\quad\)Null:\(\quad\quad\quad H_0 : \mu_{175} = \mu_{180} = \mu_{185}\)
\(\quad\)Alternate:\(\quad H_1 : \mu_{i} \neq \mu{j}\) for at least one pair (\(i\),\(j\))

ANOVA

The ANOVA was calculated in R.

The ANOVA resulted in at least one of the means was significantly different than the others.

Table 2. ANOVA output table from our model.
  Df Sum Sq Mean Sq F value Pr(>F)
angles 2 2096 1048 16.46 6.549e-06
Residuals 39 2483 63.66 NA NA

From Table 2, we can see that the p-value is less than 0.001, and with an \(\alpha\) = 0.05, we reject the Null hypothesis. There is evidence to support the claim of a significant difference between at least one of the three sample mean Release Angle launch distances.

Residuals

The residuals were then assessed for normality and constant variance. The charts below show that the residuals are indeed homoscedastic and normal.

Figure 2. Plotting residuals to determine normality and constant variance.

Pairwise Comparisons

Pairwise comparisons were conducted to understand which pairs were statistically significantly different. The following hypotheses were created for a Tukey’s Test.


\(\quad\)Null:\(\quad\quad\quad H_0 : \mu_{i} = \mu{j}\)
\(\quad\)Alternate:\(\quad H_1 : \mu_{i} \neq \mu{j}\) for all \(i \neq j\)

The Tukey’s Test was then run in R, With the following results:

Table 3. Tukey Table output table from our ANOVA model.
  diff lwr upr p adj
180-175 5.357 -1.99 12.7 0.1908
185-175 16.93 9.582 24.28 5.257e-06
185-180 11.57 4.224 18.92 0.001265
Figure 3. Tukey plot output from our ANOVA model.

Figure 3. Tukey plot output from our ANOVA model.

From Table 3 and Figure 3, we can see that the pairwise comparisons resulted in two of the three comparisons being significant to an \(\alpha\) = 0.05.

For the ‘185-175’ and ‘185-180’ comparisons, we reject the Null hypothesis. There is evidence to support the claim of a significant difference between the sample mean distances.

For the ‘180-175’ comparison, we fail to reject the Null hypothesis. There is not enough evidence to support the claim of a significant difference between the sample mean distances.

Conclusions & Recommendations

Release Angle was shown to be a significant factor for some of the pair-wise comparisons but not all. It was not shown to be significant when pulled between 175° to the 180° position.

It is recommended to study other Release Angles to understand the impact they play on ball distance traveled throughout the range of arm motion.

Part 2

The Designed Experiment

In Part 2, we performed a designed experiment on the effect of Pin Elevation and Release Angle on the distance in which a ball is thrown (measured at its furthest distance). Specifically, we studied whether the settings of 100, 150 and 200 mm for Pin Elevation and the settings of 110°, 140° and 170° for Release Angle significantly differ in their mean distance thrown.

The other factors were set to constant values of the following:

Fire Angle = 100°
Bungee Position = 150mm
Cup Elevation = 250mm

The settings of Pin Elevation at 100, 150 and 200 mm were investigated as a fixed effect, as well as settings of the Release Angle corresponding to 110°, 140° and 170° as a random effect. We replicated the design three times.

Model Equation

The model equation is: \(\quad y_{ijk}\) = \(\mu\) + \(\tau_i\) + \(\beta_j\) + \((\tau\beta)_{ij}\) + \(\epsilon_{ijk}\)

 Where \(\tau_i\) is the fixed effect Pin Elevation
  \(\beta_j\) is the random effect Release Angle
  \((\tau\beta)_{ij}\) is the interaction effect of Pin Elevation * Release Angle
  and \(\epsilon_{ijk}\) is the error term

Hypotheses

The Hypotheses we tested were:
Pin Elevation Main Fixed Effect:
\(\quad H_0\) : \(\tau_i\) = 0 \(\forall\) i
\(\quad H_a\) : \(\tau_i \neq\) 0 for at least one \(\ i\)
Release Angle Main Random Effect:
\(\quad H_0\) : \(\beta_j\) = 0 \(\forall\) j
\(\quad H_a\) : \(\beta_j \neq\) 0 for at least one \(\ j\)
Pin Elevation * Release Angle Interaction Effect:
\(\quad H_0\) : \((\tau\beta)_{ij}\) = 0 \(\forall\) ij
\(\quad H_a\) : \((\tau\beta)_{ij} \neq\) 0 for at least one \(\ ij\)
\(\quad\)at a significance level of \(\alpha\) = 0.05

Ranomized Layout & Results

The proposed layout with a randomized run order and the recorded observations are found in Table 4:

Table 4. Random Design with Collected Response.
pinElevation releaseAngle distance
200 140 220
200 170 329
150 170 271
200 110 69
150 110 56
150 140 183
150 110 60
200 110 75
200 140 215
150 140 181
100 110 53
200 170 340
150 140 182
200 140 217
100 170 209
150 170 272
100 110 48
150 110 64
100 140 144
100 170 204
100 110 50
100 170 204
200 110 68
200 170 334
150 170 269
100 140 146
100 140 144

Mixed Effects Model

The result of the General ANOVA Mixed Effects Model shown in Table 5:

Table 5. General ANOVA Mixed Effects Model.
  Df Sum Sq Mean Sq F value Pr(>F)
pinElevation 2 24569 12284 5.579 0.06964
releaseAngle 2 199787 99893 10414 2.669e-28
pinElevation:releaseAngle 4 8808 2202 229.6 3.528e-15
Residual 18 172.7 9.593 NA NA

The interaction effect is significant with a p-value < 0.001 at a significance level of \(\alpha\) = 0.05. We also see that the Release Angle is also significant with a p-value < 0.001 and the Pin Elevation is ‘nearly’ significant to an \(\alpha\) of 0.05, with a p-value = 0.07. We can see the stated impacts in Figure 4 below.

Figure 4. Interaction Plot of Experimental Data.

Residuals

The residuals were then assessed for normality and constant variance. From the charts below the residuals are indeed homoscedastic and normal.

Figure 5. Plotting residuals to determine normality and constant variance.

Part 3

The Designed Experiment

In Part 3, we performed a designed experiment to determine the effect of the available factors of Fire Angle, Bungee Position, Release Angle, Pin Elevation, and Cup Elevation on the distance in which a ball is thrown (measured at its furthest distance). We designed this experiment as a single replicate of a \(2^5\) factorial design with the low and high level of the factors as shown in Table 6.

Table 6. List of factors, levels and settings.

Factor Low Level (-1) High Level (+1)
Fire Angle 90 degrees 110 degrees
Bungee Position 130mm 150mm
Release Angle 140 degrees 170 degrees
Pin Elevation 125mm 175mm
Cup Elevation 225mm 275mm

Randomized Layout & Results

The proposed layout (Table 7) with a randomized run order was generated. Trial runs were taken and the data was collected. The observations were recorded and are presented in the table below.

Table 7. Runs, settings, and results

Fire Angle Bungee Position Release Angle Pin Elevation Cup Elevation Response
110 130 170 175 225 237
110 150 170 175 225 260
110 150 170 175 275 343
90 150 170 175 275 189
110 130 140 175 225 135
110 150 140 125 275 154
90 150 140 175 225 121
90 150 170 175 225 153
110 130 170 175 275 312
90 150 140 175 275 152
110 130 140 175 275 171
110 150 140 125 225 114
90 150 170 125 275 169
90 130 140 175 225 114
110 150 140 175 225 145
90 150 170 125 225 135
90 130 170 175 225 149
110 150 140 175 275 197
90 130 140 125 225 98
90 150 140 125 275 134
90 130 170 125 275 157
90 130 140 125 275 125
110 150 170 125 275 262
110 130 170 125 225 180
90 130 170 125 225 124
90 130 140 175 275 143
110 130 140 125 225 101
90 130 170 175 275 183
110 130 170 125 275 238
110 150 170 125 225 199
90 150 140 125 225 104
110 130 140 125 275 140

Original Model Equation

The original model equation for this experiment is as follows:

Distance = A + B + C + D + E + AB + AC + AD + AE + BC + BD + BE + CD + CE + DE + ABC + ABD + ABE + ACD + ACE + ADE + BCD + BCE + BDE + CDE + ABCD + ABCE + ABDE + ACDE + BCDE + ABCDE

Where: A = Fire Angle; B = Bungee Position; C = Release Angle; D = Pin Elevation; E = Cup Elevation

Significant Factors/Interactions

The half-normal plot for all of the factors and interactions was created which identified the significant factors. Figure 6. Half-normal plot of factors from experimental runs.

The significant terms were found to be:

Signficant Factors/Interactions
Fire_Angle:Release_Angle:Cup_Elevation
Fire_Angle:Bungee_Position
Fire_Angle:Release_Angle:Pin_Elevation
Release_Angle:Cup_Elevation
Release_Angle:Pin_Elevation
Fire_Angle:Cup_Elevation
Bungee_Position
Fire_Angle:Pin_Elevation
Pin_Elevation
Fire_Angle:Release_Angle
Cup_Elevation
Fire_Angle
Release_Angle

Final ANOVA for Model Equation

As we reduced the model, the insignificant factors/interactions were built into the error term. The following represents the ANOVA with only significant factors and interactions built in (to an \(\alpha\) = 0.05)

Analysis of Variance Model
  Df Sum Sq Mean Sq F value Pr(>F)
Bungee_Position 1 1568 1568 71.73 1.078e-07
Pin_Elevation 1 10153 10153 464.4 2.643e-14
Cup_Elevation 1 15313 15313 700.4 7.297e-16
Fire_Angle 1 27495 27495 1258 4.137e-18
Release_Angle 1 40755 40755 1864 1.246e-19
Fire_Angle:Bungee_Position 1 288 288 13.17 0.001917
Release_Angle:Cup_Elevation 1 544.5 544.5 24.91 9.473e-05
Release_Angle:Pin_Elevation 1 741.1 741.1 33.9 1.624e-05
Fire_Angle:Cup_Elevation 1 1152 1152 52.7 9.502e-07
Fire_Angle:Pin_Elevation 1 2016 2016 92.22 1.659e-08
Fire_Angle:Release_Angle 1 11476 11476 525 9.108e-15
Fire_Angle:Release_Angle:Cup_Elevation 1 264.5 264.5 12.1 0.002682
Fire_Angle:Release_Angle:Pin_Elevation 1 406.1 406.1 18.58 0.0004215
Residuals 18 393.5 21.86 NA NA

Final Model Equation

\[ \begin{aligned} \operatorname{\widehat{response}} &= 169.94 + 7(\operatorname{Bungee\_Position}) + 17.81(\operatorname{Pin\_Elevation}) + 21.88(\operatorname{Cup\_Elevation})\ + \\ &\quad 29.31(\operatorname{Fire\_Angle}) + 35.69(\operatorname{Release\_Angle}) + 3(\operatorname{Bungee\_Position}_{\operatorname{Fire\_Angle}} \times \operatorname{Fire\_Angle}_{\operatorname{Bungee\_Position}}) + 4.13(\operatorname{Cup\_Elevation}_{\operatorname{Release\_Angle}} \times \operatorname{Release\_Angle}_{\operatorname{Cup\_Elevation}})\ + \\ &\quad 4.81(\operatorname{Pin\_Elevation}_{\operatorname{Release\_Angle}} \times \operatorname{Release\_Angle}_{\operatorname{Pin\_Elevation}}) + 6(\operatorname{Cup\_Elevation}_{\operatorname{Fire\_Angle}} \times \operatorname{Fire\_Angle}_{\operatorname{Cup\_Elevation}}) + 7.94(\operatorname{Pin\_Elevation}_{\operatorname{Fire\_Angle}} \times \operatorname{Fire\_Angle}_{\operatorname{Pin\_Elevation}}) + 18.94(\operatorname{Fire\_Angle} \times \operatorname{Release\_Angle})\ + \\ &\quad 2.87(\operatorname{Cup\_Elevation}_{\operatorname{Fire\_Angle}} \times \operatorname{Fire\_Angle}_{\operatorname{Release\_Angle}} \times \operatorname{Release\_Angle}_{\operatorname{Cup\_Elevation}}) + 3.56(\operatorname{Pin\_Elevation}_{\operatorname{Fire\_Angle}} \times \operatorname{Fire\_Angle}_{\operatorname{Release\_Angle}} \times \operatorname{Release\_Angle}_{\operatorname{Pin\_Elevation}}) \end{aligned} \]

References

Montgomery, D. C. (2013). Design and analysis of Experiments (8th ed.). John Wiley & Sons, Inc. 

Six sigma virtual catapult. SigmaZone. (2020, May 28). Retrieved December 2, 2021, from https://sigmazone.com/catapult/.


Source Code

# Setup Libraries
library(dplyr)
library(tidyr)
library(readr)
library(knitr)
library(agricolae)
library(lawstat)
library(car)
library(GAD)
library(BSDA)
library(pwr)
library(WebPower)
library(ggplot2)
library(ggfortify)
library(ggpubr)
library(SixSigma)
library(DoE.base)
library(FrF2)
library(equatiomatic)

#------------------------------------------------------------------------------
#------------------------------------------------------------------------------
# Part 1

effectiveReleaseAngle = c(175,180,185)
firingAngle  = 90   #degrees
bungeePos    = 200  #mm
pinElevation = 200  #mm
cupElevation = 300  #mm
alpha        = 0.05 

#------------------------------------------------------------------------------
# Determine how many samples should be collected to detect a mean 
#   difference with a medium effect (i.e. 50% of the standard deviation) 
#   and a pattern of maximum variability with a probability of 75%.  
                                                                                                    
power1 = 0.75

#d is the range of means divided by sigma, k is the number of populations
d = mean(diff(effectiveReleaseAngle))
sd1 = sd(effectiveReleaseAngle)

# sigma = 5
# power = .75
# u = k - 1
# u = 2
# a = 0.05

kgroups = length(effectiveReleaseAngle)

#The effect is given by d/2 when k is even
#The effect is given by d*sqrt(k^2-1)/(2*k) when k is odd
fCal = d/sd1*sqrt(kgroups^2-1)/(2*kgroups)

sampleNumber <- pwr.anova.test(n = NULL,
                               k = kgroups,
                               f = fCal,
                               sig.level = alpha,
                               power = power1)

plot(sampleNumber)
#------------------------------------------------------------------------------
# Propose a layout using the number of samples from part (a) with 
#   randomized run order
n = ceiling(sampleNumber$n)

# set seed here for reproducibility
design <- design.crd(trt = effectiveReleaseAngle,
                     r = n,
                     seed = 1234)
design$book

#------------------------------------------------------------------------------
# Collect data and record observations on layout proposed in part (b)

recordedDistance <- c(270,275,281,272,283,269,281,
                      286,266,265,279,282,260,278,
                      281,264,270,267,277,275,267,
                      297,285,278,259,264,268,286,
                      295,263,288,268,250,273,265,
                      273,255,268,268,283,254,275)

#------------------------------------------------------------------------------
# Perform hypothesis test and check residuals.  Be sure to comment and 
#   take corrective action if necessary.
dat <- data.frame(angles   = design$book$effectiveReleaseAngle,
                  distance = recordedDistance)
dataov <- aov(distance~angles,data = dat)
summary(dataov)

datHSD <- TukeyHSD(dataov)
datHSD
plot(datHSD)
autoplot(dataov)
#------------------------------------------------------------------------------
#   If the null hypothesis is rejected, investigate pairwise comparisons. 


#------------------------------------------------------------------------------
#   State conclusions and make recommendation. 
                                                                                                   

#------------------------------------------------------------------------------
#------------------------------------------------------------------------------
# Part 2

# Determine effect of Pin Elevations and Release Angle, 
#   when Fire Angle at 100 Degrees
# 
# Bungee Position is 150m & Cup Elevation is 250mm
# 
# Pin Elevation:Fixed Effect,  100, 150, 200 mm.
# Release Angle:Random Effect, 110, 140, 170 degrees.

# model equation:
# distance~pinElevation+releaseAngle+pinElevation*releaseAngle

# init
pinElevation <- as.fixed(c(100,150,200))
releaseAngle <- as.random(c(110,140,170))

# 3x3 factorial with 3 replications
trts<-c(3,3)
design3x3 <- design.ab(trt=trts, r=3, design="crd",seed=1234)

# replace names & values
names(design3x3$book)[names(design3x3$book) == c("A","B")] <- c("pinElevation","releaseAngle")
design3x3$book$pinElevation <- pinElevation[match(design3x3$book$pinElevation,1:3)]
design3x3$book$releaseAngle <- releaseAngle[match(design3x3$book$releaseAngle,1:3)]

design3x3$book

distanceRecorded <- c(220,329,271,69,56,183,60,75,215,
                      181,53,340,182,217,209,272,48,64,
                      144,204,50,204,68,334,269,146,144)

dat2 <- data.frame(pinElevation = design3x3$book$pinElevation,
                   releaseAngle = design3x3$book$releaseAngle,
                   distance     = distanceRecorded)

dat2aov <- aov(distance~pinElevation+releaseAngle+pinElevation*releaseAngle,data = dat2)
gad(dat2aov)

autoplot(dat2aov)
#pinElevation vs releaseAngle
interaction.plot(x.factor = dat2$pinElevation, #x-axis variable
                 trace.factor = dat2$releaseAngle, #variable for lines
                 response = dat2$distance, #y-axis variable
                 fun = median, #metric to plot
                 ylab = "distance",
                 xlab = "pinElevation",
                 col = c("green", "blue", "purple"),
                 lty = 1, #line type
                 lwd = 2, #line width
                 trace.label = "releaseAngle")
dat2 %>% group_by(pinElevation, releaseAngle) %>%
  summarise(
    count = n(),
    mean = mean(distance, na.rm = TRUE),
    sd = sd(distance, na.rm = TRUE))

## Randomized Layout & Results

Fire_Angle      <- c(90 ,110) # Degrees
Bungee_Position <- c(130,150) # mm
Release_Angle   <- c(140,170) # Degrees
Pin_Elevation   <- c(125,175) # mm
Cup_Elevation   <- c(225,275) # mm

#(a & b)
trts<-c(2,2,2,2,2)
design2by5 <- design.ab(trt=trts, r=1, design="crd",seed=1234)
#design2by5

# replace names & values
colnames(design2by5$book)[-(1:2)] <- c("Fire_Angle",
                                       "Bungee_Position",
                                       "Release_Angle",
                                       "Pin_Elevation",
                                       "Cup_Elevation")
design2by5_table <- design2by5
design2by5_table$book$Fire_Angle      <- Fire_Angle[match(design2by5$book$Fire_Angle,1:2)]
design2by5_table$book$Bungee_Position <- Bungee_Position[match(design2by5$book$Bungee_Position,1:2)]
design2by5_table$book$Release_Angle   <- Release_Angle[match(design2by5$book$Release_Angle,1:2)]
design2by5_table$book$Pin_Elevation   <- Pin_Elevation[match(design2by5$book$Pin_Elevation,1:2)]
design2by5_table$book$Cup_Elevation   <- Cup_Elevation[match(design2by5$book$Cup_Elevation,1:2)]


# Results of the Trials
response <- c(237,260,343,189,135,154,121,153,312,152,
              171,114,169,114,145,135,149,197,98,134,157,
              125,262,180,124,143,101,183,238,199,104,140)

# Create the Table
design2by5table <- cbind(design2by5_table$book$Fire_Angle,
                         design2by5_table$book$Bungee_Position,
                         design2by5_table$book$Release_Angle,
                         design2by5_table$book$Pin_Elevation,
                         design2by5_table$book$Cup_Elevation,
                         response)

colnames(design2by5table) <- c("Fire Angle",
                              "Bungee Position",
                              "Release Angle",
                              "Pin Elevation",
                              "Cup Elevation",
                              "Response")

knitr::kable(design2by5table)

#(c)
io <- c(-1,1)
design2by5$book$Fire_Angle      <- io[match(design2by5$book$Fire_Angle,1:2)]
design2by5$book$Bungee_Position <- io[match(design2by5$book$Bungee_Position,1:2)]
design2by5$book$Release_Angle   <- io[match(design2by5$book$Release_Angle,1:2)]
design2by5$book$Pin_Elevation   <- io[match(design2by5$book$Pin_Elevation,1:2)]
design2by5$book$Cup_Elevation   <- io[match(design2by5$book$Cup_Elevation,1:2)]

design2by5$book <- cbind(design2by5$book,response)

datmodel <- lm(response~Fire_Angle*
                        Bungee_Position*
                        Release_Angle*
                        Pin_Elevation*
                        Cup_Elevation,
               data = design2by5$book)

modelFactorEff <- coef(datmodel)[-1]*2

halfnormal(datmodel)

halfEff <- halfnormal(datmodel)
# significant factors/interactions
knitr::kable(halfEff$signif, col.names = "Signficant Factors/Interactions")

# Final ANOVA Table
datmodelsig <- lm(response~
                    Fire_Angle:Release_Angle:Cup_Elevation + 
                    Fire_Angle:Bungee_Position + 
                    Fire_Angle:Release_Angle:Pin_Elevation + 
                    Release_Angle:Cup_Elevation + 
                    Release_Angle:Pin_Elevation + 
                    Fire_Angle:Cup_Elevation + 
                    Bungee_Position + 
                    Fire_Angle:Pin_Elevation + 
                    Pin_Elevation + 
                    Fire_Angle:Release_Angle + 
                    Cup_Elevation + 
                    Fire_Angle + 
                    Release_Angle,
                   data = design2by5$book)


dataovsig <- aov(datmodelsig)
summary(dataovsig)

## Final Model Equation
equatiomatic::extract_eq(datmodelsig, wrap = TRUE, use_coefs = TRUE)