Pre & Post Tests
Hunter Ratliff, @HunterRatliff1
August 1, 2016
Author: @HunterRatliff1
Published to: RPubs
Source Code: Available on Github
- Repo: HunterRatliff1/HSSC2016
- File: ~/HSSC2016/HS/TestResults.Rmd
Objectives
These objectives were taken from the Summer STEM Grant to Dell Med School Reporting Matrix. The matrix outlined nine objectives (see below) across three variables (i.e. columns).
In this report
- Objective #1: Increase students’ interest in health careers as measured by post-survey
- Objective #2: Improve students’ focus and interest in school
- Objective #3: Improve students’ enrollment in STEM courses at their school
- Objective #4: Improve the academic credentials of students applying to college
- Objective #5: Increase middle and high-school students’ (especially minority and disadvantaged sudents’) science knowledge and preparedness for health professions
| Historical | Projected | |
|---|---|---|
| #1 | 80% of students surveyed indicated that they were more likely to pursue a health science career after completing the camp | 80% of students surveyed will indicate greater likelihood of pursuing a health science career after completing the camp |
| #2 |
Not measured
|
In post-camp survey, 85% of students will indicate their understanding of the need to focus and do well in school |
| #3 | In 2015, 85% of campers stated that they learned the importance of math and science in the real world by attending the camp | In post-camp survey, 85% of students will indicate their intention to enroll in STEM courses at their school† |
| #4 |
Not measured
|
Through content-rich questions on pre and post-tests, 85% of students will demonstrate knowedge gained - thus improved academic readiness |
| #5 |
Not measured
|
Through content-rich questions on pre and post-tests, 85% of students will demonstrate increased science knowledge and preparedness for health professions |
| † Can further assess this goal at the end of each subsequent academic year by tracking students’ choices and performance in STEM courses | ||
Not measured here
- Objective #6: Identify promising students and foster their further development†
- Objective #7: Number of students participating in summer program (number of students registered & number of students completing)
- Objective #8: Dosage and duration of program
- Objective #9: Program continuation/attendance during the following school year
| Historical | Projected | |
|---|---|---|
| #6 | In 2015, 100 promising students were identified by their school counselors and instructed/ encouraged by program leadership and summer camp counselors | In 2016, 200 promising students will be identified by their school counselors and instructed/encouraged by program leadership and summer camp counselors‡ |
| #7 | A total of 100 students enrolled in the 2015 summer camp: 50 high-school-age, 50 middle-school-age. All completed the program | Number of students participating in the camp will double to 200: 100 high-school level, 100 middle-school level. Goal is that 90% will complete the summer program |
| #8 | In 2015, camps were 8 hours/day, 5 days a week. Two one-week sessions were held | Two one-week sessions will be held: one for middle school, one for high school. Each camp will be 8 hours/day, 5 days/week |
| #9 |
School year sessions were not held
|
50%-75% of participants will extend their involvement through planned (weekend) activities during academic year 2016-17‡ |
|
† Can assess ongoing development of students by monitoring their continued participation in the program and sustained interest and performance in STEM ‡ New element (not measurable within the summer time frame) is that 50%-75% of participants will extend their involvement through planned activities during academic year 2016-17 |
||
Test Results
First, let’s compare the distributions of the pre-tests and post-tests. Later, we can dive into the weeds by looking at the questions and individual student’s imporvements, but to begin we’ll consider the aggregate distribution of scores:
Before the camp
The distribution of the students’ scores coming into the camp were around the 40’s (mean = 44.22%, median = 42.11%; SD = 14.25%), meaning that our “average” student answered ~40 percent of the questions correctly.
Of course, not everyone scored the average score; some outpreformed the aveage and some scored a bit lower. The figure below shows this distribution as a histogram & density curve with scores on the x-axis. The relative height of each bar (or the curve) represents the higher proportion of students who earned that score.
Figure 1A
After the camp
The distribution of the camper’s scores after the camp were substantially higher than before, with the midpoint now hovering around the 80’s (mean = 80.95%, median = 84.21%; SD = 13.72%). This, in turn, indicates that our “average” student has now answered ~80% of the questions correctly.
The figure below is similar to the last figure (Figure 1A), but this time it has been constructed using the post-camp dataset. Clearly the strong right-skew indicates that most of the campers improved their scores by the end of the camp.
Figure 1B
It’s important to note, this level of analysis only considers the aggregate distributions of scores, and indicates little information about how individual students’ scores changed. In the following section, we’ll look at these same data matched in a pairwise fashon.
Paired Student’s T-test
In essence, we’re concerned with how each students’ score changed after they attended our camp
In order to consider how individual students’ scores changed during the camp, we’ll want to run a paired t-test. Recall that paired t-tests are simply one sample t-test of the difference of the matched samples. This is represented by the following equation:
\[ Score_{∆} = Score_{post} - Score_{pre} \]
Running this test on our dataset (disregarding any unmatched pairs) yeilds the following results:
# Run a paired Student's T-test (one-sided)
t.test(Dems$After, Dems$Before, alternative = "greater", paired=T)##
## Paired t-test
##
## data: Dems$After and Dems$Before
## t = 24.033, df = 65, p-value < 2.2e-16
## alternative hypothesis: true difference in means is greater than 0
## 95 percent confidence interval:
## 0.3472939 Inf
## sample estimates:
## mean of the differences
## 0.3732057Considering our benchmark for “improvement” is that a students’ score improves (meaning After - Before > 0.00%), this test clearly indicates that there was indeed improvement. According to our test, the “mean of the differences” was a + 37.32% improvement in scores.
Perhaps this is more easily shown if we consider the underlying distribution that this test is testing:
Figure 1C
Finally, we want to consider the effect size of this relationship. To accomplish this, we’ll want to determine the Cohen’s d (\(\frac{M_{after} - M_{Before}}{Pooled Standard Deviation}\)) which we find to be sufficiently large
# Determine effect size (Cohen's d)
cohen.d(Dems$After, Dems$Before, paired=T, na.rm=T)##
## Cohen's d
##
## d estimate: 2.958283 (large)
## 95 percent confidence interval:
## inf sup
## 2.454101 3.462465Scores
Figure 2
Figure 3A
Figure 3B
Figure 4A
Figure 4B
Questions
Pre-Test
Figure 5A
Figure 5B
Post-Test
Figure 6A
Figure 6B
Breakdown by topic
| QID | Letter | Choice | Before | After |
|---|---|---|---|---|
| Cardiopulmonary | ||||
| Alveolar surface tension in the lung is used to describe which of the following? | ||||
| Q14 | A | ✘ Stability of the lung surface | 42.1% | 8.3% |
| Q14 | B | ✘ How the lung is affected by gravity or weight | 26.3% | 1.2% |
| Q14 | C | ✓ How the lung is affected by hydrogen bonds in water | 21.1% | 90.5% |
| Q14 | D | ✘ How the pressure in the drops with temperature changes | 8.4% | |
| The newborn lung is made up of millions of alveoli (tiny air sacs). What chemical helps to allow the alveoli to expand during breathing | ||||
| Q16 | A | ✘ Phospholipids | 29.5% | 3.6% |
| Q16 | B | ✘ Carbonic acid | 18.9% | 6.0% |
| Q16 | C | ✘ Epinephrine | 37.9% | 9.5% |
| Q16 | D | ✓ Surfactant | 13.7% | 81.0% |
| Large arteries in the body have the capacity to sustain pressure during diastole because of | ||||
| Q17 | A | ✘ Contraction of its innermost layer | 22.1% | |
| Q17 | B | ✓ The elasticity of its wall | 43.2% | 91.7% |
| Q17 | C | ✘ Valves located in the artery | 18.9% | 3.6% |
| Q17 | D | ✘ Their large size | 15.8% | 4.8% |
| How many valves and chambers are in the human heart | ||||
| Q19 | A | ✘ 4 valves, 3 chambers | 23.4% | |
| Q19 | B | ✘ 3 valves, 4 chambers | 41.5% | |
| Q19 | C | ✘ 3 chambers, 3 valves | 4.3% | 1.2% |
| Q19 | D | ✓ 4 chambers, 4 valves | 30.9% | 98.8% |
| Infectious Disease | ||||
| Factors that lead to bacteria becoming resistant to antibiotics are: | ||||
| Q12 | A | ✘ Previous antibiotic use | 12.6% | 8.3% |
| Q12 | B | ✘ Alterations in the bacterial cell wall | 3.2% | 1.2% |
| Q12 | C | ✘ Agricultural use of antibiotics | 3.2% | 1.2% |
| Q12 | D | ✘ Transfer of genetic material from bacteria to another bacteria | 3.2% | 1.2% |
| Q12 | E | ✓ All of the above | 77.9% | 88.1% |
| In order to decrease the incidence of antibiotic resistant bacteria | ||||
| Q13 | A | ✘ Physicians should treat all patients with fever with antibiotics. | 8.4% | 2.4% |
| Q13 | B | ✘ Extend the duration of antibiotic therapy for at least two weeks. | 13.7% | 3.6% |
| Q13 | C | ✓ Limit the use of antibiotics to bacterial diseases and try to use narrow spectrum antibiotics. | 75.8% | 92.9% |
| Q13 | D | ✘ Give animals used for human consumption antibiotics so that they do not develop bacterial infections that can spread to humans. | 2.1% | 1.2% |
| Zika virus is an emerging infection with the following characteristic | ||||
| Q15 | A | ✘ It only causes infection in pregnant women | 15.8% | 4.8% |
| Q15 | B | ✘ It can be reversed with proper antiviral treatment | 9.5% | 1.2% |
| Q15 | C | ✓ It has been shown to cause a neurological disease called Guillain-Barre disease | 36.8% | 63.1% |
| Q15 | D | ✘ Is only caused by the bite of an infected mosquito | 32.6% | 27.4% |
| Q15 | E | ✘ There have been no cases of Zika virus infection in Texas | 3.2% | 3.6% |
| Ebola virus is a very virulent virus which causes severe infection in humans. The following statements are true regarding Ebola except the following: | ||||
| Q18 | A | ✘ Infectious virus and/or viral RNA can persist for weeks to months in certain bodily fluids of convalescent patients. | 11.7% | 4.8% |
| Q18 | B | ✓ Ebola virus is transmitted by the Aedes Aegypti mosquito | 42.6% | 64.3% |
| Q18 | C | ✘ Patients with Ebola virus disease commonly suffer from severe vomiting and diarrhea | 11.7% | 9.5% |
| Q18 | D | ✘ The incubation period is typically 6 to 12 days, but can range from 2 to 21 days. | 19.1% | 14.3% |
| Q18 | E | ✘ The laboratory diagnosis of Ebola virus infection is made by the detection of viral antigens or RNA in blood or other body fluids | 13.8% | 7.1% |
| Dengue is an infectious disease caused by the flavivius dengue. The following statement(s) are true about Dengue | ||||
| Q20 | A | ✘ Dengue can be prevented by the administration of a vaccine 3 weeks prior to travel to an area of high risk. | 29.8% | 3.6% |
| Q20 | B | ✘ Treatment of Dengue includes prompt administration of broad spectrum antibiotics | 8.5% | 4.8% |
| Q20 | C | ✓ Dengue commonly causes a low white blood cell count and a low platelet count | 44.7% | 86.9% |
| Q20 | D | ✘ Dengue is one of the main cause of diarrhea in travelers | 6.4% | 3.6% |
| Q20 | E | ✘ Dengue affects only patients who travel to Africa and South America | 7.4% | 1.2% |
| Microbiology | ||||
| The bacteria that Selman Waksman originally identified as a source of antibiotics are members of the genus: | ||||
| Q02 | A | ✘ Streptococcus | 26.3% | 19.0% |
| Q02 | B | ✘ Staphylococcus | 21.1% | 7.1% |
| Q02 | C | ✓ Streptomyces | 20.0% | 64.3% |
| Q02 | D | ✘ Actinomyces | 15.8% | 1.2% |
| Q02 | E | ✘ None of the above are correct | 8.4% | 3.6% |
| Q02 | F | ✘ All of the above are correct | 7.4% | 4.8% |
| The goal of streaking for isolation is to: | ||||
| Q04 | A | ✘ Dilute a concentrated population of cells | 11.6% | 16.7% |
| Q04 | B | ✘ Generate a bacterial lawn | 1.1% | 1.2% |
| Q04 | C | ✘ Separate a mixed population of bacteria | 24.2% | |
| Q04 | D | ✘ Dilute a concentrated population of cells AND generate a bacterial lawn | 15.8% | 14.3% |
| Q04 | E | ✓ Dilute a concentrated population of cells AND separate a mixed population of bacteria | 45.3% | 65.5% |
| Q04 | F | ✘ None of the above are correct | 2.1% | 2.4% |
| About 1mL of juice from raw chicken containing 1X106 CFU/mL Salmonella was cleaned using a bleach agent that claims to kill 99.9% of bacteria. Assuming the bleach agent was used properly, how many bacteria are still alive on the counter top? | ||||
| Q07 | A | ✘ 0 cells | 3.2% | 2.4% |
| Q07 | B | ✘ 10 cells | 38.3% | 23.8% |
| Q07 | C | ✘ 100 cells | 21.3% | 19.0% |
| Q07 | D | ✓ 1000 cells | 22.3% | 44.0% |
| Q07 | E | ✘ None of the above are correct | 13.8% | 10.7% |
| The correct shape of a zone of inhibition in a Kirby-Bauer disc assay should be: | ||||
| Q11 | A | ✓ Circle | 28.7% | 82.1% |
| Q11 | B | ✘ Circular blob | 14.9% | 3.6% |
| Q11 | C | ✘ Oval | 41.5% | 8.3% |
| Q11 | D | ✘ Triangle | 7.4% | 2.4% |
| Q11 | E | ✘ None are correct | 6.4% | 3.6% |
| Neuroscience | ||||
| Imagine that your eyes are closed and someone is touching your arm. What is the name for the cells in your skin that detect this touch and create a message to send to your brain about this sensation? | ||||
| Q03 | A | ✘ Gustatory receptors | 14.9% | 3.6% |
| Q03 | B | ✘ Thermoreceptors | 36.2% | 1.2% |
| Q03 | C | ✘ Magnetoreceptors | 12.8% | |
| Q03 | D | ✓ Mechanoreceptors | 36.2% | 95.2% |
| In what way can a neuron signal that a touch is occurring with the same amount of pressure but lasting for a longer time? | ||||
| Q05 | A | ✘ The neuron can fire more action potentials in the same amount of time. | 25.3% | 38.1% |
| Q05 | B | ✘ The neuron can fire fewer action potentials in the same amount of time. | 13.7% | 4.8% |
| Q05 | C | ✓ The train of action potentials can continue for a longer amount of time. | 53.7% | 57.1% |
| Q05 | D | ✘ The train of action potentials can continue for a shorter amount of time. | 7.4% | |
| What mechanisms produce an action potential in a neuron? | ||||
| Q10 | A | ✘ Water flowing through the extracellular space | 4.2% | 1.2% |
| Q10 | B | ✘ Cells exploding like popcorn | 7.4% | 1.2% |
| Q10 | C | ✓ Charged particles (ions) moving through channels that open in a cell’s membrane | 83.2% | 97.6% |
| Q10 | D | ✘ Air moving across a cell’s membrane | 5.3% | |
| Pharmacy | ||||
| What degree do pharmacists earn in the US? | ||||
| Q01 | A | ✘ BSPharm (Bachelors of Science in Pharmacy) | 36.2% | 2.4% |
| Q01 | B | ✘ PhD (Doctor of Philosophy in Pharmacy) | 23.4% | 4.8% |
| Q01 | C | ✓ PharmD (Doctor of Pharmacy) | 25.5% | 92.9% |
| Q01 | D | ✘ Masters of Science in Chemistry | 13.8% | |
| Pharmacists can add flavors to the medicine they compound | ||||
| Q06 | A | ✓ TRUE | 61.7% | 100% |
| Q06 | B | ✘ FALSE | 37.2% | |
| Over-the-counter medications take the same amount of time to work, regardless of dosage form | ||||
| Q09 | A | ✘ TRUE | 37.2% | 17.9% |
| Q09 | B | ✓ FALSE | 62.8% | 82.1% |
License
--- LICENSE ---
Copyright (C) 2016 Hunter Ratliff
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.This is an R Markdown document. Markdown is a simple formatting syntax for authoring HTML, PDF, and MS Word documents. For more details on using R Markdown see http://rmarkdown.rstudio.com.
In the spirit of Reproducible Research, below is the information About the R Session at the time it was compiled:
devtools::session_info()
## setting value
## version R version 3.2.4 (2016-03-10)
## system x86_64, darwin13.4.0
## ui X11
## language (EN)
## collate en_US.UTF-8
## tz America/Chicago
## date 2016-08-08
##
## package * version date
## assertthat 0.1 2013-12-06
## cellranger 1.0.0 2015-06-20
## chron 2.3-47 2015-06-24
## colorspace 1.2-6 2015-03-11
## curl 0.9.7 2016-04-10
## data.table 1.9.6 2015-09-19
## DBI 0.4-1 2016-05-08
## devtools 1.11.1.9000 2016-05-10
## digest 0.6.10 2016-08-02
## dplyr * 0.5.0 2016-06-24
## effsize * 0.6.4 2016-08-01
## evaluate 0.9 2016-04-29
## foreign 0.8-66 2015-08-19
## formatR 1.4 2016-05-09
## formattable * 0.1.7.1 2016-05-22
## ggplot2 * 2.1.0 2016-05-03
## ggthemes * 3.0.2 2016-05-03
## gtable 0.2.0 2016-02-26
## haven 0.2.0 2015-04-09
## htmlTable * 1.6 2016-07-24
## htmltools 0.3.5 2016-03-21
## htmlwidgets 0.7 2016-08-02
## httpuv 1.3.3 2015-08-04
## jsonlite 1.0 2016-07-01
## knitr * 1.13 2016-05-09
## labeling 0.3 2014-08-23
## lazyeval 0.2.0 2016-06-12
## magrittr 1.5 2014-11-22
## markdown 0.7.7 2015-04-22
## memoise 1.0.0 2016-01-29
## mime 0.5 2016-07-07
## munsell 0.4.3 2016-02-13
## openxlsx 3.0.0 2015-07-03
## plyr 1.8.4 2016-06-08
## R6 2.1.2 2016-01-26
## Rcpp 0.12.6 2016-07-19
## readODS 1.6.2 2016-03-09
## readr 0.2.2 2015-10-22
## readxl 0.1.1 2016-03-28
## reshape2 1.4.1 2014-12-06
## rio * 0.4.0 2016-05-01
## rmarkdown 1.0 2016-07-08
## scales * 0.4.0 2016-02-26
## shiny 0.13.2.9003 2016-05-03
## stringi 1.1.1 2016-05-27
## stringr * 1.0.0 2015-04-30
## tibble 1.1 2016-07-04
## tidyr * 0.5.1 2016-06-14
## urltools 1.4.0 2016-04-12
## withr 1.0.1 2016-02-04
## xml2 0.1.2 2015-09-01
## xtable 1.8-2 2016-02-05
## yaml 2.1.13 2014-06-12
## source
## CRAN (R 3.2.0)
## CRAN (R 3.2.0)
## CRAN (R 3.2.0)
## CRAN (R 3.2.0)
## CRAN (R 3.2.4)
## CRAN (R 3.2.0)
## cran (@0.4-1)
## Github (hadley/devtools@46bcd74)
## cran (@0.6.10)
## CRAN (R 3.2.5)
## CRAN (R 3.2.5)
## CRAN (R 3.2.5)
## CRAN (R 3.2.4)
## cran (@1.4)
## Github (renkun-ken/formattable@9e6511d)
## Github (hadley/ggplot2@59c503b)
## Github (jrnold/ggthemes@331d830)
## CRAN (R 3.2.3)
## CRAN (R 3.2.0)
## CRAN (R 3.2.5)
## CRAN (R 3.2.4)
## cran (@0.7)
## CRAN (R 3.2.0)
## cran (@1.0)
## cran (@1.13)
## CRAN (R 3.2.0)
## cran (@0.2.0)
## CRAN (R 3.2.0)
## CRAN (R 3.2.0)
## CRAN (R 3.2.3)
## cran (@0.5)
## CRAN (R 3.2.3)
## CRAN (R 3.2.0)
## cran (@1.8.4)
## CRAN (R 3.2.3)
## cran (@0.12.6)
## CRAN (R 3.2.4)
## CRAN (R 3.2.0)
## CRAN (R 3.2.4)
## CRAN (R 3.2.0)
## CRAN (R 3.2.5)
## cran (@1.0)
## CRAN (R 3.2.3)
## Github (rstudio/shiny@7e303b4)
## cran (@1.1.1)
## CRAN (R 3.2.0)
## cran (@1.1)
## cran (@0.5.1)
## CRAN (R 3.2.4)
## CRAN (R 3.2.3)
## CRAN (R 3.2.0)
## CRAN (R 3.2.3)
## CRAN (R 3.2.0)