READING COMPREHENSION

Source: https://bedfordnhlibrary.org/sites/bedfordnhlibrary.org/files/books.jpg

Background

“This study investigated the effectiveness of explicit instruction in ‘think aloud’ as a means to promote elementary students’ comprehension monitoring abilities” (Baumann & Jones, 1992).

Details of the Study (click to view)

The following explanation of the experiment comes directly from the original research article (Baumann & Jones, 1992).

Fourth-grade students from an elementary school in a rural midwestern community participated in the study. The school had adopted a team teaching plan, and three fourth-grade teachers shared an open classroom area. The teachers grouped the students for reading instruction across the three classes according to reading ability. One teacher each worked with a high [ability reading group], [one teacher worked with a] middle [ability group], and [the other teacher worked with the] low reading [ability] group. All groups used basal reading materials and were formed at the beginning of the school year on the basis of standardized achievement test data, prior performance in the basal reading program, and teacher judgment.

Sixty-eight of the 72 students in the fourth-grade team participated in the study; the four nonparticipating students were mainstreamed special education students for whom the instructional materials were considered to be too frustrating to read. The 68 students were assigned randomly to one of the three experimental groups (Strat = 23 students; DRTA = 23 students; Basal = 22 students). Because of illness and transfer to another school, incomplete data were obtained for one subject each from the Strat and DRTA groups. Thus, the final sample consisted of 66 students, 32 girls and 34 boys.

Posttests 1-3 were group-administered, quantitative measures obtained for all students in the sample. Posttest 1 was an error detection test. Posttest 2 was a comprehension monitoring questionnaire. Posttest 3 was a modified cloze test. To statistically account for students’ preexperimental differences in comprehension monitoring abilities, two pretests were constructed, administered, and used as covariates in data analyses for Posttests 1-3.

Pretest 1. The first pretest involved an error detection task designed to evaluate students’ preintervention ability to monitor their comprehension…. Students were provided the following directions which were printed on the pretest and read aloud by the experimenter: ‘Read the story carefully. Try to understand it as you read. There will be 16 sentences that do not belong in the story. Look for them as you read. Ask yourself, “What does not make sense in this story?” Underline the 16 sentences that do not make sense. Check your work when you are finished. If you cannot read a word, raise your hand, and we will help you.’ Preceding Pretest 1, students were given a brief practice exercise that required them to identify obvious intruded sentences in two short texts. Pretest 1 was scored by calculating the number of correctly identified intruded sentences.

Pretest 2. The second pretest queried students about the strategies they believed to be useful in promoting their understanding of stories. In this pretest, modeled after an instrument developed by Schmitt (1988, 1990), the students were presented with 15 multiple-choice items like the following:

When I read, it is a good idea to:

A. sound out words I don’t know.

B. make some guesses about what will happen in the story.

C. make a list of all the details in the story.

D. look up new words in the dictionary.

Posttest 1: Error detection test. Posttest 1 was the same in form as Pretest 1: with 16 intruded sentences lexically consistent at a local level but semantically inconsistent at a global level.

Posttest 2: Comprehension monitoring questionnaire. Posttest 2, consisting of 18 items, was a slightly expanded version of Pretest 2.

Posttest 3: Degrees of Reading Power. The Degrees of Reading Power (DRP) (1986) is a standardized, commercially produced instrument that employs a modified cloze format (words are deleted from a selection and for each omission the student selects the most appropriate word from a list of five). According to the authors, DRP tests “are holistic measures of how well the messages within text are understood. . . . DRP tests focus measurement on determining how well students process or construct meaning from paragraphs as they read through a selection” (DRP Handbook, 1986, p. 1). Thus, the DRP, at one level, is a general reading comprehension assessment tool.

The data for this study are contained in the Baumann dataset.

In this study done in 1992, scores were retrieved from 2 different reading comprehension tests taken in a fourth-grade class in a rural midwest town. There was a pre-test and a post-test for each test. There were a total of 66 students whose data was collected, of which 32 were girls, 34 were boys. The students were randomly seperated into 3 experimental groups and were taught using a different teaching style. The styles are as follows:

Basal
DRTA
Strat

Refer for specifics on teaching style: (Baumann & Jones, 1992)

The students were then given two tests (or the pre-tests): One to evaluate the student’s error detection and preintervention ability - the other to determine if the student understands the strategies to better comprehend what they are reading.

Later, they were given similar tests (or post-tests) on the same subject after they had been taught under the specific teaching style and their scores were recorded respectively. This gave a general idea of there reading comprehension before and after the teaching style was implemented.

It is our job to find out which teaching style produces better results for reading comprehension in fourth-grade students. We will need to manipulate the data we are given so it can be viewed in different terms in order to find the answer to the question we will be asking. (Explained below)

Getting the Composite Score

The data we are given was the number of questions right on the test. The pre-tests and post-tests of test two actually had a differing number of questions. Therefore, just subtracting the post from the pre wouldn’t give the actual change in the tests.

Because of this issue, we will take the number of questions right and divide it by the total number questions in the respective test to get the composite score. When we do this to the post’s and pre’s of both tests, we can then compare them to each other accurately.

At this point, we almost have the data where it needs to be, but for the sake of our question, we will take the differences of the post’s and pre’s of the two tests and combine them and divide by two to get an average change in composite scores. This number is a general idea of how the students’ reading comprehension improved or decreased after being taught in a specific style.

An example is given below. The values are the calculated score percentages according to the test. (Given as decimals)

The Change of Composite Score, 0.04125, means that this particular student’s reading comprehension increased by 4.125% during this experiment.

The Kruskal-Wallis Test

From here, we can run our Kruskal-Wallis Test to determine if there is a difference in one of they styles of teaching. Our hypotheses are as follows:

\[ H_o: All\ teaching\ styles\ produce\ the\ same\ results.\ \] \[ H_a: At\ least\ one\ teaching\ style's\ result\ is\ stochastically\ different. \]

Our level of significance is 0.05

Baumann$pretest.1percentage <- Baumann$pretest.1/16
Baumann$pretest.2percentage <- Baumann$pretest.2/15
Baumann$posttest.1percentage <- Baumann$post.test.1/16
Baumann$posttest.2percentage <- Baumann$post.test.2/18
Baumann$Test1dif <- Baumann$posttest.1percentage-Baumann$pretest.1percentage
Baumann$Test2dif <- Baumann$posttest.2percentage-Baumann$pretest.2percentage
Baumann$TestAvgDiff <- (Baumann$Test1dif+Baumann$Test2dif)/2

pander(kruskal.test(TestAvgDiff~group, data=Baumann))

Kruskal-Wallis rank sum test: `TestAvgDiff` by `group`
Test statistic	df	P value
21.81	2	1.834e-05 * * *

We see that our P-value is lower than our level of significance and therefore, we reject our null hypothesis. We can conclude that there is a difference in at least one of the teaching styles.

To visualize what is happening with the data, we will create a side-by-side box plot of the different styles and make a comparison.

boxplot(TestAvgDiff~group, data=Baumann, col=c("orange","navyblue","red"), ylab="Change in Composite Score (given in decimals)", main="Change in Reading Comprehension")

pander(favstats(TestAvgDiff~group, data=Baumann))

Table continues below
group	min	Q1	median	Q3	max	mean
Basal	-0.3986	-0.2257	-0.1483	-0.06024	0.06597	-0.141
DRTA	-0.1347	-0.06875	-0.002083	0.07639	0.2347	0.004703
Strat	-0.2389	-0.04722	0.06771	0.09097	0.1535	0.02456

sd	n	missing
0.1171	22	0
0.09687	22	0
0.1032	22	0

Inference

One of the first things we need to recognize is that for the post test for Test 1 was significantly more difficult. Therefore, most of the students’ post test scores for Test 1 were lower. This can mean that the composite scores may not be a completely accurate, but they give a general idea of how the students did over time.

From our box plots, we can see that the Basal teaching style is stochastically different from the other two styles. From reading the content provided in the link, we read that the Basal style is the normal style of teaching and previous tests have been based off of it. This can be a significant issue because the change in the average score for the Basal style is -14.1%. This means that in future tests, the average score for reading comprehension in fourth-graders is likely to continue to drop.

We also see that both the DRTA style and Strat style look almost similar and their average mean percentage is above zero. This means that in future tests, their scores will likely increase over time. The Strat style has the lowest average change in reading comprehension, but does have the greater average mean.

To conclude, we can see that the Basal teaching will cause the continued drop of reading comprehension in fourth-graders in coming years. It is advised to find a different style of teaching, whether it be from the DTRA style or the Strat style. More research may be required to determine what is the most effective.