Tutorial B

Note: The focus for learning/tutoring is on Sectin E - Multiple Cases Analysis. Section B is for background information.

B. Information about the Gruenke et al 2014 study

Relevance and intervention logic

Reading comprehension is the ability to construct and extract meaning from a written text. It is considered to be the most critical skill that is needed to succeed in school. If readers have serious difficulties to gather relevant information from a historical account, a mathematical word problem, or a passage in a biology book, they are bound to fail in most every task that is put before them.

This single-case study examined the effects of a graphic organizing strategy on the ability of children to improve their text comprehension abilities. Participants were six students between ten and fourteen years old with major problems in understanding what they read. The intervention intended to teach them to visually highlight key elements of a passage, and thus, to deepen their understanding of it (story mapping).

The study was conducted in Germany. Three 5th grade students from a regular education public school and three 8th grade students from a school for children with learning difficulties served as subjects. Four of them were female (Anna, Bella, Christina, and Dunja), two of them were male (Egor and Fabian) (names altered, for anonymity).

Dependent variable measurement

18 narratives from three different German story books were selected. All of them were altered in a way that it was possible to formulate exactly ten comprehension questions about each tale that covered its main content. The comprehension questions were stated in a way that only one specific and distinct answer was possible to be counted as correct. Subsequently, we standardized the texts, so that each of them consisted of exactly 150 words. In a preliminary survey, the stories and comprehension questions were presented to twenty low achieving children between 9 and 10 years old in order to identify items that were either too easy or too hard to solve. We involved the insights from this preliminary survey to compose the final version of our question sets.

In the course of the study, each student was individually presented with a different story and a different set of comprehension questions for 18 consecutive school days. The order of the tales was randomly chosen for each child. Each student was asked to read a respective story out loud and then to write down the answers to the corresponding questions on a worksheet.

The intervention

To teach the boys and girls how to better comprehend narrative texts by using a story map, the student instructor followed a procedure outlined by Idol (1987):

1. Modeling phase: the teacher demonstrates how a story map is used by reading a tale out loud and by stopping whenever important information is mentioned to fill out parts of his or her worksheet,
2. lead phase: the children read stories independently and complete their maps while the teacher prompts and encourages them to review their work results and to add details that they might have overlooked,
3. test phase: the children read texts, draw maps of their own, ask questions pertaining to the content, answer them, and fill in the components into their maps without close supervision by the teacher; the teacher only intervenes if the students ask for or obviously needs help.

Design

An AB multiple baseline design (MBD) across subjects was applied.

E. Multiple cases analysis

Definition and characteristics of a multiple baseline design (MBD)

Example for what an MBD minimally looks like:

plotSC(c(Anna, Bella, Christina))

Question: What do you notice in this display?

The MBL addresses the thorny problem of how to study a dependent variable when it is not feasible, ethical, or desirable to reverse the initial effect.

Question Q_MBD_1: Why was there not a reversal design (ABA, or ABAB) used in this study?

writeLines(Q_MBD_1)

In learning studies, we expect knowledge/skill to be maintained
(at least in the short run) rather than dropping after instruction.

The MBL grew from the simple AB design in which repeated measurement of a variable in baseline was used to document a basic effect following introduction of the intervention (independent variable) in the B condition. The often cited problem with AB designs is not their inability to document change, but their lack of control for competing explanations for that change. Experimental research designs are constructed to allow both demonstration of change, and the inference that it is unlikely that anything other than the independent variable was responsible for the observed change in the dependent variable. Multiple baseline designs achieve this dual goal by

1. ensuring that manipulation of the independent variable is “active” rather than “passive,”
2. incorporating replication of at least three basic effects, and
3. staggering onset of the independent variable across at least three different points in time (lengths of baseline).

Active variable manipulation

The researcher actively manipulates the independent variable by selecting when and how the independent variable is introduced. As an example, a study examining the effects of teacher reinforcement on student performance would be judged to “actively” manipulate the independent variable by selecting in advance how much reinforcement a teacher would provide to a student in a class. Passive manipulation would involve simply monitoring how much reinforcement typically occurs, and this passive approach would not allow sufficient control to meet the criteria of an experimental single-case design.

Question: How is active control exercted in this study?

Replication of at least three basic effects with staggered onset of the indepedent variable

The staggered onset of the independent variable is the most signature feature of the MBL design. The staggered onset feature is achieved by first defining at least three “data series.” A data series is a set of repeated measurements (e.g., the rate of social initiations for a child each day). Defining three data series could be achieved by measuring the dependent variable across each of three or more students. The data for each student would establish a data series. Three or more data series could also, however, be created by monitoring the behavior of one student in three or more different routines (e.g., snack, morning circle, science time, play time). Also, a data series could be applied to different behaviors measured for a single participant (e.g., focusing on faces, turn taking, joint attention). These are called MBL designs across participants, settings, or behaviors, respectively.

Question: The AB design in this case is across subjects (students); does that mean there can be AB designs across other entities? For instance, across different times in the day for one subject? Across different aspects of behavior (e.g., frequency, onset, intensity) for one subject?

The intervention points were randomly determined within a preset range (1-8) for each participant.

Question Q_MBD_2: Why was the length of the baseline randomly chosen?

writeLines(Q_MBD_2)

In this way, the baseline history becomes a random factor (the opposite would
be a *fixed* factor); this allows to generalize over the baseline: For a pre-intervention 
of any length between 1 and 8 days, the intervetion can be assumed to work (if it works).

Let’s do some analysis on the six cases:

We can put all six cases into one vector so that we can apply functions to all cases when we need that.

Question Q_MBD_3: How do you create the vector allcases from the six cases?

# Your answer

`

writeLines(Q_MBD_3)

allcases <- (c(Anna, Bella, Christina, Dunja, Egor, Fabian))

Stats for all cases:

describeSC(allcases)

Describe Single-Case Data

Design:  A B 

NA

And we can look at plots of multiple students in one graph, like the first three that come from a normal school:

plotSC(c(Anna, Bella, Christina))

Question: How would you produce a multi-case plot for the three cases from the special needs school (Duna, Egor and Fabian)?

# code here

Question: What do you conclude from a visual inspection across the cases?

Your observations:

More overlap indices:

overlapSC(allcases)

Cannot compute exact p-value with tiesCannot compute exact p-value with tiesCannot compute exact p-value with tiesCannot compute exact p-value with tiesCannot compute exact p-value with tiesCannot compute exact p-value with ties

Overlap Indices

Design:  A B 
Comparing phase 1 against phase 2 

               Anna  Bella Christina Dunja   Egor Fabian
PND          100.00 100.00     85.71 83.33 100.00   0.00
PEM          100.00 100.00     85.71 91.67 100.00 100.00
PET          100.00 100.00    100.00 91.67 100.00   0.00
NAP          100.00 100.00     90.18 90.97 100.00  83.75
NAP.rescaled 100.00 100.00     80.36 81.94 100.00  67.50
PAND         100.00 100.00     80.56 80.56 100.00  77.78
TAU_U          0.43   0.53      0.49  0.46   0.46   0.06
Base_Tau       0.67   0.77      0.53  0.60   0.72  -0.69
Diff_mean      4.14   5.82      4.07  3.92   4.48   3.45
Diff_trend     0.43   0.00      0.69  0.41  -0.07  -1.30
SMD            5.07   7.13      4.99  5.20   8.17   1.11

Overall evaluation

Question: What do the data show regarding the effectiveness of the intervention?

Your evaluation 

This ends the descriptive analysis of the six cases. For more on statisticla analysis with SCAN, see the paper mentioned above.