Uncovering what matters:

Analyzing transitional relations among contribution types in knowledge-building discourse

Bodong Chen, Monica Resendes
University of Toronto
LAK14, March 28, 2014







The Taste of Time

Temporality in learning

  • Temporal aspects of teaching and learning are extremely important (Mercer, 2008)
  • Of broad interest in learning analytics research
    • e.g., Dyke et al., 2012; Kapur, 2011; Reimann, Markauskaite, Bannert, 2014; Wise & Chiu, 2011
    • LAK14: Raca, Tormey, & Dillenbourg, 2014; Papamitsiou, Terzis, & Economides, 2014; Aghababyan, 2014; etc.



Time

However...

  • Time is found playing almost no role as a variable in educational research (Barbera, Gros & Kirschner, 2014)
    • “Coding-and-Counting” (Suthers, 2006)
  • Learning theories generally do not take temporality into consideration (Roth, 2006)
  • A methodological gap – little guidance is available for gathering and analyzing temporal data (Littleton, 1999)
  • A technological gap – tool availability and accessibility

Knowledge Building

A case

Knowledge Building

“The production and continual improvement of ideas of value to a community” (Scardamalia & Bereiter, 2003)

  • continual idea improvement
  • emergent communal discourse
  • collective responsibility

Knowledge Forum

Metadiscourse in knowledge building


Analytic tool

  • to serve as a medium


Pedagogical interventions

  • to engage students

How do “good” dialogues look like?

Ways of contributing to explanation-building dialogues

Categories Examples
Questioning “Why leaves change color?”
Theorizing “I think it's because of the cold weather.”
Obtaining evidence “Let's put a tree in our classroom to test that theory.”
Working with evidence “I read that chlorophyll keeps leaves green. Maybe it goes into the tree to keep warm for the winter.”
Synthesis & analogies “Put our knowledge together: Because the sap can't get to the leaf, the chlorophyll dies and the leaf change color.”
Supporting discussion “I like this idea.”
“We should not talk about irrelevant stuff.”

(Chuy et al., 2011)

Resendes, Chuy, Chen, Bereiter, & Scardamalia, 2011, CSCL

Can we go further than “coding and counting”?

Any underlying patterns that distinguish productive knowledge-building discourse?

Lag-sequential Analysis (LsA)


  • Easier data preparation
  • One line of code to run LsA
# Function for comparing LsA measures
CompareLsAMeasures(measure="freq", ncodes, lag=1, adjacent=TRUE)

Participants and Data

  • Gr 1-6 students from a Knowledge Building lab school
  • One science unit per class
  • 1101 Knowledge Forum notes in total
Grades Units Number of notes
Grade 1 Water 298
Grade 2 Trees 117
Grade 3 Fungus 193
Grade 4 Rocks and Minerals 262
Grade 5/6 Astronomy 231

Primary Data Analysis

Content analysis (done prior to the present study)

  • Contribution types / ways of contributing (Chuy et al., 2011)
  • Inquiry threads: conceptual lines of inquiry (Zhang et al., 2007)
  • Productivity of each inquiry thread: based on presence of efforts to improve explanations

Secondary Data

Grades # of notes # of threads # Productive # Improvable
Grade 1 298 12 9 3
Grade 2 117 6 4 2
Grade 3 193 8 5 3
Grade 4 262 11 6 5
Grade 5/6 231 13 7 6

Secondary Data Analysis

  • Compare basic contribution measures
    • Count of contribution units
    • Count/percentage of contribution types
  • Transitional relations among contribution types
    • Lag-sequential Analysis

1. Comparing Basic Measures

Between productive and improvable dialogues, no significant difference was found on basic contribution measures.

Measures Productive Improvable
# of units 20.90 (9.15) 23.84 (12.44)
# of units–merged 14.23 (7.58) 15.74 (9.68)
Questioning 4.77 (3.33) 5.53 (3.47)
Theorizing 9.19 (5.49) 11.89 (6.34)
Obtaining evidence 2.42 (1.50) 1.89 (2.08)
Working with evidence 1.32 (2.06) 0.84 (1.64)
Synthesizing and Analogies 0.42 (0.92) 0.58 (1.02)
Supporting discussion 2.77 (3.04) 3.11 (2.66)

Coded data

# The first 10 lines of the data
#     Each line represent a note
head(dfc, 10)

  grade        thread noteid coding
1     4 FORMING ROCKS      1      Q
2     4 FORMING ROCKS      4      Q
3     4 FORMING ROCKS      6      Q
4     4 FORMING ROCKS      7      T
5     4 FORMING ROCKS      9      T
6     4 FORMING ROCKS     12     WE
7     4 FORMING ROCKS     13      T
8     4 FORMING ROCKS     15      S
9     4 FORMING ROCKS     16      T
10    4 FORMING ROCKS     18      Q

Transitional frequency

# Subset notes of ONE thread
dfc_sub <- subset(dfc, thread == t)
# Compute trasitional matrix for this thread
v_m <- GetTransactionalMatrix(dfc_sub$coding, ncodes, lag, adjacent)

     [,1] [,2] [,3] [,4] [,5] [,6]
[1,]    2    2    1    1    0    1
[2,]    2    3    2    5    0    4
[3,]    0    2    0    2    0    0
[4,]    2    4    0    2    0    2
[5,]    0    0    0    0    0    0
[6,]    0    5    1    0    0    2

2. Compare transitional Frequencies

  • Frequent transitions between questioning and theorizing
  • Productive threads had richer transitions involving:
    • working with evidence
    • synthesis and analogies

Adjusted Residuals

# Compute adjusted residuals
GetAdjustedResiduals(v_m, adjacent)

      [,1]  [,2]  [,3]  [,4] [,5]  [,6]
[1,]  1.03 -0.42  0.55 -0.55   -1 -0.26
[2,] -0.42 -1.75  0.63  1.08   -1  0.94
[3,] -0.90  0.63 -0.65  1.40   -1 -0.97
[4,]  0.44  0.33 -1.12 -0.19   -1  0.21
[5,] -1.00 -1.00 -1.00 -1.00   -1 -1.00
[6,] -1.34  1.76  0.40 -1.67   -1  0.59

Yule's Q

# Compute Yule's Q
GetYulesQ(v_m, ncodes)

      [,1]  [,2]  [,3]  [,4] [,5]  [,6]
[1,]  0.55 -0.19  0.32 -0.30   -1 -0.23
[2,] -0.06 -0.56  0.32  0.37   -1  0.23
[3,] -1.00  0.32 -1.00  0.61   -1 -1.00
[4,]  0.32  0.12 -1.00 -0.08   -1  0.00
[5,] -1.00 -1.00 -1.00 -1.00   -1 -1.00
[6,] -1.00  0.60  0.24 -1.00   -1  0.18

# Flatten Yule's Q Matrix into a vector
as.vector(t(v_m))

Compare two types of dialogues

# Combine vectors of all 50 threads
#     resulting in a 50x38 table
dft_lag_measures[1:6, 1:6]

  type count  X1_1  X2_1  X3_1  X4_1 ...
1    e    45  1.03 -0.42  0.55 -0.55
2    e    10 -0.79  1.29 -0.86 -1.00
3    i     9 -0.38  1.50 -0.57 -1.00
4    e    15 -2.35  2.84 -1.00 -1.00
5    i    35 -0.28 -1.46  1.72 -1.00
6    e    11 -0.33 -0.49  0.14 -1.00
...

# t-test: transition 1 -> 1 (questioning)
t.test(X1_1 ~ type, data=dft_lag_measures)

3. Compare transitional patterns

Significance tests

CompareLsAMeasures(measure="yule", ncodes, lag=1, adjacent=TRUE)
Next Move
Current Move 1 2 3 4 5 6
1. Questioning + -
2. Theorizing + + -
3. Obtaining Evidence +
4. Working with Evidence +
5. Syntheses & Analogies
6. Supporting Discussion - -

+: significantly more frequent in effective dialogues; -: vice versa

3. Compare transitional patterns (cont.)

  • Productive dialogues
    • working more constructively with resources
    • increasingly deepened questioning and theorizing
    • problematizing proposed explanations
  • Improvable dialogues
    • responding to theorizing and questioning by merely giving opinions

Summary

  • Temporality truly matters
  • Temporal patterns distinguish effective knowledge building discourse

In addition

  • Temporality at the community level
  • An R implementation of LsA: addressing the practice gaps

Limitations

  • Drawbacks of LsA
    • overestimation of significant results
  • The individual level is missing
  • Discourse is nonlinear
  • Other actions (e.g., reading) are not considered

Future work

  • Designing analytic tools for students
  • Connecting levels, and richer actions
  • Refining the algorithm

Thank You!