Classroom Observation in the 2025 Tennessee Educator Preparation Report Card
How much does observation shape institution overall scores, and how useful is it as a predictor?
Bradley Holtz
October 23, 2026
Summary
- Classroom observation correlates moderately with institutions’ overall report card scores (r about 0.5 for the share of teachers scoring 4+). Part of that is built in, because observation is part of the score.
- Once Provider Impact points are removed from the overall score, observation shows essentially no relationship with the remaining domains (Candidate Profile, Employment, Candidate Assessment). The sample is small (at most 37 institutions), so a modest relationship can’t be ruled out.
- TVAAS tracks the overall score more closely than observation does, though it also carries more points (25 versus 15 of 80).
- Removing the two observation metrics and rescaling to the remaining 65 points changes institution scores by about 5 percentage points on average, leaves rankings largely intact, and moves 9 of 37 institutions into a different rating category.
- This analysis shows how much observation moves the current score. It cannot show whether observation is a valid measure of preparation quality, which would need teacher-level outcome data.
Data and scoring rules
The workbook has one sheet per report card domain, each with a multi-row header. The Provider Impact domain is worth 40 of 80 possible points. According to the 2023 technical report, its four scored metrics are:
| Metric | Points | Floor share | Ceiling share |
|---|---|---|---|
| Observation 3+ | 9 | 85% | 95% |
| Observation 4+ | 6 | 40% | 70% |
| TVAAS 3+ | 15 | 40% | 70% |
| TVAAS 4+ | 10 | 10% | 30% |
Points are linear between the floor (0 points) and the ceiling (full points). Level of Overall Effectiveness (LOE) is reported but unscored, so it does not feed the overall score. “Excluding observation” in this report therefore means removing the two observation metrics (15 of 80 points) from the overall score and rescaling to the remaining 65 points.
path <- file.path("data", "2025_Data_Download_TNEPPs.xlsx")
num <- function(x) suppressWarnings(as.numeric(x))
# Headers span several rows, so columns are addressed by position (c1, c2, ...)
read_positional <- function(sheet) {
x <- read_excel(path, sheet = sheet, .name_repair = "unique_quiet")
setNames(x, paste0("c", seq_along(x)))
}
pi_df <- read_positional("Provider Impact") |>
slice(-(1:2)) |>
transmute(inst_code = c1, institution = c2,
obs3 = num(c3), obs3_n = num(c4),
obs4 = num(c5), obs4_n = num(c6),
tv3 = num(c7), tv3_n = num(c8),
tv4 = num(c9), tv4_n = num(c10),
loe3 = num(c11), loe3_n = num(c12))
pr_df <- read_positional("Performance Ratings") |>
slice(-1) |>
transmute(inst_code = c1, rating = c3,
overall_pct = num(c4), overall_pts = num(c5), overall_poss = num(c6),
pi_pts = num(c17), pi_poss = num(c18)) |>
mutate(overall_ex_pi = (overall_pts - pi_pts) / (overall_poss - pi_poss))
pts <- function(x, lo, hi, max) max * pmin(pmax((x - lo) / (hi - lo), 0), 1)
dat <- inner_join(pi_df, pr_df, by = "inst_code") |>
filter(if_all(c(obs3, obs4, tv3, tv4, overall_pct), \(x) !is.na(x))) |>
mutate(p_obs3 = pts(obs3, .85, .95, 9), p_obs4 = pts(obs4, .40, .70, 6),
p_tv3 = pts(tv3, .40, .70, 15), p_tv4 = pts(tv4, .10, .30, 10),
pi_pred = p_obs3 + p_obs4 + p_tv3 + p_tv4)37 institutions have all four scored Provider Impact metrics and an overall score. Institutions with no rating (for example, those with fewer than 10 cohort members) are excluded.
Check on the rebuilt scoring rule. Provider Impact points reconstructed from the rubric differ from the published points by at most 0.09 points (rounding), so the observation points computed below are reliable.
How strongly does observation track overall scores?
Each measure is the share of an institution’s scored teachers at level 3+ or 4+. The “overall excluding Provider Impact” outcome is the score from Candidate Profile, Employment and Candidate Assessment alone. It is free of the built-in overlap with observation.
cor_tab <- function(df, predictors = c("obs3", "obs4", "tv3", "tv4")) {
expand.grid(measure = predictors,
outcome = c("overall_pct", "overall_ex_pi"),
stringsAsFactors = FALSE) |>
rowwise() |>
mutate(n = nrow(df),
pearson = cor(df[[measure]], df[[outcome]]),
ci_lo = cor.test(df[[measure]], df[[outcome]])$conf.int[1],
ci_hi = cor.test(df[[measure]], df[[outcome]])$conf.int[2],
spearman = cor(df[[measure]], df[[outcome]], method = "spearman")) |>
ungroup() |>
mutate(outcome = recode(outcome, overall_pct = "Overall score",
overall_ex_pi = "Overall excl. Provider Impact"),
measure = recode(measure, obs3 = "Observation 3+", obs4 = "Observation 4+",
tv3 = "TVAAS 3+", tv4 = "TVAAS 4+"))
}
all_cor <- cor_tab(dat)ggplot(all_cor, aes(pearson, measure)) +
geom_vline(xintercept = 0, color = "grey60") +
geom_pointrange(aes(xmin = ci_lo, xmax = ci_hi)) +
facet_wrap(~outcome) +
labs(x = "Pearson correlation (95% CI)", y = NULL,
title = "Correlation of Provider Impact measures with overall score",
subtitle = paste0("All ", nrow(dat), " institutions"))
all_cor |>
transmute(Measure = measure, Outcome = outcome,
`Pearson r` = round(pearson, 2),
`95% CI` = sprintf("(%.2f, %.2f)", ci_lo, ci_hi),
Spearman = round(spearman, 2)) |>
arrange(Outcome, Measure) |>
kable()| Measure | Outcome | Pearson r | 95% CI | Spearman |
|---|---|---|---|---|
| Observation 3+ | Overall excl. Provider Impact | 0.00 | (-0.33, 0.32) | 0.11 |
| Observation 4+ | Overall excl. Provider Impact | 0.05 | (-0.28, 0.37) | 0.16 |
| TVAAS 3+ | Overall excl. Provider Impact | 0.26 | (-0.07, 0.54) | 0.17 |
| TVAAS 4+ | Overall excl. Provider Impact | 0.18 | (-0.16, 0.47) | 0.18 |
| Observation 3+ | Overall score | 0.32 | (-0.01, 0.58) | 0.42 |
| Observation 4+ | Overall score | 0.48 | (0.19, 0.70) | 0.56 |
| TVAAS 3+ | Overall score | 0.74 | (0.54, 0.86) | 0.75 |
| TVAAS 4+ | Overall score | 0.77 | (0.59, 0.87) | 0.84 |
dat |>
select(institution, obs4, `Overall score` = overall_pct,
`Overall excl. Provider Impact` = overall_ex_pi) |>
pivot_longer(c(`Overall score`, `Overall excl. Provider Impact`),
names_to = "outcome", values_to = "score") |>
ggplot(aes(100 * obs4, 100 * score)) +
geom_point() +
facet_wrap(~outcome) +
labs(x = "Observation 4+ (% of scores at level 4 or above)",
y = "Score (% of possible points)")
With observation included in the outcome (left), the relationship is moderate. Without Provider Impact (right), there is no visible pattern.
Comparison with TVAAS
TVAAS correlates more strongly with the overall score than observation does. Because TVAAS carries more points (25 versus 15), some of that gap is also mechanical. For the score excluding Provider Impact, none of the four measures shows a correlation distinguishable from zero.
Robustness
Small programs and the choice of N cutoff
Shares from institutions with few teachers are noisy, and TVAAS N-counts are much smaller than observation N-counts. The table below treats each share as a binomial proportion and estimates a rough reliability (the fraction of between-institution variance not due to sampling noise). It ignores clustering of teachers within schools, so true reliability is probably lower.
rel <- function(p, n) 1 - mean(p * (1 - p) / n) / var(p)
cut_res <- lapply(c(0, 20, 30, 50, 75, 100, 150), \(cut) {
s <- filter(dat, obs4_n >= cut, tv4_n >= cut)
tibble(cutoff = cut, n_inst = nrow(s),
rel_obs4 = rel(s$obs4, s$obs4_n), rel_tv4 = rel(s$tv4, s$tv4_n),
r_obs4 = cor(s$obs4, s$overall_pct), r_tv4 = cor(s$tv4, s$overall_pct),
r_obs4_expi = cor(s$obs4, s$overall_ex_pi),
r_tv4_expi = cor(s$tv4, s$overall_ex_pi))
}) |> bind_rows()
cut_res |>
mutate(across(where(is.double), \(x) round(x, 2))) |>
kable(col.names = c("Min N", "Institutions", "Reliability obs 4+", "Reliability TVAAS 4+",
"r obs 4+ (overall)", "r TVAAS 4+ (overall)",
"r obs 4+ (excl. PI)", "r TVAAS 4+ (excl. PI)"))| Min N | Institutions | Reliability obs 4+ | Reliability TVAAS 4+ | r obs 4+ (overall) | r TVAAS 4+ (overall) | r obs 4+ (excl. PI) | r TVAAS 4+ (excl. PI) |
|---|---|---|---|---|---|---|---|
| 0 | 37 | 0.88 | 0.54 | 0.48 | 0.77 | 0.05 | 0.18 |
| 20 | 29 | 0.94 | 0.63 | 0.69 | 0.74 | 0.17 | 0.06 |
| 30 | 24 | 0.95 | 0.72 | 0.68 | 0.72 | 0.11 | 0.03 |
| 50 | 16 | 0.96 | 0.48 | 0.58 | 0.72 | 0.09 | 0.21 |
| 75 | 11 | 0.98 | 0.59 | 0.60 | 0.85 | -0.28 | 0.22 |
| 100 | 9 | 0.98 | 0.52 | 0.40 | 0.93 | -0.61 | 0.25 |
| 150 | 6 | 0.93 | 0.68 | -0.27 | 0.90 | -0.82 | 0.43 |
cut_res |>
select(cutoff, n_inst, `Observation 4+` = r_obs4, `TVAAS 4+` = r_tv4) |>
pivot_longer(c(`Observation 4+`, `TVAAS 4+`), names_to = "measure", values_to = "r") |>
ggplot(aes(cutoff, r, linetype = measure)) +
geom_line() +
geom_point() +
geom_text(aes(label = n_inst), vjust = -1, size = 3, check_overlap = TRUE) +
labs(x = "Minimum N-count required (both measures)", y = "Correlation with overall score",
linetype = NULL, caption = "Labels show the number of institutions remaining at each cutoff")
Above a cutoff of about 30 the sample falls to 24 institutions or fewer and the correlations become unstable, so I would not read anything into the changes at higher cutoffs. A cutoff of about 30 on both measures is a defensible compromise.
N-weighted correlations on all institutions
As an alternative to a hard cutoff, institutions are weighted by the N-count of the measure being correlated. Confidence intervals are percentile bootstrap intervals (2,000 resamples of institutions).
wcor <- function(x, y, w) {
mx <- weighted.mean(x, w); my <- weighted.mean(y, w)
sum(w * (x - mx) * (y - my)) / sqrt(sum(w * (x - mx)^2) * sum(w * (y - my)^2))
}
preds <- tribble(~measure, ~wvar,
"obs3", "obs3_n", "obs4", "obs4_n", "tv3", "tv3_n", "tv4", "tv4_n")
set.seed(6139)
w_res <- expand.grid(measure = preds$measure,
outcome = c("overall_pct", "overall_ex_pi"),
stringsAsFactors = FALSE) |>
left_join(preds, by = "measure") |>
rowwise() |>
mutate(w_pearson = wcor(dat[[measure]], dat[[outcome]], dat[[wvar]]),
boot = list(replicate(2000, {
i <- sample(nrow(dat), replace = TRUE)
wcor(dat[[measure]][i], dat[[outcome]][i], dat[[wvar]][i])
})),
ci_lo = quantile(boot, .025, na.rm = TRUE),
ci_hi = quantile(boot, .975, na.rm = TRUE)) |>
ungroup() |>
mutate(outcome = recode(outcome, overall_pct = "Overall score",
overall_ex_pi = "Overall excl. Provider Impact"),
measure = recode(measure, obs3 = "Observation 3+", obs4 = "Observation 4+",
tv3 = "TVAAS 3+", tv4 = "TVAAS 4+"))
w_res |>
transmute(Measure = measure, Outcome = outcome,
`Weighted r` = round(w_pearson, 2),
`95% bootstrap CI` = sprintf("(%.2f, %.2f)", ci_lo, ci_hi)) |>
arrange(Outcome, Measure) |>
kable()| Measure | Outcome | Weighted r | 95% bootstrap CI |
|---|---|---|---|
| Observation 3+ | Overall excl. Provider Impact | -0.22 | (-0.56, 0.28) |
| Observation 4+ | Overall excl. Provider Impact | -0.18 | (-0.56, 0.34) |
| TVAAS 3+ | Overall excl. Provider Impact | 0.14 | (-0.25, 0.43) |
| TVAAS 4+ | Overall excl. Provider Impact | 0.14 | (-0.22, 0.47) |
| Observation 3+ | Overall score | 0.30 | (-0.12, 0.68) |
| Observation 4+ | Overall score | 0.47 | (0.13, 0.77) |
| TVAAS 3+ | Overall score | 0.68 | (0.51, 0.83) |
| TVAAS 4+ | Overall score | 0.75 | (0.62, 0.86) |
The weighted results are close to the unweighted ones. Weights are raw N, so a few very large programs can dominate, particularly for observation (N up to about 1,400).
Influential institutions
d100 <- filter(dat, obs3_n > 100)
fit_pi <- lm(overall_ex_pi ~ obs4, data = d100)
d100 |>
mutate(cooks_d = cooks.distance(fit_pi),
r_without = sapply(seq_len(n()), \(i) cor(obs4[-i], overall_ex_pi[-i]))) |>
arrange(desc(cooks_d)) |>
slice_head(n = 4) |>
transmute(Institution = institution, `Obs 4+` = round(obs4, 3),
`Overall excl. PI` = round(overall_ex_pi, 2),
`Cook's D` = round(cooks_d, 2), `r without this institution` = round(r_without, 2)) |>
kable(caption = "Observation 4+ vs. overall score excluding Provider Impact, institutions with more than 100 observed teachers (n = 25). Full-sample r = 0.01.")| Institution | Obs 4+ | Overall excl. PI | Cook’s D | r without this institution |
|---|---|---|---|---|
| University of Tennessee - Chattanooga | 0.335 | 0.83 | 0.86 | 0.23 |
| Rutherford Teach Now | 0.786 | 0.92 | 0.38 | -0.15 |
| Tusculum University | 0.621 | 0.46 | 0.10 | 0.00 |
| Lee University | 0.407 | 0.60 | 0.09 | -0.06 |
sens <- d100 |>
filter(!institution %in% c("University of Tennessee - Chattanooga", "Rutherford Teach Now"))
cor_tab(sens, c("obs3", "obs4", "tv3", "tv4")) |>
transmute(Measure = measure, Outcome = outcome, n,
`Pearson r` = round(pearson, 2), `95% CI` = sprintf("(%.2f, %.2f)", ci_lo, ci_hi),
Spearman = round(spearman, 2)) |>
arrange(Outcome, Measure) |>
kable(caption = "Sensitivity analysis: N > 100, excluding UT Chattanooga and Rutherford Teach Now.")| Measure | Outcome | n | Pearson r | 95% CI | Spearman |
|---|---|---|---|---|---|
| Observation 3+ | Overall excl. Provider Impact | 23 | 0.15 | (-0.28, 0.53) | 0.15 |
| Observation 4+ | Overall excl. Provider Impact | 23 | 0.09 | (-0.34, 0.48) | 0.09 |
| TVAAS 3+ | Overall excl. Provider Impact | 23 | -0.11 | (-0.50, 0.31) | -0.05 |
| TVAAS 4+ | Overall excl. Provider Impact | 23 | -0.04 | (-0.44, 0.38) | 0.04 |
| Observation 3+ | Overall score | 23 | 0.34 | (-0.08, 0.66) | 0.45 |
| Observation 4+ | Overall score | 23 | 0.60 | (0.24, 0.81) | 0.65 |
| TVAAS 3+ | Overall score | 23 | 0.52 | (0.14, 0.77) | 0.60 |
| TVAAS 4+ | Overall score | 23 | 0.68 | (0.38, 0.86) | 0.82 |
UT Chattanooga (lowest observation 4+ share, high score on the other domains) and Rutherford Teach Now (the opposite corner) pull the correlation in opposite directions, so the near-zero result excluding Provider Impact is fragile in both directions. The correlation with the overall score is stable when any single institution is dropped. These two institutions were chosen from the diagnostics above, so this sensitivity analysis is data-driven and should not be treated as the primary result.
What happens to scores without classroom observation?
The counterfactual score removes the observation points (Observation 3+ and 4+) from each institution’s earned points and drops 15 points from the total possible, leaving 65.
rate <- \(x) cut(x, c(-Inf, .5, .75, Inf),
labels = c("Does Not Meet", "Meets", "Exceeds"), right = FALSE)
cf <- dat |>
mutate(obs_pts = p_obs3 + p_obs4,
pct_no_obs = (overall_pts - obs_pts) / (overall_poss - 15),
diff_pp = 100 * (pct_no_obs - overall_pct),
rating_actual = rate(overall_pct), rating_no_obs = rate(pct_no_obs),
rank_actual = rank(-overall_pct), rank_no_obs = rank(-pct_no_obs))
cf |>
summarise(Institutions = n(),
`Mean change (pp)` = mean(diff_pp),
`Mean absolute change (pp)` = mean(abs(diff_pp)),
`Range of change (pp)` = sprintf("%.1f to %.1f", min(diff_pp), max(diff_pp)),
`Correlation of scores` = cor(overall_pct, pct_no_obs),
`Spearman` = cor(overall_pct, pct_no_obs, method = "spearman"),
`Mean absolute rank shift` = mean(abs(rank_actual - rank_no_obs)),
`Rating category changes` = sum(rating_actual != rating_no_obs)) |>
mutate(across(where(is.double), \(x) round(x, 2))) |>
pivot_longer(everything(), names_to = "Quantity", values_to = "Value", values_transform = as.character) |>
kable()| Quantity | Value |
|---|---|
| Institutions | 37 |
| Mean change (pp) | -3.35 |
| Mean absolute change (pp) | 5.3 |
| Range of change (pp) | -14.6 to 9.5 |
| Correlation of scores | 0.95 |
| Spearman | 0.94 |
| Mean absolute rank shift | 2.76 |
| Rating category changes | 9 |
cf_long <- cf |>
select(institution, `With observation` = overall_pct, `Without observation` = pct_no_obs) |>
pivot_longer(-institution, names_to = "scenario", values_to = "score") |>
mutate(score = 100 * score)
ggplot(cf, aes(y = reorder(institution, overall_pct))) +
geom_vline(xintercept = c(50, 75), linetype = "dashed", color = "grey60") +
geom_segment(aes(x = 100 * overall_pct, xend = 100 * pct_no_obs), color = "grey50") +
geom_point(data = cf_long, aes(x = score, y = institution, color = scenario)) +
labs(x = "Overall score (% of possible points)", y = NULL, color = NULL,
title = "Institution overall scores with and without classroom observation",
caption = "Dashed lines: 50% (Meets) and 75% (Exceeds) rating thresholds") +
theme(legend.position = "top")
ggplot(cf, aes(diff_pp)) +
geom_histogram(binwidth = 2.5, boundary = 0, color = "white") +
geom_vline(xintercept = 0, linetype = "dashed") +
labs(x = "Change in overall score when observation is excluded (percentage points)",
y = "Institutions")
cf |>
count(`Rating with observation` = rating_actual, `Rating without observation` = rating_no_obs) |>
pivot_wider(names_from = `Rating without observation`, values_from = n, values_fill = 0) |>
kable(caption = "Rating category with and without observation (rows = with observation). Ratings are computed from score thresholds (below 50%, 50-74.9%, 75% and above) and match the published ratings for most but not all institutions, probably because of rounding near the boundaries.")| Rating with observation | Does Not Meet | Meets | Exceeds |
|---|---|---|---|
| Does Not Meet | 3 | 1 | 0 |
| Meets | 4 | 14 | 1 |
| Exceeds | 0 | 3 | 11 |
cf |>
filter(rating_actual != rating_no_obs) |>
transmute(Institution = institution,
`With observation (%)` = round(100 * overall_pct, 1),
`Without observation (%)` = round(100 * pct_no_obs, 1),
`Change (pp)` = round(diff_pp, 1),
`Rating with` = rating_actual, `Rating without` = rating_no_obs) |>
arrange(desc(abs(`Change (pp)`))) |>
kable(caption = "Institutions whose rating category changes when observation is excluded.")| Institution | With observation (%) | Without observation (%) | Change (pp) | Rating with | Rating without |
|---|---|---|---|---|---|
| Maryville College | 57.8 | 47.1 | -10.7 | Meets | Does Not Meet |
| University of Tennessee - Martin | 54.8 | 44.5 | -10.2 | Meets | Does Not Meet |
| Teach For America - Nashville-Chattanooga | 69.5 | 79.0 | 9.5 | Meets | Exceeds |
| Tusculum University | 52.1 | 43.5 | -8.6 | Meets | Does Not Meet |
| Johnson University | 57.5 | 49.4 | -8.1 | Meets | Does Not Meet |
| Lincoln Memorial University | 76.5 | 71.1 | -5.4 | Exceeds | Meets |
| University of Memphis | 76.5 | 71.1 | -5.4 | Exceeds | Meets |
| Belmont University | 78.9 | 74.2 | -4.7 | Exceeds | Meets |
| Lee University | 49.4 | 52.5 | 3.1 | Does Not Meet | Meets |
On average institutions lose points when observation is removed, because observation 3+ is near its ceiling for most institutions and so contributes almost full points. The institutions that lose the most do well on observation but less well on the other metrics. Most rating changes are institutions within about 10 points of a rating boundary.
Is observation a worthwhile predictor of overall scores?
- Overall score. Observation 4+ is moderately correlated with the overall score, but observation is part of that score, so this is partly mechanical.
- Remaining domains. Observation shows essentially no relationship with the score from Candidate Profile, Employment and Candidate Assessment. With 23 to 37 institutions I cannot rule out a modest relationship.
- Redundancy. Removing observation and rescaling leaves institution rankings largely unchanged (correlation about 0.95). If the goal is to predict or rank on the overall score, observation adds little beyond TVAAS and the other measures. Observation 3+ is the least informative because most institutions are near its ceiling.
- Rating changes. Even so, about a quarter of institutions would change rating category, so the observation metrics matter at the margin.
Limits. The sample is small, the results are sensitive to a few institutions, and several analysis choices (N cutoffs, weights, rescaling the score to 65 points) were mine. I did not formally test differences between correlations. Most importantly, a weak link to the other report card domains does not mean observation is uninformative about teaching quality. Judging that would need teacher-level data, such as whether observation scores predict later student growth or retention.