Predicting Run-Pass Options from the Pre-Snap

IS 470 Assignment 2 · NFL Big Data Bowl 2025 tracking data

The idea

In a run-pass option (RPO) the quarterback decides after the snap whether to hand off or throw, based on what one defender does. Once the ball is snapped it is too late for the defence to adjust, so the only real counter is to see it coming.

This project builds a model that gives a live probability that a play is an RPO, updating from the moment the offence sets until the snap, and uses it to find the most common tells coaches should watch for.

• Data: NFL Big Data Bowl 2025 player tracking, weeks 1–9 of the 2022 season: 15,782 plays.
• Target: PFF’s RPO label. Only 10.3% of plays are RPOs.
• Rule: only information a defence can see before the snap is used.

Inspiration. The live, frame-by-frame idea comes from the 2025 Big Data Bowl winner, “Exposing Coverage Tells in the Presnap” (Bajaj and Sandwar, NYU), which predicted man or zone coverage as the defence moved. This project turns that idea on the offence. The second model follows SumerSports’ public transformer, which reads tracking frames directly.

How the model works

Input. One row per frame (every 0.2 s) between lineset and snap. Each row describes the look at that moment:

Group	Examples
Situation	down, distance, field position, clock, score
Personnel / formation	formation, receiver alignment, RBs, TEs, WRs
Backfield	QB depth, RB depth relative to QB, RB side vs receiver strength
Receivers	average and widest split, formation width, TEs attached to the line
Defence	box count, deep safeties, defenders on the line, safety movement
Motion	who has moved, how far, jet / orbit / across motion

Output. XGBoost (many small decision trees added together) gives the probability of an RPO at every frame:

\[P(\text{RPO}) = \frac{1}{1 + e^{-\text{score}}}\]

A logistic regression with one row per play is the baseline it has to beat.

Linked frames. Neighbouring frames are near copies, so roughly 465,000 frames really come from about 15,800 plays. Each frame is weighted by 1 ÷ (frames in its play), every frame of a play stays on the same side of the train/test split, and all results are counted per play.

Testing. Leave-one-week-out: train on 8 weeks, test on the 9th, repeat for all 9 weeks. Because RPOs are rare, the main measures are ROC-AUC, PR-AUC and balanced accuracy (the average of how well RPOs are caught and non-RPOs cleared; 50% is a coin flip).

Results

XGBoost vs the baseline

kable(cmp[, .(Model = model, `ROC-AUC` = roc_auc, `± sd` = roc_auc_sd,
              `PR-AUC` = pr_auc, `Balanced accuracy` = balanced_acc,
              `Top-10% hit rate` = top10_hit_rate)],
      caption = "Mean across the 9 held-out weeks, at the snap")

Mean across the 9 held-out weeks, at the snap

Model	ROC-AUC	± sd	PR-AUC	Balanced accuracy	Top-10% hit rate
base rate (always average)	0.500	NA	0.103	0.500	0.103
logistic regression	0.756	0.020	0.242	0.707	0.277
xgboost	0.868	0.014	0.466	0.795	0.473

How to read this table. Each number is averaged over the 9 weeks the model never trained on.

• ROC-AUC: pick one random RPO and one random non-RPO. ROC-AUC is how often the model gives the RPO the higher probability. 0.5 is a coin flip; 1.0 is perfect.
• PR-AUC: how clean the model’s top predictions are. It starts at the base rate (10.3%) for a model that knows nothing, so it is the toughest measure for a rare play like an RPO.
• Balanced accuracy: the average of the share of RPOs caught and the share of non-RPOs correctly cleared. It is not fooled by RPOs being rare; 50% is a coin flip.
• Top-10% hit rate: of the 10% of plays the model is most suspicious of, the share that really were RPOs.

XGBoost reaches a ROC-AUC of 0.868 and balanced accuracy of 79.5%, against 0.756 and 70.7% for the logistic regression. Of the 10% of plays it rates most likely, 47% are RPOs, about 4.6 times the base rate. The gap holds in every held-out week:

p <- ggplot(wk, aes(week, roc_auc, colour = model)) +
  geom_line(linewidth = 1) + geom_point(size = 2) +
  scale_x_continuous(breaks = 1:9) +
  scale_colour_manual(values = c(xgboost = "#D55E00",
                                 `logistic regression` = "#0072B2")) +
  labs(x = "Held-out week", y = "ROC-AUC", colour = NULL)
ggplotly(p)

How to read this chart. Each point is one week of games the model never saw while training. The higher the line, the better the model ranks RPOs above other plays. Orange (XGBoost) sits above blue (the baseline) in every week, and both lines are fairly flat, so the result is not driven by one lucky week.

Are the probabilities honest?

When the model says 25%, about a quarter of those plays should be RPOs.

p <- ggplot(cal, aes(predicted, actual)) +
  geom_abline(linetype = 2, colour = "grey50") +
  geom_point(aes(size = plays, text = paste0(bin, "<br>plays: ", plays)),
             colour = "#D55E00") +
  scale_x_continuous(labels = scales::percent, limits = c(0, 0.85)) +
  scale_y_continuous(labels = scales::percent, limits = c(0, 0.85)) +
  labs(x = "Predicted chance of RPO", y = "Actual RPO rate", size = "Plays")
ggplotly(p, tooltip = "text")

How to read this chart. Plays are grouped by the probability the model gave them (left to right), and each point shows how often those plays really were RPOs (bottom to top). Bigger points hold more plays. On the dashed line the model is exactly right; above it the model was too cautious; below it, too confident.

The points sit close to the diagonal. The model is slightly cautious between 20% and 70% (for example, plays rated 59% were RPOs 68% of the time) and slightly over-confident in the top bin, which only holds 81 plays.

How early can a defence be warned?

w <- melt(warn, id.vars = "secs_before_snap",
          measure.vars = c("roc_auc", "balanced_acc"))
w[, variable := fifelse(variable == "roc_auc", "ROC-AUC", "Balanced accuracy")]
p <- ggplot(w, aes(secs_before_snap, value, colour = variable)) +
  geom_line(linewidth = 1) + geom_point(size = 2) +
  scale_x_reverse(breaks = warn$secs_before_snap,
                  labels = ifelse(warn$secs_before_snap == 0, "snap",
                                  paste0(warn$secs_before_snap, "s"))) +
  scale_colour_manual(values = c("#D55E00", "#0072B2")) +
  labs(x = "Seconds before the snap", y = NULL, colour = NULL)
ggplotly(p)

On the same 11,041 plays (all set for at least 4 seconds), ROC-AUC is still 0.847 4 seconds before the snap, against 0.861 at the snap. Almost all of the signal is there as soon as the offence lines up, which gives a defence time to check into a call.

How to read this chart. Moving left means asking the model earlier, with fewer seconds of pre-snap information. Higher is better on both lines. A line that stays flat as you move left means the model already knows most of what it will know at the snap.

What it means for a coach. Once the offence is set, the warning is nearly as good as it will ever get. That leaves several seconds to signal a check, such as keeping the conflict defender home or rotating a safety.

What drives the model (SHAP)

SHAP splits each prediction into how much every feature pushed it toward or away from an RPO:

\[\text{score} = \text{average} + \text{effect}_1 + \text{effect}_2 + \dots\]

SHAP in plain English. Every play starts from the same place: the average play, about a 10% chance of an RPO. Each feature then adds or subtracts a little. A positive SHAP value pushes the play toward an RPO, and a negative one pushes it away. The values are on the model’s internal log-odds scale, which adds up neatly. This table converts them into something easier to picture:

SHAP value	Effect on the odds of an RPO
+1.0	about 2.7 × more likely
+0.5	about 1.6 × more likely
+0.25	about 1.3 × more likely
0	no effect
−0.5	about 0.6 × (40% less likely)
−1.0	about 0.37 × (63% less likely)

Which features matter most

How to read this chart. Each bar is the average size of a feature’s push, in either direction, over all frames. A long bar means the feature changes predictions a lot; it does not say whether it pushes toward or away from an RPO (the next chart does that). Dummy columns such as each formation are added together into one bar.

The same bars added up by type of information:

Formation is the strongest single feature, followed by offensive team, down, receiver splits and the running back’s position relative to the quarterback. By group, situation and backfield alignment carry the most weight, then receiver spacing and formation. Motion and the defence’s look matter far less: the RPO is given away by how the offence lines up, not by what moves afterwards.

What it means for a coach. Read the formation and the backfield first. Shotgun with the back level with the quarterback on 1st or 2nd down is where most RPOs come from. The defence’s own alignment barely changes what the offence calls.

Do high or low values raise the chance?

How to read this chart. Every dot is one frame. Its left-right position is that feature’s SHAP value for that frame: right of the centre line pushes toward an RPO, left pushes away. The colour is the feature’s value: yellow = high, purple = low. For yes/no rows such as formationSHOTGUN or teamGB, yellow means “yes”. A row that spreads far to both sides is a feature that can swing a prediction a lot.

Reading a few rows:

• formationSHOTGUN: yellow dots (shotgun) sit to the right and purple dots (not shotgun) to the left. Shotgun raises the chance of an RPO; any other formation lowers it.
• down: purple (1st down) is right and orange/yellow (3rd and 4th down) is far left. Late downs push hard away from an RPO.
• mean_wide_split: yellow (wide splits) stretches far to the right. Wide receivers lined up far from the ball push toward an RPO.
• yardsToGo: purple (short yardage) is on the right and long yardage on the left.
• teamGB / teamCLE / teamLV: the yellow dots show that being Green Bay pushes toward an RPO, while Cleveland and Las Vegas push hard away. That is team habit, not a tell in the look itself.

Where the risk jumps

How to read these charts. The bottom axis is the feature’s value; the side axis is its SHAP value, so its push toward (+) or away from (−) an RPO. Each dot is one frame, coloured by formation. Look for where the dots cross zero: that is the point where the feature switches from lowering to raising the chance. A steep section is where a small change on the field matters a lot.

RB depth vs QB

rb_dx is the running back’s depth relative to the quarterback (negative = behind). The risk peaks when the back is level with or up to about a yard behind the QB, the classic shotgun mesh, and falls sharply once he is beside or ahead of the QB.

RB side vs strength

+1 means the back lines up on the side with more receivers. That raises the chance of an RPO; lining up away from the strength lowers it. A likely reason: the run goes one way while the throw goes to the receiver side.

Receiver splits

The chance climbs steeply as the average split goes from about 10 to 15 yards. Wide splits pull defenders out of the box.

Down

1st down is the most RPO-friendly and 2nd down is still positive. 3rd and 4th down push strongly away, and the push is strongest from shotgun: a shotgun look on 3rd down usually means a pass, not an RPO.

Yards to go

RPOs live on 1st and 2nd down and short yardage, and almost disappear on 3rd and long.

QB depth

Shotgun depth (about 5 yards) pushes toward an RPO; under center pushes away.

Why single plays were flagged

How to read these charts. A waterfall explains one play. It starts at the bottom with E[f(x)], the average play (about −2.2 on the log-odds scale, roughly a 10% chance). Each bar then adds (yellow) or subtracts (red) one feature’s push, biggest first, until it reaches f(x) at the top, the play’s final score. To turn the final score into a probability: 0 is 50%, +1 is about 73%, +2 is about 88%, and −2.6 is about 7%. “Other features” groups the many small pushes. These charts use the final model trained on all weeks, so the top value can differ slightly from the held-out probability in the title.

Correct RPO call

Wide splits, short yardage, the back on the strength side and shotgun all push this play toward an RPO.

False alarm

This play looked exactly like an RPO before the snap: shotgun, wide splits, a back level with the QB, short yardage. That is the nature of an RPO; the same look is used for plain runs and passes.

Missed RPO

An under-center, singleback RPO from a team that rarely runs them. Plays like this are rare, so the model gives them a very low chance.

The tells: what coaches should watch for

Each look combines the top SHAP features. It is scored two ways:

\[\text{Lift} = \frac{\text{RPO rate when the look appears}}{\text{RPO rate overall}} \qquad \text{Coverage} = \text{share of all RPOs that show the look}\]

How to read this chart. Each bar is one look. Bar length is the lift: how many times more often an RPO happens from that look than on an average play. The dashed line at 1 is “no different from average”. The label at the end is the coverage: the share of all RPOs that came from that look. The most useful tells have both a long bar (strong) and a big label (common).

datatable(tells[, .(Rank = rank, Look = look, Plays = plays,
                    `RPO rate` = rpo_rate, Lift = lift,
                    `Share of all RPOs` = coverage,
                    `Already visible at lineset` = visible_at_lineset,
                    `Teams using it most (RPOs of plays)` = top_teams)],
          rownames = FALSE, options = list(pageLength = 15, dom = "t",
                                           scrollX = TRUE)) |>
  formatPercentage(c("RPO rate", "Share of all RPOs",
                     "Already visible at lineset"), 0)

Table columns. Plays: how often the look appeared. RPO rate: how often it was an RPO. Lift: RPO rate divided by the overall rate. Share of all RPOs: coverage. Already visible at lineset: how often the look was already in place the moment the offence set, so how early a defence can see it. Teams using it most: the three teams with the most RPOs from the look.

Reading the tell sheet:

• The strongest tell is shotgun with the running back on the receiver strength side, especially on early downs: an RPO about 3.5 times as often as normal.
• The most common tell is shotgun. It covers 83% of all RPOs, but it is also a very common look, so it is a “be alert” signal rather than a firm call.
• Attached tight ends and a receiver split wide (15+ yards) add to the risk.
• Visible early: most of these looks are already set at lineset. The exception is the running back’s side, which is often settled later, after a shift.

Which teams use each look

How to read this chart. Each square is one team (columns) and one of the top six looks (rows). Darker orange means that team runs an RPO more often when it shows that look. Teams are sorted from fewest RPOs (left) to most (right). Grey squares have fewer than 20 plays, too few to trust. Hover for the exact numbers.

top_looks <- tells[order(rank)][1:6, look]
h <- use[look %in% top_looks]
team_order <- h[, .(r = weighted.mean(rpo_rate_in_look, plays_in_look)),
                by = team][order(r), team]
h[plays_in_look < 20, rpo_rate_in_look := NA]   # too few plays to trust
h[, `:=`(team = factor(team, team_order),
         look = factor(look, rev(top_looks)))]
p <- ggplot(h, aes(team, look, fill = rpo_rate_in_look,
                   text = paste0(team, "<br>", look, "<br>RPO rate: ",
                                 fifelse(is.na(rpo_rate_in_look), "too few plays",
                                         pct(rpo_rate_in_look, 0)),
                                 "<br>plays: ", plays_in_look))) +
  geom_tile(colour = "white") +
  geom_tile(data = h[is.na(rpo_rate_in_look)], fill = "grey80", colour = "white") +
  scale_fill_gradient(low = "#F2F2F2", high = "#D55E00", na.value = "grey80",
                      labels = scales::percent, name = "RPO rate") +
  scale_y_discrete(labels = function(x) vapply(x, function(s)
    paste(strwrap(s, 30), collapse = "\n"), "")) +
  labs(x = NULL, y = NULL) +
  theme(axis.text.x = element_text(angle = 90, vjust = 0.5))
ggplotly(p, tooltip = "text")

The same look means very different things by opponent (grey cells have fewer than 20 plays). GB, PHI, KC, BUF and NYG run RPOs from these looks far more than anyone else, while TB, CLE, LV and NO almost never do. A scouting report should pair each tell with that week’s opponent.

The table below ranks the looks within each team: rank 1 is the look after which that team runs RPOs most often. Sort or filter by team to get an opponent’s own tell list. Lift for team compares the look with that team’s overall RPO rate, so 2.0 means twice as often as the team usually runs RPOs. Looks with at least 20 plays for that team are ranked first; looks with fewer plays are ranked after them and marked Small sample in the last column, because a rate from a handful of plays can be misleading.

pinfo <- fread(file.path(out, "play_info.csv"))
team_rate <- pinfo[, .(team_rpo_rate = mean(rpo)), by = .(team = possessionTeam)]
tu <- merge(use, team_rate, by = "team")
tu[, team_lift := rpo_rate_in_look / team_rpo_rate]
tu[, reliable := plays_in_look >= 20]
setorder(tu, team, -reliable, -rpo_rate_in_look, -plays_in_look)
tu[, team_rank := seq_len(.N), by = team]
tu[, sample := fifelse(reliable, "20+ plays", "Small sample (< 20 plays)")]
datatable(tu[, .(Team = team, `Rank for team` = team_rank, Look = look,
                 `Plays in look` = plays_in_look,
                 `Share of team's plays` = share_in_look,
                 `RPO rate in look` = rpo_rate_in_look,
                 `Team's overall RPO rate` = team_rpo_rate,
                 `Lift for team` = round(team_lift, 2),
                 `League-wide rank` = rank, `Sample size` = sample)],
          rownames = FALSE, filter = "top",
          options = list(pageLength = 15, scrollX = TRUE,
                         order = list(list(0, "asc"), list(1, "asc")))) |>
  formatPercentage(c("Share of team's plays", "RPO rate in look",
                     "Team's overall RPO rate"), 0) |>
  formatStyle("Look", `min-width` = "300px")

Live pre-snap demo

Press Play to run the play and Pause to stop it at any moment (Play then continues from there), or drag the slider. On the left, players move from lineset to snap, with the offence attacking to the right and the motion player highlighted. On the right is the model’s probability that the play is an RPO, using the held-out prediction from a model that never saw that week. Hover over the line for the top three reasons behind the probability at that moment.

How to read the demo. Left: the field from above at that moment. The line of scrimmage is the dotted line, the offence (orange) attacks to the right, the defence is blue, and the player who goes in motion is green. Right: the model’s probability that this play is an RPO, drawn as the play unfolds. Grey dotted lines mark line set and the moment each offensive player starts moving, detected from the tracking itself rather than the data’s motion tags, which are often recorded once the motion has already finished. Watch for the moments where the orange line jumps: they line up with something changing on the field, and hovering over the dot tells you which features caused the change.

dplays <- fread(file.path(demo, "demo_plays.csv"))
dprob  <- fread(file.path(demo, "demo_probabilities.csv"))
dtrk   <- fread(file.path(demo, "demo_tracking.csv"))
devt   <- fread(file.path(demo, "demo_events.csv"))

move_speed     <- 1.0   # yards/second that counts as "moving"
move_min_yards <- 2     # ignore players who shuffle less than this in total
side_cols <- c(Offence = "#D55E00", `Offence (motion)` = "#009E73",
               Defence = "#0072B2", Football = "#6B3E26")

make_demo <- function(id) {
  tr <- dtrk[demo_id == id]
  pr <- dprob[demo_id == id][order(frameId)]
  # Event labels: "line set" from the data, then the moment each offensive
  # player actually STARTS moving, found from the tracking itself (the data's
  # man_in_motion / shift tags are often placed when the motion is finished)
  ev <- devt[demo_id == id & event == "line_set", .(frameId, label = "line set")]
  mv <- dtrk[demo_id == id & side %in% c("Offence", "Offence (motion)") &
               !(position %in% c("T", "G", "C", "QB"))][order(nflId, frameId)]
  mv[, speed := sqrt((x_rel - shift(x_rel))^2 + (y - shift(y))^2) * 10, by = nflId]
  mv[, moving := frollapply(speed >= move_speed, 3, all, align = "left"), by = nflId]
  mv[, total := sum(sqrt(diff(x_rel)^2 + diff(y)^2)), by = nflId]
  starts <- mv[total >= move_min_yards & moving == TRUE,
               .(frameId = min(frameId)), by = .(nflId, position)][order(frameId)]
  if (nrow(starts)) {
    # players starting within 0.3 s of each other share one label (a shift)
    starts[, grp := cumsum(c(1, diff(frameId) > 3))]
    starts <- starts[, .(frameId = min(frameId),
                         label = paste(paste(unique(position), collapse = ", "),
                                       if (.N > 1) "start moving" else "starts moving")),
                     by = grp]
    ev <- rbind(ev, starts[, .(frameId, label)])
  }
  # carry the latest prediction forward to every tracking frame
  setkey(pr, frameId)
  fr <- data.table(frameId = sort(unique(tr$frameId)))
  fr <- pr[, .(frameId, p, reasons)][fr, roll = TRUE, on = "frameId"]
  fr[, t := round((frameId - min(frameId)) / 10, 1)]
  tr <- merge(tr, fr[, .(frameId, t)], by = "frameId")
  tr[, hover := fifelse(side == "Football", "Football",
                        paste0(displayName, " (", position, ")"))]
  # growing probability line: each animation frame holds the path so far
  acc <- rbindlist(lapply(fr$t, function(s) fr[t <= s][, step := s]))
  cur <- fr[, .(t, p, reasons, step = t)]

  field <- plot_ly(tr, x = ~x_rel, y = ~y, frame = ~t, ids = ~as.character(nflId),
                   color = ~side, colors = side_cols, type = "scatter",
                   mode = "markers+text", text = ~position,
                   textposition = "top center", textfont = list(size = 9),
                   hovertext = ~hover, hoverinfo = "text",
                   marker = list(size = 11, line = list(width = 1, color = "white"))) |>
    add_segments(x = 0, xend = 0, y = 0, yend = 53.3, inherit = FALSE,
                 line = list(color = "grey", dash = "dot"), showlegend = FALSE,
                 hoverinfo = "none")
  prob <- plot_ly() |>
    add_lines(data = fr, x = ~t, y = ~p, line = list(color = "#DDDDDD", width = 2),
              hoverinfo = "none", showlegend = FALSE) |>
    add_lines(data = acc, x = ~t, y = ~p, frame = ~step,
              line = list(color = "#D55E00", width = 3), showlegend = FALSE,
              hoverinfo = "none") |>
    add_markers(data = cur, x = ~t, y = ~p, frame = ~step,
                marker = list(color = "#D55E00", size = 12), showlegend = FALSE,
                hovertext = ~paste0("<b>", round(100 * p), "% RPO</b><br>", reasons),
                hoverinfo = "text")
  if (nrow(ev)) {
    ev[, t := round((frameId - min(tr$frameId)) / 10, 1)]
    setorder(ev, t)
    ev[, ylab := 0.99 - 0.06 * ((seq_len(.N) - 1) %% 3)]   # stagger labels
    for (k in seq_len(nrow(ev)))
      prob <- prob |> add_segments(x = ev$t[k], xend = ev$t[k], y = 0, yend = 1,
                                   inherit = FALSE, color = I("grey60"),
                                   line = list(color = "grey60", dash = "dot", width = 1),
                                   showlegend = FALSE, hoverinfo = "none")
    prob <- prob |> add_text(data = ev, x = ~t, y = ~ylab, text = ~label,
                             textposition = "bottom right",
                             textfont = list(size = 10, color = "grey30"),
                             showlegend = FALSE, hoverinfo = "none")
  }
  info <- dplays[demo_id == id]
  subplot(field, prob, nrows = 1, widths = c(0.5, 0.5), margin = 0.06,
          titleX = TRUE, titleY = TRUE) |>
    layout(xaxis = list(title = "Yards from the line", range = c(-12, 18),
                        zeroline = FALSE),
           yaxis = list(title = "", range = c(0, 53.3), showticklabels = FALSE),
           xaxis2 = list(title = "Seconds since the offence set",
                         range = c(0, max(fr$t) + 0.2)),
           yaxis2 = list(title = "Chance of RPO", range = c(0, 1),
                         tickformat = ".0%"),
           legend = list(orientation = "h", x = 0, y = 1.02, yanchor = "bottom"),
           margin = list(t = 40)) |>
    animation_opts(frame = 100, transition = 0, redraw = FALSE) |>
    animation_slider(currentvalue = list(prefix = "Seconds since set: "),
                     font = list(color = "rgba(0,0,0,0)")) ->
    widget
  # our own Play and Pause buttons (plotly's default button cannot pause)
  play  <- list(label = "\u25B6 Play", method = "animate",
                args = list(NULL, list(fromcurrent = TRUE, mode = "immediate",
                                       frame = list(duration = 100, redraw = FALSE),
                                       transition = list(duration = 0))))
  pause <- list(label = "\u275A\u275A Pause", method = "animate",
                args = list(list(NULL), list(mode = "immediate",
                                             frame = list(duration = 0, redraw = FALSE),
                                             transition = list(duration = 0))))
  # plotly writes its own Play-only menu when the plot is built, so build
  # first and then swap in the Play + Pause pair
  widget <- plotly_build(widget)
  menu <- widget$x$layout$updatemenus[[1]]
  menu$buttons <- list(play, pause)
  menu$direction <- "left"
  menu$xanchor <- "left"
  menu$pad <- list(t = 60, r = 10)
  widget$x$layout$updatemenus <- list(menu)
  # move the slider right so it does not sit under the buttons
  widget$x$layout$sliders[[1]]$x <- 0.2
  widget$x$layout$sliders[[1]]$len <- 0.8
  widget$height <- 620    # room for the slider and legend
  widget
}

Example 1: RPO - probability climbs

CAR vs SF | Q1, 2nd & 2 | SHOTGUN, 3x1 receivers

Held-out probability 33% at lineset → 86% at the snap. What happened: (6:35) (Shotgun) D.Foreman up the middle to SF 43 for 5 yards (O.Burks).

---

Example 2: RPO - probability climbs

ARI vs SEA | Q3, 2nd & 10 | SHOTGUN, 3x1 receivers

Held-out probability 10% at lineset → 63% at the snap. What happened: (2:08) (Shotgun) E.Benjamin up the middle to ARZ 32 for 2 yards (C.Barton).

---

Example 3: Not an RPO - probability falls

HOU vs DEN | Q3, 2nd & 8 | SHOTGUN, 3x1 receivers

Held-out probability 69% at lineset → 21% at the snap. What happened: (2:22) (No Huddle, Shotgun) D.Mills up the middle to HST 13 for 2 yards (C.Sterns).

---

The transformer: does reading frames in order help?

Why include a transformer?

XGBoost only sees what we chose to measure, such as the running back’s depth or the average receiver split. Those features carry decades of football knowledge, but they can miss things nobody thought to measure. A transformer is the type of model behind modern language AI, and SumerSports released one for this Big Data Bowl that learns straight from player tracking. It was worth testing for three reasons:

• It learns its own features. It reads every player’s position, speed and direction directly, so it could find tells we never wrote down, such as how one receiver lines up relative to his defender.
• It reads the play in order. XGBoost sees the history of a play only through summaries like “motion so far”. A transformer sees the sequence itself, so the timing and order of a shift or a motion can matter.
• It explains itself differently. Its attention shows which players and which moments it focused on, a second view of the play alongside SHAP.

How it works

1. Players become tokens. At every frame (every 0.5 s), each of the 22 players is described by position relative to the ball, speed, acceleration, direction, body orientation, distance moved since lineset, offence or defence, and position group.
2. Attention between players. Every player “looks at” every other player and learns which relationships matter, such as running back to quarterback, or receiver to the defender across from him.
3. A summary of the frame. The 22 players are combined into one description of the frame, weighted toward the players that matter most.
4. Attention across frames. Each frame looks back at the earlier frames (never ahead), so the model follows how the look develops while staying live.
5. A probability at every frame. The same leave-one-week-out testing and one-play-one-vote weighting as XGBoost, so the two are compared on equal terms.

What we found

p <- ggplot(tfc, aes(secs_before_snap, roc_auc, colour = model)) +
  geom_line(linewidth = 1) + geom_point(size = 2) +
  scale_x_reverse(breaks = 0:4, labels = c("snap", paste0(1:4, "s"))) +
  scale_colour_manual(values = c(xgboost = "#D55E00", transformer = "#0072B2")) +
  labs(x = "Seconds before the snap", y = "ROC-AUC (held-out weeks)", colour = NULL)
ggplotly(p)

How to read this chart. Both models are scored on the same held-out plays at each moment before the snap; higher is better. The line that stays higher further to the left gives the more useful early warning.

XGBoost is the better model. The transformer only edges ahead on the snap frame itself (0.880 vs 0.867). One second earlier it falls to 0.770, against XGBoost’s 0.860, and it stays behind at every earlier point.

Why it did not beat XGBoost

• The snap frame gives it a peek at the play. A genuine advantage would not appear on the last frame only. At the snap frame the play has already started: the ball moves and linemen fire out. The transformer reads raw speed and acceleration, so that one frame partly shows what happens after the snap. XGBoost, built on alignment features, barely changes between one second out and the snap.
• Not enough data for a model that starts from scratch. About 15,800 plays, of which only about 1,600 are RPOs, is small for a transformer. XGBoost starts with football knowledge built into its features; the transformer has to discover it all from raw positions.
• It was not told who the offence is. The transformer received down, distance, field position, score and time, but not the team or the formation label. Team is XGBoost’s second-strongest feature, so part of the gap is simply missing information.
• Limited training. It was trained on a regular computer’s CPU, which limited its size and how long it could train.

So XGBoost is kept: before the snap, which is the only moment a defence can use, the transformer does not beat it.

Why it is still worth keeping in the project

• Fixable weaknesses. Ending its frames half a second before the snap, adding the team and formation as inputs, and training on more seasons (or on a GPU) address every reason above.
• The two models can be combined. Averaging the two models’ probabilities often beats either one on its own, because they make different kinds of mistakes.
• New features to build. Where the transformer’s attention points to players or moments that XGBoost has no feature for, those can be measured and added to XGBoost.

What the transformer looked at

How to read this chart. This is one held-out RPO at the snap, seen from above with the offence attacking to the right. Each circle is a player (orange offence, blue defence). The bigger the circle, the more attention the other players paid to him when the model made its prediction.

The attention lands where a coach would look: on the running back and quarterback (the mesh point of the RPO), and on the widest receivers and the defenders covering them, who make up the throwing half of the option. These are the same areas SHAP ranks highly for XGBoost, which suggests both models picked up the same football logic.

Where the model misses RPOs

pred <- fread(file.path(out, "play_predictions_at_snap.csv"))
pred <- merge(pred, pinfo[, .(gameId, playId, formation, alignment, down,
                              yardsToGo)], by = c("gameId", "playId"))
best_cut <- function(y, p) {
  cuts <- seq(0.02, 0.6, 0.01)
  ba <- sapply(cuts, function(t) (mean(p[y == 1] >= t) + mean(p[y == 0] < t)) / 2)
  cuts[which.max(ba)]
}
cut0 <- best_cut(pred$rpo, pred$p_xgb)
pred[, flagged := p_xgb >= cut0]

# snap-frame alignment and motion, if the cached features are available
ff_file <- file.path(out, "frame_features.csv")
if (file.exists(ff_file)) {
  ff <- fread(ff_file, select = c("gameId", "playId", "is_snap_frame",
                                  "rb_beside_qb", "rb_dx", "rb_lat_abs",
                                  "n_skill_moved", "secs_since_ls"))[is_snap_frame == 1]
  pred <- merge(pred, ff, by = c("gameId", "playId"), all.x = TRUE)
}

team_tier <- team_rate[, .(team, tier = fcase(
  team_rpo_rate < 0.05, "Rarely runs RPOs (< 5%)",
  team_rpo_rate < 0.15, "Average RPO team (5-15%)",
  default = "RPO-heavy team (15%+)"))]
pred <- merge(pred, team_tier, by = "team")
pred[, `:=`(
  Formation = fcase(formation == "SHOTGUN", "Shotgun", formation == "PISTOL", "Pistol",
                    formation %in% c("SINGLEBACK", "I_FORM", "JUMBO"), "Under center",
                    default = "Other (empty, wildcat)"),
  Down = fifelse(down <= 2, "1st or 2nd down", "3rd or 4th down"),
  Distance = fcase(yardsToGo <= 3, "1-3 yards to go", yardsToGo <= 6, "4-6 yards to go",
                   yardsToGo <= 10, "7-10 yards to go", default = "11+ yards to go"),
  `Offence's RPO habits` = tier)]
groups <- c("Formation", "Down", "Distance", "Offence's RPO habits")
if ("rb_beside_qb" %in% names(pred)) {
  pred[, `:=`(
    Backfield = fcase(rb_beside_qb == 1, "RB beside QB",
                      !is.na(rb_dx) & rb_dx < -1.5 & rb_lat_abs < 1, "RB directly behind QB",
                      is.na(rb_dx), "No running back", default = "RB elsewhere"),
    Motion = fifelse(n_skill_moved >= 1, "Motion before the snap", "No motion"),
    `Time set before snap` = fcase(secs_since_ls < 2, "Under 2 s (quick snap)",
                                   secs_since_ls <= 5, "2-5 s", default = "Over 5 s"))]
  groups <- c(groups, "Backfield", "Motion", "Time set before snap")
}

rp <- pred[rpo == 1]
recall_all <- mean(rp$flagged)
seg <- rbindlist(lapply(groups, function(g)
  rp[, .(RPOs = .N, caught = mean(flagged), avg_p = mean(p_xgb)),
     by = .(situation = get(g))][, group := g]))
seg <- seg[RPOs >= 30 & !is.na(situation)]
seg[, gap := caught - recall_all]
weak <- seg[order(caught)][1:min(4, .N)]
gun_share <- mean(rp$Formation %in% c("Shotgun", "Pistol"))

The model flags a play when its probability is at least 11%, the cutoff that gives the best balanced accuracy. At that cutoff it catches 79% of all RPOs. The chart shows how that catch rate changes by situation. Bars to the left of the dashed line are situations where it misses more RPOs than usual.

How to read this chart. Each bar is a type of situation, and its length is the share of RPOs in that situation that the model flagged. The dashed line is the overall catch rate. Orange bars fall below it, so those are the situations where the model misses more RPOs than usual. The number in brackets is how many RPOs that situation had.

seg[, label := paste0(group, ": ", situation, " (", RPOs, " RPOs)")]
seg[, group := factor(group, levels = groups)]
seg[, label := factor(label, levels = rev(seg[order(group, -caught), label]))]
p <- ggplot(seg, aes(caught, label, fill = gap < 0,
                     text = paste0(situation, "<br>RPOs: ", RPOs,
                                   "<br>caught: ", pct(caught, 0),
                                   "<br>average predicted: ", pct(avg_p, 0)))) +
  geom_col(width = 0.7) +
  geom_vline(xintercept = recall_all, linetype = 2, colour = "grey40") +
  scale_x_continuous(labels = scales::percent, limits = c(0, 1)) +
  scale_fill_manual(values = c(`TRUE` = "#D55E00", `FALSE` = "#0072B2"),
                    guide = "none") +
  labs(x = "Share of RPOs the model caught", y = NULL)
ggplotly(p, tooltip = "text")

The weakest situations in this data are:

cat(paste0("* **", weak$situation, "** (", weak$group, "): caught ",
           pct(weak$caught, 0), " of ", weak$RPOs, " RPOs, with an average predicted chance of ",
           pct(weak$avg_p, 0), ".\n"), sep = "")

• Under center (Formation): caught 19% of 175 RPOs, with an average predicted chance of 9%.
• RB directly behind QB (Backfield): caught 26% of 218 RPOs, with an average predicted chance of 10%.
• Rarely runs RPOs (< 5%) (Offence’s RPO habits): caught 36% of 100 RPOs, with an average predicted chance of 13%.
• RB elsewhere (Backfield): caught 41% of 54 RPOs, with an average predicted chance of 16%.

Why these RPOs are missed

• They don’t look like RPOs. 88% of RPOs come from shotgun or pistol, so RPOs from under center, with the back directly behind the QB, or on 3rd down and long are rare. The model has few examples of them and learns to treat those looks as “almost never an RPO”. The missed-RPO waterfall above is this case: a singleback RPO where the formation and the offence’s history both pushed the probability down.
• Team history pulls predictions down. Team is the second-strongest feature. When an offence that rarely runs RPOs (TB, CLE, LV, NO) does call one, its history pushes the probability toward zero even if the look is right. The same thing happens when a team changes its tendencies mid-season.
• Some RPOs are decided after the snap. The pre-snap look for many RPOs is identical to a plain inside run or a quick pass. Before the snap there is simply nothing that separates them, which caps how many any pre-snap model can catch and produces false alarms on the same looks.
• Quick snaps can leave little to see. When the offence snaps soon after setting, there is less time for motion or alignment details to show (check the Time set before snap rows above).
• Label noise. The RPO label is PFF’s judgment; some “misses” may be plays charted differently from how they were designed.

How to address them

1. Give rare looks more weight in training. Up-weight RPOs from under center, 3rd down and non-shotgun looks, or train a small separate model for under-center plays, so the model does not write them off.
2. Make the team effect smarter. Replace the fixed team label with each team’s RPO rate up to that week (a rolling tendency). The model then adapts when a team changes, and can be shrunk toward the league average for teams with few RPOs.
3. Add features that separate RPOs from look-alike runs. For example, the position of the defender the quarterback would read (the overhang or apex linebacker), the box count relative to blockers using a better box definition, and receiver stance or alignment depth.
4. More data. Add more seasons so rare situations have enough examples, and check missed plays on film to find mislabelled ones.
5. Use the probability as a range, not a yes/no. For coaches, a low-but-rising probability on a rare look is still useful. A “watch” tier just below the cutoff would catch more of these plays, at the cost of some extra alarms.

Limitations and next steps

• Team identity is the second-strongest feature. That is useful for scouting a specific opponent, but part of the accuracy reflects who the offence is rather than a general tell. Rerunning with use_team <- FALSE would measure the pure tells.
• The PFF formation label is assigned at the snap. On frames before a shift, it reveals a little of where the offence ends up.
• One season, nine weeks. Rare looks, such as under-center RPOs, have few examples.
• RPO labels are PFF judgments, so some false alarms may be labelling noise. Flagged plays should be checked on film.
• Next steps: fix the weaknesses above; end the transformer’s frames before the snap and retrain it; add the RPO’s outcome (EPA) to show which defensive responses worked against each tell; and build a weekly opponent tell sheet.