1 Start here

This document is the complete executable codebase. It does not source the old scripts or require objects from an existing R session. Keep the data and config folders next to it. Open this file in RStudio and select Knit, or run:

rmarkdown::render("Stock_Forecast_Master.Rmd", envir = new.env())

Install requirements once, if necessary:

install.packages(c("DBI", "RSQLite", "quantmod", "TTR", "zoo", "ranger",
                   "jsonlite", "knitr", "rmarkdown"))

The default is an offline, fresh model run using the supplied database. It rebuilds features and backtests; it never silently substitutes legacy RDS results. The initial full run includes all three stocks, all four horizons, MU earnings comparisons, calibration, and the semiconductor panel. It can take several minutes or longer, depending on the computer. There is no background scheduler or automatic trading.

To download newer prices, set refresh_prices: true. For earnings downloads, set refresh_earnings: true and put ALPHAVANTAGE_KEY=your_key in your .Renviron file, then restart R. Do not publish the key. Pandoc is required for HTML rendering and is bundled with RStudio. Internet access is required only for installation and requested data refreshes, not for the default analysis.

Use execute: false to render documentation without running models. That is explicitly a documentation-only report, not a forecast. The use_earnings parameter optionally includes event features in the main individual-stock models; keep it false unless you supply matching earnings files for every selected ticker. The separate MU comparison tests whether earnings features help.

2 Methods and interpretation

2.1 What is forecast

For each horizon h, the target is log adjusted-price return:

\[y_{t,h}=\log(P^{adj}_{t+h}/P^{adj}_{t}).\]

Each stock and horizon uses its own quantile random forest. The panel model pools 20 semiconductor stocks, with scale-free features, and forecasts MU at 5 and 20 trading days. It is compared with the MU specialist; the two are not blended.

All technical price inputs use adjusted OHLC consistently. The forest predicts adjusted-price returns. Published price equivalents multiply the current raw close by the exponentiated return quantile. This preserves today’s quoted-price anchor, but these are return-implied price equivalents, not a separate model of future dividends or splits. Adjusted close itself is not presented as a quoted market price. A median return is labeled median, not an expected mean return.

2.2 Honest evaluation

Annual expanding-window tests use only rows whose target end date is strictly before the first test observation. This purges overlapping training labels by actual dates, including in the panel. Individual-stock benchmarks predict zero return and the training sample’s average return. Tables show MAE in log-return units, directional accuracy, and empirical interval coverage.

Overlapping test windows are dependent. Counts divided by horizon are only a rough non-overlap count, not a formal effective sample size. Pooled panel results also have cross-stock dependence. The panel universe was supplied as a fixed list; this is not a point-in-time constituent universe, so survivorship/selection bias has not been eliminated. No transaction costs, portfolio returns, or trading strategy performance are inferred from these prediction metrics.

2.3 Uncertainty summaries

Quantiles produce nominal 50%, 80%, and 95% intervals. Probability estimates use an equally spaced quantile grid and are approximate model probabilities. The narrowest 30% log-return interval is retained as the concentration zone; its midpoint is not a statistically estimated mode. Legacy likely_price_mode is kept only as a compatibility alias for that midpoint.

The original 0–100 confidence score is preserved as a backtest quality score: half directional edge and half closeness of observed 80% coverage to 80%. It is not a probability of correctness. Raw and conformal forecasts retain separate model identifiers; the score describes raw backtest performance, not a new claim of calibrated forecast accuracy.

Conformal calibration uses a chronological holdout, purges labels crossing its start, and selects non-overlapping calibration outcomes. A further untouched chronological audit set measures calibrated coverage. Non-overlap alone does not establish exchangeability in market data, so nominal coverage is a target, not an unconditional time-series guarantee. Sparse calibration makes very high coverage intervals unavailable rather than fabricating finite bounds. The calibrated forest fits an older training subset; its median can differ from the forest trained on all available labels.

2.4 Earnings information

The supplied CSVs contain historical reporting dates and EPS fields. Calendar features are supported. Downloaded EPS and surprises are retained for inspection but are not silently introduced into the predictor set. A historical report date alone does not prove when that date was announced. Therefore past-event features are the default; forward-looking event features require a separate schedule CSV with earnings_date and known_on columns. Without that evidence, future events are treated as unknown. No approximate quarterly placeholders enter training.

Dates are calendar dates; horizon membership is evaluated against a trading-day calendar, not by comparing calendar-day distances with trading-day horizons. Historical features become available only on the next observed trading day after an earnings date, a conservative convention when release time is unknown.

3 Configuration and dependencies

Setting Value
Execution Fresh offline or refreshed-data run
Tickers MU, TSLA, PLTR
Horizons (trading days) 5, 20, 63, 126
Refresh prices FALSE
Panel TRUE
Calibration TRUE
Trees: backtest / forecast / panel 300 / 1000 / 300
Seed 42

4 Database and input integrity

The original database is retained in data/stock_model.sqlite. Execution creates results/working_stock_model.sqlite and modifies only that working copy. The input snapshot’s 12 legacy predictions remain intact. Feature tables are rebuilt from validated prices; legacy backtest summaries are not treated as fresh evidence.

The supplied snapshot has 48 rows before the configured 2015 start date, including 1975 dates for MU and index ETFs, with malformed download timestamps. Their true dates cannot safely be reconstructed. They are exported to a quarantine CSV and excluded from the working analysis. No replacement prices are invented. All other rows remain subject to structural validation; this is not external verification of the vendor’s historical prices.

fix_date <- function(x) {
  if (inherits(x, "Date")) return(x)
  if (is.numeric(x)) return(as.Date(x, origin = "1970-01-01"))
  as.Date(as.character(x))
}

with_db <- function(path, fun) {
  con <- DBI::dbConnect(RSQLite::SQLite(), path)
  on.exit(DBI::dbDisconnect(con), add = TRUE)
  fun(con)
}

initialize_database <- function(path) {
  with_db(path, function(con) DBI::dbExecute(con, "
    CREATE TABLE IF NOT EXISTS prices_daily (
      ticker TEXT NOT NULL, date TEXT NOT NULL, open REAL, high REAL, low REAL,
      close REAL, adjusted_close REAL, volume REAL, source TEXT,
      downloaded_at TEXT, PRIMARY KEY (ticker, date))"))
  invisible(path)
}

prepare_database <- function(input_path, output_dir) {
  work <- file.path(output_dir, "working_stock_model.sqlite")
  if (normalizePath(input_path, mustWork = FALSE) ==
      normalizePath(work, mustWork = FALSE)) stop("Input and working database must differ.")
  if (file.exists(input_path)) {
    src <- DBI::dbConnect(RSQLite::SQLite(), input_path)
    dst <- DBI::dbConnect(RSQLite::SQLite(), work)
    tryCatch(RSQLite::sqliteCopyDatabase(src, dst), finally = {
      DBI::dbDisconnect(src)
      DBI::dbDisconnect(dst)
    })
  } else if (!cfg$refresh_prices) {
    stop("Missing input database. Restore data/stock_model.sqlite or enable refresh_prices.")
  } else if (file.exists(work)) {
    stop("Input missing but working database exists; choose a new output_dir for a fresh download.")
  }
  initialize_database(work)
  work
}

validate_prices <- function(path, from, output_dir) {
  x <- with_db(path, function(con) DBI::dbReadTable(con, "prices_daily"))
  if (!nrow(x)) stop("No price rows available.")
  x$date <- fix_date(x$date)
  numeric_cols <- c("open", "high", "low", "close", "adjusted_close", "volume")
  if (!all(numeric_cols %in% names(x))) stop("Incomplete price schema.")
  reason <- rep("", nrow(x))
  mark <- function(mask, label) {
    mask[is.na(mask)] <- TRUE
    reason[mask] <<- ifelse(nzchar(reason[mask]),
                           paste(reason[mask], label, sep = "; "), label)
  }
  mark(is.na(x$date), "Unparseable date")
  mark(x$date < as.Date(from), "Before configured start date")
  mark(x$date > Sys.Date(), "Future observation date")
  mark(!Reduce(`&`, lapply(x[numeric_cols], is.finite)), "Nonfinite OHLCV")
  mark(x$close <= 0 | x$adjusted_close <= 0 | x$open <= 0 | x$low <= 0,
       "Nonpositive price")
  mark(x$volume < 0 | x$high < x$low | x$high < pmax(x$open, x$close) - 1e-6 |
         x$low > pmin(x$open, x$close) + 1e-6, "Invalid OHLCV ordering")
  mark(is.na(x$ticker) | !nzchar(x$ticker), "Missing ticker")
  bad <- nzchar(reason)
  quarantined <- cbind(x[bad, , drop = FALSE], reason = reason[bad])
  write.csv(quarantined, file.path(output_dir, "quarantined_prices.csv"), row.names = FALSE)
  clean <- x[!bad, , drop = FALSE]
  if (anyDuplicated(clean[c("ticker", "date")])) stop("Duplicate ticker/date prices; resolve before modeling.")
  clean <- clean[order(clean$ticker, clean$date), ]
  clean$date <- as.character(clean$date)
  with_db(path, function(con) DBI::dbWithTransaction(con, {
    DBI::dbExecute(con, "DELETE FROM prices_daily")
    DBI::dbAppendTable(con, "prices_daily", clean)
  }))
  list(rows_retained = nrow(clean), rows_quarantined = sum(bad),
       tickers = length(unique(clean$ticker)))
}

save_predictions <- function(rows, path) {
  if (is.null(rows) || !nrow(rows)) return(invisible(NULL))
  table <- "predictions_consolidated"
  rows <- as.data.frame(rows)
  with_db(path, function(con) DBI::dbWithTransaction(con, {
    if (!DBI::dbExistsTable(con, table)) {
      DBI::dbWriteTable(con, table, rows[FALSE, ], row.names = FALSE)
      DBI::dbExecute(con, paste0("CREATE UNIQUE INDEX IF NOT EXISTS consolidated_key ON ",
        table, " (ticker, observation_date, horizon, model_version)"))
    }
    fields <- DBI::dbListFields(con, table)
    if (!setequal(fields, names(rows))) stop("Prediction schema changed; use a new output_dir.")
    DBI::dbWriteTable(con, "tmp_consolidated_predictions", rows[, fields], overwrite = TRUE)
    quoted <- paste(DBI::dbQuoteIdentifier(con, fields), collapse = ", ")
    DBI::dbExecute(con, paste0("INSERT OR REPLACE INTO ", table, " (", quoted,
      ") SELECT ", quoted, " FROM tmp_consolidated_predictions"))
    DBI::dbRemoveTable(con, "tmp_consolidated_predictions")
  }))
  invisible(rows)
}

5 Market and earnings downloads

Downloads are explicit operations. Price updates upsert by ticker/date. If a requested download fails, the run stops instead of silently labeling stale data as refreshed. The supplied offline snapshot remains usable without credentials.

download_market_data <- function(tickers, from = cfg$data_from, db_path) {
  initialize_database(db_path)
  for (ticker in unique(tickers)) {
    message("Downloading ", ticker)
    z <- quantmod::getSymbols(ticker, src = "yahoo", from = from,
                              auto.assign = FALSE, warnings = FALSE)
    d <- data.frame(ticker = ticker, date = as.character(zoo::index(z)),
      open = as.numeric(quantmod::Op(z)), high = as.numeric(quantmod::Hi(z)),
      low = as.numeric(quantmod::Lo(z)), close = as.numeric(quantmod::Cl(z)),
      adjusted_close = as.numeric(quantmod::Ad(z)), volume = as.numeric(quantmod::Vo(z)),
      source = "quantmod/yahoo", downloaded_at = format(Sys.time(), tz = "UTC", usetz = TRUE))
    if (!nrow(d)) stop("No prices downloaded for ", ticker)
    with_db(db_path, function(con) DBI::dbWithTransaction(con, {
      DBI::dbWriteTable(con, "tmp_prices", d, overwrite = TRUE)
      fields <- paste(DBI::dbQuoteIdentifier(con, names(d)), collapse = ", ")
      DBI::dbExecute(con, paste0("INSERT OR REPLACE INTO prices_daily (", fields,
        ") SELECT ", fields, " FROM tmp_prices"))
      DBI::dbRemoveTable(con, "tmp_prices")
    }))
  }
  invisible(tickers)
}

fetch_earnings <- function(ticker, key = Sys.getenv("ALPHAVANTAGE_KEY"), out_dir = "config") {
  if (!nzchar(key)) stop("Set ALPHAVANTAGE_KEY in .Renviron before refreshing earnings.")
  url <- paste0("https://www.alphavantage.co/query?function=EARNINGS&symbol=",
                utils::URLencode(ticker, reserved = TRUE), "&apikey=",
                utils::URLencode(key, reserved = TRUE))
  res <- tryCatch(jsonlite::fromJSON(url), error = function(e)
    stop("Earnings request failed for ", ticker, "; check connectivity and API access."))
  if (!is.null(res$Note) || !is.null(res$Information) || !is.null(res$`Error Message`))
    stop("Earnings provider rejected request for ", ticker, "; check quota and credentials.")
  q <- res$quarterlyEarnings
  if (!is.data.frame(q) || !nrow(q)) stop("No quarterly earnings for ", ticker)
  num <- function(v) suppressWarnings(as.numeric(v))
  full <- data.frame(ticker = ticker, reported_date = as.Date(q$reportedDate),
    fiscal_period_end = as.Date(q$fiscalDateEnding), reported_eps = num(q$reportedEPS),
    estimated_eps = num(q$estimatedEPS), surprise = num(q$surprise),
    surprise_pct = num(q$surprisePercentage))
  full <- full[!is.na(full$reported_date), ]
  full <- full[order(full$reported_date), ]
  if (!nrow(full)) stop("No valid earnings dates.")
  dir.create(out_dir, recursive = TRUE, showWarnings = FALSE)
  write.csv(full, file.path(out_dir, paste0("earnings_full_", ticker, ".csv")), row.names = FALSE)
  write.csv(data.frame(earnings_date = unique(full$reported_date)),
            file.path(out_dir, paste0("earnings_", ticker, ".csv")), row.names = FALSE)
  full
}

fetch_earnings_batch <- function(tickers, pause = 15, ...) {
  out <- list()
  for (i in seq_along(tickers)) {
    out[[tickers[i]]] <- fetch_earnings(tickers[i], ...)
    if (i < length(tickers)) Sys.sleep(pause)
  }
  out
}

load_earnings_dates <- function(ticker, dir = "config") {
  path <- file.path(dir, paste0("earnings_", ticker, ".csv"))
  if (!file.exists(path)) stop("Missing earnings dates: ", path)
  x <- read.csv(path)
  if (!"earnings_date" %in% names(x)) stop("Expected earnings_date column.")
  d <- as.Date(x$earnings_date)
  if (anyNA(d)) stop("Invalid earnings dates in ", path)
  sort(unique(d))
}

make_earnings_template <- function(ticker, dir = "config") {
  path <- file.path(dir, paste0("earnings_", ticker, ".csv"))
  if (!file.exists(path)) write.csv(data.frame(earnings_date = character()), path, row.names = FALSE)
  invisible(path)
}

inspect_earnings <- function(ticker, dir = "config") {
  x <- read.csv(file.path(dir, paste0("earnings_full_", ticker, ".csv")))
  list(records = nrow(x), by_year = table(substr(x$reported_date, 1, 4)), data = x)
}

6 Features, calendars, and targets

MACD is computed in price units with percent=FALSE before normalization; this avoids dividing an already percentage-based MACD by price. OBV is retained for inspection but excluded from the model. Each horizon records its actual target end date for leakage checks. Missing labels at the recent tail do not prevent forecasting the latest complete feature row.

Future NYSE-style session dates are estimated with a built-in holiday calendar. The calendar covers standard holidays, observed days, Good Friday, Juneteenth from 2022, and the 2025-01-09 closure. Unscheduled future closures cannot be known; future target dates are explicitly estimates. Historical target labels always use observed stock sessions.

safe_ratio <- function(a, b) ifelse(is.finite(b) & abs(b) > 1e-12, a / b, NA_real_)

feature_columns <- function() c(
  "Dist_SMA20", "Dist_SMA50", "Dist_SMA200", "Range_Position_252", "Range_Position_60",
  "RSI_14", "MACD_Norm", "MACD_HistN", "ATR_Pct", "BB_pctB", "ADX_14", "Return",
  "Vol_20", "ret_SPY", "ret_QQQ", "ret_SOXX", "rel_spy", "rel_soxx")

feature_columns_panel <- feature_columns

earnings_columns <- function() c(
  "days_until_earnings", "days_since_earnings", "is_earnings_day", "post_earn_d1",
  "post_earn_d2", "post_earn_d3", "post_earn_d4", "post_earn_d5", "pre_earn_week",
  "earnings_in_horizon", "earnings_schedule_known")

feature_columns_with_earnings <- function() c(feature_columns(), earnings_columns())

calculate_technical_features <- function(db_path) {
  prices <- with_db(db_path, function(con)
    DBI::dbGetQuery(con, "SELECT * FROM prices_daily ORDER BY ticker, date"))
  pieces <- lapply(split(prices, prices$ticker), function(x) {
    x$date <- fix_date(x$date)
    x <- x[order(x$date), ]
    if (nrow(x) < 260) stop("Insufficient price history for ", x$ticker[1])
    ratio <- x$adjusted_close / x$close
    a <- x$adjusted_close
    hi <- x$high * ratio
    lo <- x$low * ratio
    hlc <- cbind(hi, lo, a)
    x$SMA20 <- TTR::SMA(a, 20)
    x$SMA50 <- TTR::SMA(a, 50)
    x$SMA200 <- TTR::SMA(a, 200)
    x$Dist_SMA20 <- safe_ratio(a - x$SMA20, x$SMA20)
    x$Dist_SMA50 <- safe_ratio(a - x$SMA50, x$SMA50)
    x$Dist_SMA200 <- safe_ratio(a - x$SMA200, x$SMA200)
    for (w in c(60, 252)) {
      low <- zoo::rollapplyr(lo, w, min, fill = NA)
      high <- zoo::rollapplyr(hi, w, max, fill = NA)
      x[[paste0("Range_Position_", w)]] <- safe_ratio(a - low, high - low)
      if (w == 252) x$Dist_Annual_Support <- safe_ratio(a - low, low)
    }
    x$RSI_14 <- TTR::RSI(a, 14)
    macd <- TTR::MACD(a, nFast = 12, nSlow = 26, nSig = 9, percent = FALSE)
    x$MACD <- macd[, "macd"]
    x$MACD_Hist <- macd[, "macd"] - macd[, "signal"]
    x$MACD_Norm <- x$MACD / a
    x$MACD_HistN <- x$MACD_Hist / a
    x$ATR_14 <- TTR::ATR(hlc, 14)[, "atr"]
    x$ATR_Pct <- x$ATR_14 / a
    bb <- TTR::BBands(a, n = 20, sd = 2)
    x$BB_pctB <- safe_ratio(a - bb[, "dn"], bb[, "up"] - bb[, "dn"])
    x$OBV <- TTR::OBV(a, x$volume)
    x$ADX_14 <- TTR::ADX(hlc, 14)[, "ADX"]
    x$Return <- c(NA_real_, diff(a) / head(a, -1))
    x$Vol_20 <- zoo::rollapplyr(x$Return, 20, sd, fill = NA) * sqrt(252)
    x$date <- as.character(x$date)
    x
  })
  out <- do.call(rbind, pieces)
  rownames(out) <- NULL
  with_db(db_path, function(con)
    DBI::dbWriteTable(con, "features_daily", out, overwrite = TRUE, row.names = FALSE))
  out$date <- fix_date(out$date)
  out
}

build_model_dataset <- function(target_ticker = "MU", horizons = cfg$horizons,
                                db_path = cfg$db_path, features = NULL) {
  if (is.null(features)) features <- with_db(db_path, function(con)
    DBI::dbReadTable(con, "features_daily"))
  features$date <- fix_date(features$date)
  if (!all(CONTEXT_TICKERS %in% features$ticker)) stop("Missing market context ETFs.")
  d <- features[features$ticker == target_ticker, ]
  if (!nrow(d)) stop("No features for ", target_ticker)
  d <- d[order(d$date), ]
  for (ticker in CONTEXT_TICKERS) {
    ctx <- features[features$ticker == ticker, c("date", "Return")]
    if (anyDuplicated(ctx$date)) stop("Duplicate context dates.")
    d[[paste0("ret_", ticker)]] <- ctx$Return[match(d$date, ctx$date)]
  }
  d$rel_spy <- d$Return - d$ret_SPY
  d$rel_soxx <- d$Return - d$ret_SOXX
  for (h in horizons) {
    n <- nrow(d)
    future <- seq_len(n) + h
    valid <- future <= n
    target <- rep(NA_real_, n)
    end <- rep(as.Date(NA), n)
    target[valid] <- log(d$adjusted_close[future[valid]] / d$adjusted_close[valid])
    end[valid] <- d$date[future[valid]]
    d[[paste0("target_", h, "d")]] <- target
    d[[paste0("target_end_", h)]] <- end
  }
  stopifnot(!anyDuplicated(d$date), all(diff(as.numeric(d$date)) > 0))
  rownames(d) <- NULL
  d
}

build_panel_dataset <- function(horizons = c(5, 20), db_path, features = NULL) {
  if (is.null(features)) features <- with_db(db_path, function(con)
    DBI::dbReadTable(con, "features_daily"))
  missing <- setdiff(PANEL_TICKERS, unique(features$ticker))
  if (length(missing)) stop("Panel missing tickers: ", paste(missing, collapse = ", "))
  d <- do.call(rbind, lapply(PANEL_TICKERS, function(t)
    build_model_dataset(t, horizons, db_path, features)))
  rownames(d) <- NULL
  stopifnot(!anyDuplicated(d[c("ticker", "date")]))
  d
}

nth_weekday <- function(year, month, weekday, nth) {
  d <- seq(as.Date(sprintf("%d-%02d-01", year, month)), by = "day", length.out = 31)
  d <- d[as.integer(format(d, "%m")) == month & as.integer(format(d, "%w")) == weekday]
  if (nth == -1) tail(d, 1) else d[nth]
}

easter_date <- function(y) {
  a <- y %% 19; b <- y %/% 100; c <- y %% 100; d <- b %/% 4; e <- b %% 4
  f <- (b + 8) %/% 25; g <- (b - f + 1) %/% 3
  h <- (19 * a + b - d - g + 15) %% 30; i <- c %/% 4; k <- c %% 4
  l <- (32 + 2 * e + 2 * i - h - k) %% 7; m <- (a + 11 * h + 22 * l) %/% 451
  month <- (h + l - 7 * m + 114) %/% 31
  day <- (h + l - 7 * m + 114) %% 31 + 1
  as.Date(sprintf("%d-%02d-%02d", y, month, day))
}

market_holidays <- function(years) {
  observe <- function(d, new_year = FALSE) {
    w <- as.integer(format(d, "%w"))
    if (w == 0) d + 1 else if (w == 6 && !new_year) d - 1 else d
  }
  out <- unlist(lapply(years, function(y) {
    fixed <- c(observe(as.Date(paste0(y, "-01-01")), TRUE),
               observe(as.Date(paste0(y, "-07-04"))), observe(as.Date(paste0(y, "-12-25"))))
    if (y >= 2022) fixed <- c(fixed, observe(as.Date(paste0(y, "-06-19"))))
    c(fixed, nth_weekday(y, 1, 1, 3), nth_weekday(y, 2, 1, 3), easter_date(y) - 2,
      nth_weekday(y, 5, 1, -1), nth_weekday(y, 9, 1, 1), nth_weekday(y, 11, 4, 4))
  }))
  unique(c(as.Date(out, origin = "1970-01-01"), as.Date("2025-01-09")))
}

next_sessions <- function(date, n = 126) {
  d <- seq(as.Date(date) + 1, by = "day", length.out = n * 3 + 30)
  holidays <- market_holidays(unique(as.integer(format(d, "%Y"))))
  d <- d[!as.integer(format(d, "%w")) %in% c(0, 6) & !d %in% holidays]
  head(d, n)
}

add_earnings_features <- function(df, ticker = "MU", horizon_check = 20,
                                  announce_lead = 30, dir = "config") {
  ed <- load_earnings_dates(ticker, dir)
  schedule_path <- file.path(dir, paste0("earnings_schedule_", ticker, ".csv"))
  schedule <- NULL
  if (file.exists(schedule_path)) {
    schedule <- read.csv(schedule_path)
    if (!all(c("earnings_date", "known_on") %in% names(schedule))) stop("Invalid earnings schedule schema.")
    schedule$earnings_date <- as.Date(schedule$earnings_date)
    schedule$known_on <- as.Date(schedule$known_on)
    if (anyNA(schedule) || any(schedule$known_on > schedule$earnings_date)) stop("Invalid known_on dates.")
  }
  d <- df$date
  prior <- vapply(seq_along(d), function(i) {
    past <- ed[ed < d[i]]
    if (length(past)) as.numeric(d[i] - max(past)) else 60
  }, numeric(1))
  until <- rep(announce_lead, length(d))
  known <- in_horizon <- integer(length(d))
  future_calendar <- c(d, next_sessions(max(d), max(126, horizon_check)))
  if (!is.null(schedule)) for (i in seq_along(d)) {
    visible <- schedule$earnings_date[schedule$known_on <= d[i] & schedule$earnings_date > d[i]]
    if (length(visible)) {
      event <- min(visible)
      known[i] <- 1L
      until[i] <- min(as.numeric(event - d[i]), announce_lead)
      boundary <- future_calendar[match(d[i], future_calendar) + horizon_check]
      in_horizon[i] <- as.integer(event <= boundary)
    }
  }
  df$days_until_earnings <- until
  df$days_since_earnings <- pmin(prior, 60)
  # The legacy name now means first session after a historical reporting date.
  df$is_earnings_day <- as.integer(vapply(seq_along(d), function(i) {
    before <- if (i == 1) d[i] - 1 else d[i - 1]
    any(ed >= before & ed < d[i])
  }, logical(1)))
  for (j in 1:5) df[[paste0("post_earn_d", j)]] <- as.integer(prior == j)
  df$pre_earn_week <- as.integer(known == 1 & until <= 5)
  df$earnings_in_horizon <- in_horizon
  df$earnings_schedule_known <- known
  df
}

7 Forests and walk-forward evaluation

The default forest settings preserve the original approach: quantile regression forests, fixed random seed, and separate fits per horizon. Tree counts are configurable. Outputs retain every year’s results and also pool errors by test observation count. Confidence scores are based on these pooled raw-model results.

log_progress <- function(...) {
  line <- paste0(format(Sys.time(), "%H:%M:%S"), " ", paste0(..., collapse = ""))
  cat(line, "\n", file = file.path(cfg$output_dir, "progress.log"), append = TRUE)
  message(line)
}

finite_rows <- function(d, cols) {
  if (!all(cols %in% names(d))) stop("Missing columns: ", paste(setdiff(cols, names(d)), collapse = ", "))
  Reduce(`&`, lapply(d[cols], is.finite))
}

model_data <- function(df, horizon, cols) {
  target <- paste0("target_", horizon, "d")
  end <- paste0("target_end_", horizon)
  if (!all(c(target, end) %in% names(df))) stop("Dataset does not contain horizon ", horizon)
  d <- df[finite_rows(df, c(cols, target)) & !is.na(df[[end]]), , drop = FALSE]
  d[order(d$date, d$ticker), , drop = FALSE]
}

fit_forest <- function(train, horizon, cols, trees = cfg$backtest_trees) {
  target <- paste0("target_", horizon, "d")
  if (nrow(train) < 100) stop("Insufficient labeled training rows.")
  ranger::ranger(dependent.variable.name = target, data = train[c(cols, target)],
    num.trees = as.integer(trees), quantreg = TRUE, seed = cfg$seed,
    num.threads = cfg$threads, min.node.size = 5, importance = "none")
}

forest_quantiles <- function(fit, newdata, cols, probs) {
  p <- predict(fit, data = newdata[cols], type = "quantiles",
               quantiles = probs, num.threads = cfg$threads)$predictions
  p <- matrix(p, nrow = nrow(newdata), ncol = length(probs))
  if (ncol(p) > 1) p <- t(apply(p, 1, sort))
  p
}

metric_row <- function(actual, quantiles, naive_mean, horizon, year, scope, ticker, model) {
  if (!length(actual)) return(NULL)
  data.frame(model = model, ticker = ticker, scope = scope, horizon = horizon,
    test_year = year, n_test = length(actual),
    rough_nonoverlap_count = length(actual) / horizon,
    mae_model = mean(abs(actual - quantiles[, 4])),
    mae_naive_zero = mean(abs(actual)), mae_naive_mean = mean(abs(actual - naive_mean)),
    dir_acc_model = mean(sign(actual) == sign(quantiles[, 4])),
    dir_acc_naive = mean(sign(actual) == sign(naive_mean)),
    cov50 = mean(actual >= quantiles[, 3] & actual <= quantiles[, 5]),
    cov80 = mean(actual >= quantiles[, 2] & actual <= quantiles[, 6]),
    cov95 = mean(actual >= quantiles[, 1] & actual <= quantiles[, 7]))
}

walk_forward_backtest <- function(df, horizon = 5, first_test_year = cfg$first_test_year,
                                  cols = feature_columns(), panel = FALSE,
                                  num_trees = cfg$backtest_trees) {
  d <- model_data(df, horizon, cols)
  target <- paste0("target_", horizon, "d")
  end <- paste0("target_end_", horizon)
  years <- sort(unique(as.integer(format(d$date, "%Y"))))
  qs <- c(.025, .10, .25, .50, .75, .90, .975)
  rows <- folds <- list()
  for (year in years[years >= first_test_year]) {
    test <- d[as.integer(format(d$date, "%Y")) == year, , drop = FALSE]
    start <- min(test$date)
    train <- d[d$date < start & d[[end]] < start, , drop = FALSE]
    minimum <- if (panel) 2000 else 400
    if (nrow(train) < minimum || !nrow(test)) next
    stopifnot(max(train[[end]]) < min(test$date))
    log_progress(if (panel) "Panel" else unique(df$ticker)[1], " ", horizon, "d / ", year)
    fit <- fit_forest(train, horizon, cols, num_trees)
    pr <- forest_quantiles(fit, test, cols, qs)
    naive <- mean(train[[target]])
    model <- if (panel) "panel" else "specialist"
    ticker <- if (panel) "ALL" else unique(df$ticker)[1]
    rows[[length(rows) + 1L]] <- metric_row(test[[target]], pr, naive, horizon,
                                          year, "all", ticker, model)
    if (panel && any(test$ticker == "MU")) {
      mu <- test$ticker == "MU"
      rows[[length(rows) + 1L]] <- metric_row(test[[target]][mu], pr[mu, , drop = FALSE],
        mean(train[[target]][train$ticker == "MU"]), horizon, year, "MU", "MU", model)
    }
    folds[[length(folds) + 1L]] <- data.frame(model = model, ticker = ticker,
      horizon = horizon, test_year = year, train_rows = nrow(train), test_rows = nrow(test),
      last_training_target = as.character(max(train[[end]])), first_test = as.character(start))
  }
  if (!length(rows)) stop("No valid backtest folds for ", unique(df$ticker)[1], " / ", horizon)
  out <- do.call(rbind, rows)
  attr(out, "fold_audit") <- do.call(rbind, folds)
  out
}

walk_forward_long <- walk_forward_backtest

walk_forward_panel <- function(df, horizon = 5, first_test_year = cfg$first_test_year,
                                num_trees = cfg$panel_trees) {
  walk_forward_backtest(df, horizon, first_test_year, feature_columns_panel(), TRUE, num_trees)
}

summarize_backtest <- function(bt) {
  metrics <- c("mae_model", "mae_naive_zero", "mae_naive_mean", "dir_acc_model",
               "dir_acc_naive", "cov50", "cov80", "cov95")
  groups <- split(bt, interaction(bt$model, bt$ticker, bt$scope, bt$horizon, drop = TRUE))
  out <- lapply(groups, function(x) {
    row <- x[1, c("model", "ticker", "scope", "horizon")]
    row$n_test <- sum(x$n_test)
    row$years <- nrow(x)
    row$rough_nonoverlap_count <- sum(x$rough_nonoverlap_count)
    for (m in metrics) row[[m]] <- weighted.mean(x[[m]], x$n_test)
    row
  })
  d <- do.call(rbind, out)
  rownames(d) <- NULL
  d
}

summarize_panel <- summarize_backtest

confidence_from_backtest <- function(row) {
  if (nrow(row) != 1 || !is.finite(row$dir_acc_model) || !is.finite(row$cov80))
    return(list(score = NA_real_, label = "Unavailable"))
  direction <- min(max((row$dir_acc_model - .5) / .15, 0), 1)
  coverage <- 1 - min(abs(row$cov80 - .8) / .2, 1)
  score <- round(100 * (.5 * direction + .5 * coverage))
  label <- as.character(cut(score, c(-1, 39, 59, 74, 89, 100),
    labels = c("Low", "Cautious", "Moderate", "High", "Very high")))
  list(score = score, label = label)
}

detect_regime <- function(features, ticker, as_of) {
  spy <- features[features$ticker == "SPY" & features$date <= as_of &
                    is.finite(features$SMA200), ]
  v <- features[features$ticker == ticker & features$date <= as_of &
                  is.finite(features$Vol_20), ]
  if (!nrow(spy) || !nrow(v)) return("Unavailable")
  spy <- spy[order(spy$date), ]; v <- v[order(v$date), ]
  trend <- if (tail(spy$adjusted_close, 1) > tail(spy$SMA200, 1)) "Uptrend" else "Downtrend"
  vol <- if (tail(v$Vol_20, 1) > quantile(v$Vol_20, .7)) "high-volatility" else "normal-volatility"
  paste(trend, vol, sep = ", ")
}

hdi_from_quantiles <- function(values, probs, mass = .30) {
  steps <- round(mass / diff(probs)[1])
  if (steps >= length(values)) stop("Quantile grid too short for concentration zone.")
  starts <- seq_len(length(values) - steps)
  best <- starts[which.min(values[starts + steps] - values[starts])]
  list(lo = values[best], hi = values[best + steps],
       midpoint = (values[best] + values[best + steps]) / 2)
}

latest_observation <- function(df, ticker, cols) {
  d <- df[df$ticker == ticker & finite_rows(df, cols), , drop = FALSE]
  if (!nrow(d)) stop("No complete latest features for ", ticker)
  d <- d[order(d$date), ]
  tail(d, 1)
}

prediction_row <- function(fit, latest, horizon, cols, summary_row, features,
                            version = MODEL_VERSION, calibration = NULL) {
  grid <- seq(.005, .995, .005)
  v <- as.numeric(forest_quantiles(fit, latest, cols, grid))
  q <- function(p) v[which.min(abs(grid - p))]
  zone <- hdi_from_quantiles(v, grid)
  price <- latest$close
  conf <- confidence_from_backtest(summary_row)
  intervals <- list(`50` = c(q(.25), q(.75)), `80` = c(q(.1), q(.9)),
                    `95` = c(q(.025), q(.975)))
  status <- "Raw forest quantiles"
  if (!is.null(calibration)) {
    for (level in names(intervals)) {
      m <- calibration$margins[[level]]
      intervals[[level]] <- if (is.finite(m)) intervals[[level]] + c(-m, m) else c(NA_real_, NA_real_)
    }
    # Conservatively maintain nesting across the available central intervals.
    previous <- NULL
    for (level in names(intervals)) {
      current <- intervals[[level]]
      if (all(is.finite(current))) {
        if (!is.null(previous)) current <- c(min(current[1], previous[1]), max(current[2], previous[2]))
        intervals[[level]] <- current
        previous <- current
      }
    }
    unavailable <- names(intervals)[!vapply(intervals, function(x) all(is.finite(x)), logical(1))]
    status <- if (length(unavailable)) paste("Unavailable coverage levels:", paste(unavailable, collapse = ","))
      else "Conformal margins applied; see independent audit"
  }
  convert <- function(level, index) price * exp(intervals[[level]][index])
  calibrated <- !is.null(calibration)
  data.frame(ticker = latest$ticker, prediction_time = format(Sys.time(), tz = "UTC", usetz = TRUE),
    observation_date = as.character(latest$date),
    target_date = as.character(next_sessions(latest$date, horizon)[horizon]),
    target_date_is_estimate = TRUE, horizon = horizon, current_price = price,
    predicted_return_q50 = q(.5), predicted_price_q50 = price * exp(q(.5)),
    predicted_price_q025 = convert("95", 1), predicted_price_q10 = convert("80", 1),
    predicted_price_q25 = convert("50", 1), predicted_price_q75 = convert("50", 2),
    predicted_price_q90 = convert("80", 2), predicted_price_q975 = convert("95", 2),
    likely_price_lo = if (calibrated) NA_real_ else price * exp(zone$lo),
    likely_price_hi = if (calibrated) NA_real_ else price * exp(zone$hi),
    likely_price_mode = if (calibrated) NA_real_ else price * exp(zone$midpoint),
    prob_within_5pct = if (calibrated) NA_real_ else mean(v >= log(.95) & v <= log(1.05)),
    prob_up = if (calibrated) NA_real_ else mean(v > 0),
    confidence_score = conf$score, confidence_label = conf$label,
    market_regime = detect_regime(features, latest$ticker, latest$date),
    model_version = version, interval_status = status)
}

generate_prediction <- function(df, horizon, bt_summary, features,
                                ticker = unique(df$ticker)[1], cols = feature_columns(),
                                panel = FALSE) {
  latest <- latest_observation(df, ticker, cols)
  train <- model_data(df, horizon, cols)
  # A panel can contain newer observations for other stocks; exclude unavailable labels.
  train <- train[train[[paste0("target_end_", horizon)]] <= latest$date, , drop = FALSE]
  fit <- fit_forest(train, horizon, cols, if (panel) cfg$panel_trees else cfg$forecast_trees)
  summary_row <- bt_summary[bt_summary$ticker == ticker & bt_summary$horizon == horizon, ]
  if (panel) summary_row <- summary_row[summary_row$scope == "MU", ]
  prediction_row(fit, latest, horizon, cols, summary_row, features,
                 if (panel) paste0(MODEL_VERSION, "_panel") else MODEL_VERSION)
}

generate_panel_prediction <- function(df, horizon, bt_panel_summary, features) {
  generate_prediction(df, horizon, bt_panel_summary, features, "MU", feature_columns_panel(), TRUE)
}

print_forecast <- function(p) {
  print(p[c("ticker", "observation_date", "horizon", "current_price", "predicted_price_q50",
            "predicted_price_q10", "predicted_price_q90", "prob_up", "confidence_score")], row.names = FALSE)
  invisible(p)
}

8 Conformal calibration and independent audit

The chronological split allocates approximately 60% to fitting, 25% to calibration, and 15% to the audit, with target-date purging at both boundaries. Calibration intervals are widened only; negative conformity adjustments are clipped at zero. If the finite-sample rank exceeds the number of calibration observations, that coverage level is unavailable. Independent audit coverage is reported together with its small, non-overlapping observation count.

nonoverlapping_rows <- function(d, horizon) {
  if (!nrow(d)) return(d)
  d <- d[order(d$date), ]
  end <- paste0("target_end_", horizon)
  keep <- integer()
  last_end <- as.Date("1900-01-01")
  for (i in seq_len(nrow(d))) if (d$date[i] > last_end) {
    keep <- c(keep, i)
    last_end <- d[[end]][i]
  }
  d[keep, , drop = FALSE]
}

conformal_calibrate <- function(df, horizon, cols = feature_columns()) {
  d <- model_data(df, horizon, cols)
  if (length(unique(d$ticker)) != 1) stop("Conformal calibration currently supports individual stocks.")
  n <- nrow(d)
  if (n < 700) stop("Insufficient history for chronological calibration/audit.")
  target <- paste0("target_", horizon, "d")
  end <- paste0("target_end_", horizon)
  cal_start <- d$date[floor(n * .60) + 1L]
  audit_start <- d$date[floor(n * .85) + 1L]
  train <- d[d$date < cal_start & d[[end]] < cal_start, ]
  calib <- d[d$date >= cal_start & d$date < audit_start & d[[end]] < audit_start, ]
  audit <- d[d$date >= audit_start, ]
  calib <- nonoverlapping_rows(calib, horizon)
  audit <- nonoverlapping_rows(audit, horizon)
  if (nrow(train) < 400 || nrow(calib) < 2 || !nrow(audit)) stop("Insufficient purged calibration split.")
  stopifnot(max(train[[end]]) < cal_start, max(calib[[end]]) < audit_start)
  fit <- fit_forest(train, horizon, cols, cfg$forecast_trees)
  margins <- list()
  checks <- list()
  for (coverage in c(.5, .8, .95)) {
    alpha <- 1 - coverage
    pr <- forest_quantiles(fit, calib, cols, c(alpha / 2, 1 - alpha / 2))
    scores <- pmax(pr[, 1] - calib[[target]], calib[[target]] - pr[, 2])
    rank <- ceiling((length(scores) + 1) * coverage)
    margin <- if (rank > length(scores)) Inf else max(0, sort(scores)[rank])
    key <- as.character(round(coverage * 100))
    margins[[key]] <- margin
    test <- forest_quantiles(fit, audit, cols, c(alpha / 2, 1 - alpha / 2))
    checks[[key]] <- data.frame(ticker = unique(d$ticker), horizon = horizon,
      nominal_coverage = coverage, train_rows = nrow(train), calibration_rows = nrow(calib),
      audit_rows = nrow(audit), margin = if (is.finite(margin)) margin else NA_real_,
      available = is.finite(margin),
      raw_audit_coverage = mean(audit[[target]] >= test[, 1] & audit[[target]] <= test[, 2]),
      calibrated_audit_coverage = if (is.finite(margin))
        mean(audit[[target]] >= test[, 1] - margin & audit[[target]] <= test[, 2] + margin) else NA_real_,
      training_target_end = as.character(max(train[[end]])), calibration_start = as.character(cal_start),
      calibration_target_end = as.character(max(calib[[end]])), audit_start = as.character(audit_start))
  }
  list(fit = fit, margins = margins, cols = cols, checks = do.call(rbind, checks))
}

generate_prediction_conformal <- function(df, horizon, cal, bt_summary, features) {
  ticker <- unique(df$ticker)
  latest <- latest_observation(df, ticker, cal$cols)
  row <- bt_summary[bt_summary$ticker == ticker & bt_summary$horizon == horizon, ]
  prediction_row(cal$fit, latest, horizon, cal$cols, row, features,
                 paste0(MODEL_VERSION, "_conformal"), cal)
}

run_conformal <- function(df, horizons, bt_summary, features, cols = feature_columns()) {
  predictions <- checks <- list()
  for (h in horizons) {
    cal <- conformal_calibrate(df, h, cols)
    predictions[[as.character(h)]] <- generate_prediction_conformal(df, h, cal, bt_summary, features)
    checks[[as.character(h)]] <- cal$checks
  }
  list(predictions = do.call(rbind, predictions), checks = do.call(rbind, checks))
}

9 Earnings comparison and orchestration

A single runner replaces the repeated MU, Tesla, and Palantir scripts. The named convenience functions remain available. Earnings comparisons use the same rows, fold boundaries, and horizons for both feature sets. Model failures stop the run; optional capabilities are disabled only through visible parameters.

compare_earnings_value <- function(df, ticker = "MU", horizon = 5) {
  ext <- add_earnings_features(df, ticker, horizon)
  common <- finite_rows(ext, feature_columns_with_earnings())
  ext <- ext[common, ]
  base <- walk_forward_backtest(ext, horizon, cols = feature_columns())
  extra <- walk_forward_backtest(ext, horizon, cols = feature_columns_with_earnings())
  a <- summarize_backtest(base); b <- summarize_backtest(extra)
  a$feature_set <- "Base"; b$feature_set <- "Base + earnings (known information only)"
  rbind(a, b)
}

run_stock <- function(ticker, features, horizons = cfg$horizons, earnings = cfg$use_earnings) {
  df <- build_model_dataset(ticker, horizons, features = features)
  bt <- forecasts <- audits <- calibration <- list()
  for (h in horizons) {
    d <- if (earnings) add_earnings_features(df, ticker, h) else df
    cols <- if (earnings) feature_columns_with_earnings() else feature_columns()
    folds <- walk_forward_backtest(d, h, cols = cols)
    summary <- summarize_backtest(folds)
    bt[[as.character(h)]] <- folds
    audits[[as.character(h)]] <- attr(folds, "fold_audit")
    forecasts[[as.character(h)]] <- generate_prediction(d, h, summary, features, ticker, cols)
    if (earnings) forecasts[[as.character(h)]]$model_version <- paste0(MODEL_VERSION, "_earnings")
    if (cfg$run_calibration) {
      cal <- run_conformal(d, h, summary, features, cols)
      if (earnings) cal$predictions$model_version <- paste0(MODEL_VERSION, "_earnings_conformal")
      forecasts[[paste0(h, "_cal")]] <- cal$predictions
      calibration[[as.character(h)]] <- cal$checks
    }
  }
  detail <- do.call(rbind, bt)
  list(dataset = df, backtest = detail, summary = summarize_backtest(detail),
       predictions = do.call(rbind, forecasts), fold_audit = do.call(rbind, audits),
       calibration = if (length(calibration)) do.call(rbind, calibration) else NULL)
}

run_tesla <- function(features, horizons = cfg$horizons) run_stock("TSLA", features, horizons)
run_palantir <- function(features, long = TRUE) run_stock("PLTR", features, if (long) c(5, 20, 63, 126) else c(5, 20))
run_long_horizon <- function(features) run_stock("MU", features, c(63, 126))

run_pipeline <- function() {
  writeLines("Starting fresh run", file.path(cfg$output_dir, "progress.log"))
  work <- prepare_database(cfg$db_path, cfg$output_dir)
  universe <- unique(c(cfg$tickers, CONTEXT_TICKERS, if (cfg$run_panel) PANEL_TICKERS,
                       if (cfg$compare_earnings) "MU"))
  if (cfg$refresh_prices) download_market_data(universe, cfg$data_from, work)
  if (cfg$refresh_earnings) fetch_earnings_batch(unique(c(cfg$tickers, if (cfg$compare_earnings) "MU")))
  integrity <- validate_prices(work, cfg$data_from, cfg$output_dir)
  features <- calculate_technical_features(work)
  if (!all(universe %in% features$ticker)) stop("Some configured tickers lack prices; enable refresh_prices.")
  stocks <- setNames(lapply(cfg$tickers, function(t) run_stock(t, features)), cfg$tickers)
  combine <- function(field) do.call(rbind, lapply(stocks, `[[`, field))
  forecasts <- combine("predictions")
  details <- combine("backtest")
  summaries <- combine("summary")
  folds <- combine("fold_audit")
  calibration <- combine("calibration")
  panel_summary <- NULL
  if (cfg$run_panel) {
    panel <- build_panel_dataset(c(5, 20), work, features)
    panel_bt <- panel_forecasts <- list()
    for (h in c(5, 20)) {
      detail <- walk_forward_panel(panel, h)
      summary <- summarize_panel(detail)
      panel_bt[[as.character(h)]] <- detail
      panel_forecasts[[as.character(h)]] <- generate_panel_prediction(panel, h, summary, features)
      folds <- rbind(folds, attr(detail, "fold_audit"))
    }
    panel_detail <- do.call(rbind, panel_bt)
    panel_summary <- summarize_panel(panel_detail)
    details <- rbind(details, panel_detail)
    forecasts <- rbind(forecasts, do.call(rbind, panel_forecasts))
  }
  earnings_comparison <- NULL
  if (cfg$compare_earnings) {
    mu <- build_model_dataset("MU", c(5, 20), features = features)
    earnings_comparison <- do.call(rbind, lapply(c(5, 20), function(h) compare_earnings_value(mu, "MU", h)))
  }
  save_predictions(forecasts, work)
  write.csv(forecasts, file.path(cfg$output_dir, "forecasts.csv"), row.names = FALSE)
  write.csv(details, file.path(cfg$output_dir, "backtest_by_year.csv"), row.names = FALSE)
  write.csv(summaries, file.path(cfg$output_dir, "backtest_summary.csv"), row.names = FALSE)
  write.csv(folds, file.path(cfg$output_dir, "fold_leakage_audit.csv"), row.names = FALSE)
  if (!is.null(calibration)) write.csv(calibration, file.path(cfg$output_dir, "calibration_audit.csv"), row.names = FALSE)
  if (!is.null(panel_summary)) write.csv(panel_summary, file.path(cfg$output_dir, "panel_summary.csv"), row.names = FALSE)
  if (!is.null(earnings_comparison)) write.csv(earnings_comparison, file.path(cfg$output_dir, "earnings_comparison.csv"), row.names = FALSE)
  for (t in names(stocks)) {
    saveRDS(stocks[[t]]$summary, file.path(cfg$output_dir, paste0("bt_summary_", tolower(t), ".rds")))
    if (t == "MU") saveRDS(stocks[[t]]$summary[stocks[[t]]$summary$horizon %in% c(63, 126), ],
                           file.path(cfg$output_dir, "bt_summary_long.rds"))
  }
  ranges <- do.call(rbind, lapply(split(features, features$ticker), function(d)
    data.frame(ticker = d$ticker[1], rows = nrow(d), first = as.character(min(d$date)),
               last = as.character(max(d$date)), age_calendar_days = as.integer(Sys.Date() - max(d$date)))))
  out <- list(config = cfg, integrity = integrity, ranges = ranges, forecasts = forecasts,
    summary = summaries, detail = details, fold_audit = folds, calibration = calibration,
    panel_summary = panel_summary, earnings_comparison = earnings_comparison,
    database = work, completed_at = format(Sys.time(), tz = "UTC", usetz = TRUE))
  saveRDS(out, file.path(cfg$output_dir, "run_results.rds"))
  writeLines(capture.output(sessionInfo()), file.path(cfg$output_dir, "session_info.txt"))
  jsonlite::write_json(cfg, file.path(cfg$output_dir, "run_config.json"), auto_unbox = TRUE, pretty = TRUE)
  out
}

10 Execute the complete workflow

This is the only top-level call that performs downloads, rebuilds, training, and result writes. The sections above define functions without running models.

results <- readRDS("C:/Users/zxfj0/Desktop/project MU prediction/results/run_results.rds"); cfg <- unclass(as.list(results$config))

11 Results

All results below are generated by the run above. The model observation dates are shown explicitly. An offline run using July data is a July-data forecast, not a current-market forecast. Fresh rendering does not make stale inputs current.

Completed: 2026-09-17 14:48:55 UTC

Input validation: 71104 price rows retained; 48 quarantined.

Data are not live: check the last-observation dates below. Prices were loaded from the supplied snapshot.

Price-history coverage after validation
ticker rows first last age_calendar_days
ADI ADI 2902 2015-01-02 2026-07-20 59
AMAT AMAT 2902 2015-01-02 2026-07-20 59
AMD AMD 2902 2015-01-02 2026-07-20 59
ASML ASML 2902 2015-01-02 2026-07-20 59
AVGO AVGO 2902 2015-01-02 2026-07-20 59
INTC INTC 2902 2015-01-02 2026-07-20 59
KLAC KLAC 2902 2015-01-02 2026-07-20 59
LRCX LRCX 2902 2015-01-02 2026-07-20 59
MCHP MCHP 2902 2015-01-02 2026-07-20 59
MPWR MPWR 2902 2015-01-02 2026-07-20 59
MRVL MRVL 2902 2015-01-02 2026-07-20 59
MU MU 2902 2015-01-02 2026-07-20 59
NVDA NVDA 2902 2015-01-02 2026-07-20 59
NXPI NXPI 2902 2015-01-02 2026-07-20 59
ON ON 2902 2015-01-02 2026-07-20 59
PLTR PLTR 1457 2020-09-30 2026-07-21 58
QCOM QCOM 2902 2015-01-02 2026-07-20 59
QQQ QQQ 2902 2015-01-02 2026-07-20 59
SOXX SOXX 2902 2015-01-02 2026-07-20 59
SPY SPY 2902 2015-01-02 2026-07-20 59
SWKS SWKS 2902 2015-01-02 2026-07-20 59
TER TER 2902 2015-01-02 2026-07-20 59
TSLA TSLA 2901 2015-01-02 2026-07-17 62
TSM TSM 2902 2015-01-02 2026-07-20 59
TXN TXN 2902 2015-01-02 2026-07-20 59

11.1 Forecasts and intervals

Intervals are nominal targets, not promises. Missing calibrated bounds mean there were too few non-overlapping calibration observations for that level. The full CSV includes all quantiles, concentration zones, and model identifiers.

11.1.1 MU

Model As_of Days Current Median Low80 High80 Rise_pct Quality_score
QRF 2026-07-20 5 865.46 888.49 793.46 1003.56 55.3 36
QRF_conformal 2026-07-20 5 865.46 848.45 758.05 999.23 NA 36
QRF 2026-07-20 20 865.46 1172.45 897.22 1392.66 94.5 28
QRF_conformal 2026-07-20 20 865.46 874.84 784.69 1042.16 NA 28
QRF 2026-07-20 63 865.46 2150.19 1375.62 2775.91 99.0 24
QRF_conformal 2026-07-20 63 865.46 940.26 657.68 1381.17 NA 24
QRF 2026-07-20 126 865.46 3602.27 755.44 4224.47 86.9 33
QRF_conformal 2026-07-20 126 865.46 762.02 498.83 1540.27 NA 33
QRF_panel 2026-07-20 5 865.46 834.68 748.85 938.84 29.6 34
QRF_panel 2026-07-20 20 865.46 969.79 662.63 1272.40 66.3 28

11.1.2 TSLA

Model As_of Days Current Median Low80 High80 Rise_pct Quality_score
QRF 2026-07-17 5 380.84 380.91 334.41 419.26 50.3 32
QRF_conformal 2026-07-17 5 380.84 369.46 340.00 405.74 NA 32
QRF 2026-07-17 20 380.84 391.98 342.37 462.05 60.3 3
QRF_conformal 2026-07-17 20 380.84 373.42 311.63 503.35 NA 3
QRF 2026-07-17 63 380.84 525.96 332.99 577.79 78.4 0
QRF_conformal 2026-07-17 63 380.84 434.30 255.62 704.19 NA 0
QRF 2026-07-17 126 380.84 404.33 285.56 560.17 55.3 0
QRF_conformal 2026-07-17 126 380.84 375.90 167.05 1194.09 NA 0

11.1.3 PLTR

Model As_of Days Current Median Low80 High80 Rise_pct Quality_score
QRF 2026-07-20 5 134.85 134.53 125.20 149.45 46.7 44
QRF_conformal 2026-07-20 5 134.85 136.73 119.58 154.91 NA 44
QRF 2026-07-20 20 134.85 128.72 108.62 154.77 39.2 29
QRF_conformal 2026-07-20 20 134.85 134.60 95.81 173.80 NA 29
QRF 2026-07-20 63 134.85 139.38 93.35 246.18 53.3 1
QRF_conformal 2026-07-20 63 134.85 144.56 53.39 451.63 NA 1
QRF 2026-07-20 126 134.85 264.74 91.69 344.28 75.4 49
QRF_conformal 2026-07-20 126 134.85 256.66 NA NA NA 49

11.2 Backtest performance

MAE is measured on log returns. Lower is better. Coverage is the observed fraction inside the interval; compare cov80 with 0.80, not with 1.00.

ticker horizon n_test years mae_model mae_naive_zero mae_naive_mean dir_acc_model dir_acc_naive cov80
MU.1 MU 5 1639 7 0.0625 0.0574 0.0572 0.4558 0.5326 0.7425
MU.2 MU 20 1624 7 0.1243 0.1115 0.1096 0.4704 0.5881 0.7124
MU.3 MU 63 1581 7 0.2469 0.2185 0.2136 0.5497 0.6167 0.6312
MU.4 MU 126 1518 7 0.3705 0.3396 0.3243 0.5995 0.6647 0.5744
TSLA.1 TSLA 5 1638 7 0.0736 0.0687 0.0691 0.4652 0.5153 0.7289
TSLA.2 TSLA 20 1623 7 0.1750 0.1509 0.1522 0.4553 0.5496 0.6124
TSLA.3 TSLA 63 1580 7 0.3028 0.2608 0.2687 0.4544 0.5854 0.5487
TSLA.4 TSLA 126 1517 7 0.4889 0.3796 0.4007 0.4047 0.6249 0.5320
PLTR.1 PLTR 5 634 3 0.0746 0.0649 0.0652 0.5016 0.4732 0.8249
PLTR.2 PLTR 20 619 3 0.1776 0.1368 0.1391 0.4362 0.4394 0.7173
PLTR.3 PLTR 63 576 3 0.4260 0.2818 0.2947 0.4861 0.3924 0.6042
PLTR.4 PLTR 126 513 3 0.5909 0.5676 0.5699 0.6472 0.2807 0.2924

11.3 Panel versus MU specialist

The panel table contains pooled-stock and MU-only evaluation. Compare MU-only panel rows with the MU specialist. Different training availability can produce different sets of test years; use backtest_by_year.csv for matched-year comparisons.

ticker scope horizon n_test mae_model mae_naive_zero dir_acc_model cov80
ALL all 5 32780 0.0481 0.0470 0.5207 0.7497
MU MU 5 1639 0.0584 0.0574 0.5229 0.7041
ALL all 20 32480 0.0980 0.0947 0.5443 0.7243
MU MU 20 1624 0.1168 0.1115 0.5246 0.6780

11.4 Calibration audit

The audit rows were not used to fit the forest or choose margins. These are separate diagnostic coverage estimates, not the raw backtest quality score. For long horizons the audit can be very small; report the count alongside coverage.

ticker horizon nominal_coverage calibration_rows audit_rows available raw_audit_coverage calibrated_audit_coverage
MU.5.5.50 MU 5 0.50 110 67 TRUE 0.284 0.284
MU.5.5.80 MU 5 0.80 110 67 TRUE 0.463 0.463
MU.5.5.95 MU 5 0.95 110 67 TRUE 0.701 0.716
MU.20.20.50 MU 20 0.50 31 19 TRUE 0.211 0.211
MU.20.20.80 MU 20 0.80 31 19 TRUE 0.421 0.421
MU.20.20.95 MU 20 0.95 31 19 TRUE 0.526 0.895
MU.63.63.50 MU 63 0.50 10 7 TRUE 0.000 0.143
MU.63.63.80 MU 63 0.80 10 7 TRUE 0.143 0.714
MU.63.63.95 MU 63 0.95 10 7 FALSE 0.429 NA
MU.126.126.50 MU 126 0.50 4 3 TRUE 0.000 0.667
MU.126.126.80 MU 126 0.80 4 3 TRUE 0.333 0.667
MU.126.126.95 MU 126 0.95 4 3 FALSE 0.667 NA
TSLA.5.5.50 TSLA 5 0.50 110 67 TRUE 0.507 0.507
TSLA.5.5.80 TSLA 5 0.80 110 67 TRUE 0.791 0.821
TSLA.5.5.95 TSLA 5 0.95 110 67 TRUE 0.955 0.955
TSLA.20.20.50 TSLA 20 0.50 31 19 TRUE 0.316 0.579
TSLA.20.20.80 TSLA 20 0.80 31 19 TRUE 0.684 0.895
TSLA.20.20.95 TSLA 20 0.95 31 19 TRUE 0.842 1.000
TSLA.63.63.50 TSLA 63 0.50 10 7 TRUE 0.429 0.429
TSLA.63.63.80 TSLA 63 0.80 10 7 TRUE 0.571 0.857
TSLA.63.63.95 TSLA 63 0.95 10 7 FALSE 0.857 NA
TSLA.126.126.50 TSLA 126 0.50 4 3 TRUE 0.333 1.000
TSLA.126.126.80 TSLA 126 0.80 4 3 TRUE 0.667 1.000
TSLA.126.126.95 TSLA 126 0.95 4 3 FALSE 1.000 NA
PLTR.5.5.50 PLTR 5 0.50 50 31 TRUE 0.548 0.677
PLTR.5.5.80 PLTR 5 0.80 50 31 TRUE 0.806 0.839
PLTR.5.5.95 PLTR 5 0.95 50 31 TRUE 0.903 0.968
PLTR.20.20.50 PLTR 20 0.50 14 9 TRUE 0.667 0.889
PLTR.20.20.80 PLTR 20 0.80 14 9 TRUE 0.667 0.889
PLTR.20.20.95 PLTR 20 0.95 14 9 FALSE 0.889 NA
PLTR.63.63.50 PLTR 63 0.50 4 3 TRUE 0.333 0.667
PLTR.63.63.80 PLTR 63 0.80 4 3 TRUE 0.667 1.000
PLTR.63.63.95 PLTR 63 0.95 4 3 FALSE 0.667 NA
PLTR.126.126.50 PLTR 126 0.50 2 2 TRUE 0.000 1.000
PLTR.126.126.80 PLTR 126 0.80 2 2 FALSE 0.500 NA
PLTR.126.126.95 PLTR 126 0.95 2 2 FALSE 1.000 NA

11.5 Does earnings timing help?

Both feature sets use the same comparison rows and walk-forward method. An improvement here is historical evidence, not a guarantee of future improvement.

12 Outputs, reproducibility, and change log

The run writes the following into the configured output directory:

File Purpose
working_stock_model.sqlite Validated prices, rebuilt features, preserved legacy predictions, new prediction table
forecasts.csv All specialist, panel, and calibrated forecasts
backtest_by_year.csv Annual test metrics
backtest_summary.csv Observation-weighted specialist metrics
panel_summary.csv Panel and MU-only metrics, when enabled
calibration_audit.csv Calibration counts, margins, and independent audit coverage
earnings_comparison.csv Matched base/earnings comparisons
fold_leakage_audit.csv Training-label end dates versus first test dates
quarantined_prices.csv Excluded input rows with reasons
run_results.rds Complete report results and configuration
bt_summary_*.rds Fresh summaries for individual stocks and MU long horizons
run_config.json, session_info.txt Settings and runtime/package versions

12.1 Consolidation record

  • All nine (1).R uploads were exact duplicates and contributed no new implementation.
  • Main Execution Code Pile.R, Rdata_market.R, and market data.R became one setup/download path, with upserts and portable paths.
  • prediction.R, Final prediction.R, Prediction Test Only.R, and MU Prediction Fixed Model.R became one shared forecast implementation.
  • backtest.R, Calibration and Leakage Patch.R, and long-horizon evaluation became one date-purged backtester, with a panel wrapper.
  • Panel.R and P5 Cross Section panel.R became one panel workflow. Duplicate definitions and recursive self-sourcing were removed.
  • Palantir Prediction.R, Tesla Price Model Variant All In One.R, 3 months and 6 months prediction.R, and run pipeline.R became configurable runners, without assignments into the global workspace.
  • R features technical1.R and build dataset.R became one coherent adjusted-price feature/target pipeline. MACD normalization and price-basis handling were corrected.
  • MU Earning.R and fetch earnings.R retain downloading, inspection, templates, event features, and feature comparisons. Missing declarations and recursive feature selection were repaired. Templates are empty rather than fabricated dates.
  • Confidence Level Boost Patch.R became calibration plus an independent audit. Insufficient calibration data now yields explicit unavailable bounds.
  • Quality scores retain their original formula; misleading mode/expected-return labels were corrected. Probability approximations now use a uniform quantile grid.
  • Date estimates now use a session calendar instead of multiplying horizons by 1.45.
  • Regime detection uses the requested stock’s volatility and observation date.
  • New predictions use a separate versioned table; schema migration never drops the original prediction history.
  • These correctness changes can alter numerical results. Old saved RDS summaries cannot validate the corrected models and are retained only as legacy artifacts.

12.2 Validation record

When delivered, see VALIDATION.md for the checks actually performed, their outcomes, and the exact run settings. Runtime details below are captured during this render; they do not imply live data downloads were tested.

## R version 4.6.1 (2026-06-24 ucrt)
## Platform: x86_64-w64-mingw32/x64
## Running under: Windows 11 x64 (build 26200)
## 
## Matrix products: default
##   LAPACK version 3.12.1
## 
## locale:
## [1] LC_COLLATE=Chinese (Simplified)_China.utf8  LC_CTYPE=Chinese (Simplified)_China.utf8   
## [3] LC_MONETARY=Chinese (Simplified)_China.utf8 LC_NUMERIC=C                               
## [5] LC_TIME=Chinese (Simplified)_China.utf8    
## 
## time zone: America/Buenos_Aires
## tzcode source: internal
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## loaded via a namespace (and not attached):
##  [1] Matrix_1.7-5      bit_4.6.0         jsonlite_2.0.0    dplyr_1.2.1       compiler_4.6.1   
##  [6] ranger_0.18.0     Rcpp_1.1.2        tidyselect_1.2.1  blob_1.3.0        jquerylib_0.1.4  
## [11] yaml_2.3.12       fastmap_1.2.0     lattice_0.22-9    R6_2.6.1          generics_0.1.4   
## [16] curl_7.1.0        knitr_1.51        tibble_3.3.1      DBI_1.3.0         bslib_0.11.0     
## [21] pillar_1.11.1     rlang_1.3.0       quantmod_0.4.29   cachem_1.1.0      xfun_0.60        
## [26] sass_0.4.10       bit64_4.8.2       otel_0.2.0        RSQLite_3.53.3    memoise_2.0.1    
## [31] cli_3.6.6         magrittr_2.0.5    xts_0.14.2        digest_0.6.39     grid_4.6.1       
## [36] rstudioapi_0.19.0 lifecycle_1.0.5   vctrs_0.7.3       evaluate_1.0.5    glue_1.8.1       
## [41] zoo_1.8-15        rsconnect_1.11.0  rmarkdown_2.31    TTR_0.24.4        tools_4.6.1      
## [46] pkgconfig_2.0.3   htmltools_0.5.9