Homework 8: Applied Predictive Modeling Exercises 7.2 and 7.5

RPubs URL: https://rpubs.com/umaisabdullah/Data624_Homework8

Objective

This homework compares nonlinear regression methods on two problems. Exercise 7.2 is a simulation with a known nonlinear signal, so it lets me check whether the models recover the informative predictors and how well they predict new data. Exercise 7.5 returns to the chemical manufacturing data from Homework 7 and asks whether nonlinear models improve prediction or reveal a different view of the biological and process measurements.

I use cross validation to tune models and reserve test data for a final performance check. For the chemical data I use the same split seed, training-only KNN imputation, and centering/scaling approach as in Homework 7.

Exercise 7.2: Friedman Simulation

Friedman’s simulated response is

\[ y = 10\sin(\pi x_1x_2) + 20(x_3 - 0.5)^2 + 10x_4 + 5x_5 + \epsilon, \qquad \epsilon \sim N(0, 1). \]

All ten predictors are independent uniform random variables on \([0, 1]\), but only X1 through X5 contribute to the response. This known structure makes it possible to assess both predictive performance and variable selection.

Simulate Data

set.seed(200)
trainingData <- mlbench.friedman1(200, sd = 1)
trainingData$x <- as.data.frame(trainingData$x)
names(trainingData$x) <- paste0("X", seq_len(ncol(trainingData$x)))
training_y <- as.numeric(trainingData$y)

testData <- mlbench.friedman1(5000, sd = 1)
testData$x <- as.data.frame(testData$x)
names(testData$x) <- names(trainingData$x)
test_y <- as.numeric(testData$y)

c(training_rows = nrow(trainingData$x),
  test_rows = nrow(testData$x),
  predictors = ncol(trainingData$x))

## training_rows     test_rows    predictors 
##           200          5000            10

featurePlot(x = trainingData$x, y = training_y, plot = "pairs")

The first five predictors carry the signal; the other five are noise variables. I compare K-nearest neighbors, MARS, random forest, and a radial-basis support vector machine. The models use the same 10-fold cross-validation folds, and the large simulated test set gives a precise out-of-sample comparison.

set.seed(2000)
friedman_folds <- createFolds(training_y, k = 10, returnTrain = TRUE)
ctrl <- trainControl(method = "cv", number = 10, index = friedman_folds)

fit_friedman <- function(method, seed, ...) {
  set.seed(seed)
  train(
    x = trainingData$x, y = training_y,
    method = method,
    trControl = ctrl,
    preProcess = c("center", "scale"),
    ...
  )
}

friedman_models <- list(
  KNN = fit_friedman("knn", 2001, tuneLength = 10),
  MARS = fit_friedman("earth", 2002, tuneLength = 10),
  RandomForest = fit_friedman("rf", 2003, tuneLength = 5),
  SVM_Radial = fit_friedman("svmRadial", 2004, tuneLength = 10)
)

friedman_cv <- bind_rows(lapply(names(friedman_models), function(model_name) {
  model <- friedman_models[[model_name]]
  best <- model$results[which.min(model$results$RMSE), ]
  data.frame(Model = model_name, CV_RMSE = best$RMSE,
             CV_Rsquared = best$Rsquared)
}))

friedman_test <- bind_rows(lapply(names(friedman_models), function(model_name) {
  model <- friedman_models[[model_name]]
  performance <- postResample(predict(model, newdata = testData$x), test_y)
  data.frame(Model = model_name, Test_RMSE = performance[["RMSE"]],
             Test_Rsquared = performance[["Rsquared"]])
}))

friedman_comparison <- left_join(friedman_cv, friedman_test, by = "Model") |>
  mutate(across(where(is.numeric), ~ round(.x, 4))) |>
  arrange(CV_RMSE)

friedman_comparison

##          Model CV_RMSE CV_Rsquared Test_RMSE Test_Rsquared
## 1         MARS  1.5436      0.9028    1.8136        0.8677
## 2   SVM_Radial  1.8170      0.8736    2.0667        0.8268
## 3 RandomForest  2.3836      0.7962    2.3999        0.7898
## 4          KNN  3.0276      0.7041    3.1482        0.6748

The smallest cross-validated RMSE selects the tuning configuration during training. The test columns are a separate check on new observations; I do not use the test set to retune the models.

friedman_best_cv <- friedman_comparison$Model[which.min(friedman_comparison$CV_RMSE)]
friedman_best_test <- friedman_comparison$Model[which.min(friedman_comparison$Test_RMSE)]

friedman_comparison |>
  filter(Model %in% c(friedman_best_cv, friedman_best_test))

##   Model CV_RMSE CV_Rsquared Test_RMSE Test_Rsquared
## 1  MARS  1.5436      0.9028    1.8136        0.8677

Which models appear to give the best performance? MARS gives the best result on both cross-validation and the 5,000-row test set. Its CV RMSE is 1.5436 (R2 0.9028); test RMSE is 1.8136 (R2 0.8677). The ranking is MARS, radial SVM, random forest, then KNN on both measures. MARS’s flexible hinge terms capture this signal efficiently, while KNN is hurt most by the five noise predictors in its distance calculations.

Does MARS Select the Informative Predictors?

I use MARS variable importance and compare the selected variables with the known signal variables, X1 through X5. The importance values rank variables used by the fitted MARS model; a zero value means the predictor did not contribute to the fitted model.

mars_model <- friedman_models$MARS
mars_importance <- varImp(mars_model, scale = FALSE)$importance
mars_importance$Predictor <- rownames(mars_importance)
mars_importance$Overall <- mars_importance[, setdiff(names(mars_importance), "Predictor")[1]]
mars_importance <- mars_importance |>
  select(Predictor, Overall) |>
  arrange(desc(Overall))

mars_importance

##    Predictor   Overall
## X1        X1 100.00000
## X4        X4  84.21578
## X2        X2  67.21639
## X5        X5  45.44416
## X3        X3  34.63259
## X6        X6  11.90397

mars_selected <- mars_importance$Predictor[mars_importance$Overall > 0]
informative <- paste0("X", 1:5)
mars_signal_check <- data.frame(
  Predictor = informative,
  Selected_by_MARS = informative %in% mars_selected
)
mars_signal_selected <- intersect(mars_selected, informative)
mars_noise_selected <- setdiff(mars_selected, informative)
mars_noise_importance <- mars_importance$Overall[
  mars_importance$Predictor %in% mars_noise_selected
]

mars_signal_check

##   Predictor Selected_by_MARS
## 1        X1             TRUE
## 2        X2             TRUE
## 3        X3             TRUE
## 4        X4             TRUE
## 5        X5             TRUE

Does MARS select the informative predictors? Yes. MARS gives nonzero importance to all five true predictors (X1, X4, X2, X5, X3). It also selects X6, a noise predictor; the other four noise variables are excluded. Its importance (11.9) is below that of every true signal predictor, so selection is highly informative but not perfect for this 200-row sample.

Exercise 7.5: Chemical Manufacturing Process

The chemical manufacturing data contain 176 production runs, 57 predictors, and percent yield as the outcome. Twelve predictors describe the biological input material and 45 describe manufacturing settings. Unlike the Friedman simulation, the useful relationships are not known in advance.

Load and Inspect the Data

data(ChemicalManufacturingProcess)
cmp <- ChemicalManufacturingProcess

predictors_raw <- cmp[, -1, drop = FALSE]
yield <- as.numeric(cmp[, 1])

c(runs = nrow(cmp), predictors = ncol(predictors_raw),
  biological = sum(grepl("^Biological", names(predictors_raw))),
  process = sum(grepl("^Manufacturing", names(predictors_raw))),
  missing_cells = sum(is.na(predictors_raw)))

##          runs    predictors    biological       process missing_cells 
##           176            57            12            45           106

Impute, Split, and Pre-process

I preserve the Homework 7 procedure. The data are split before imputation, and the KNN imputation model is fitted only on the training rows. Thus, test observations do not contribute to the imputation statistics or neighbors. Near-zero-variance filtering is also decided using training rows only. Model centering and scaling happen within caret’s resampling, preventing information from validation folds from leaking into their training folds.

set.seed(624)
chem_idx <- createDataPartition(yield, p = 0.75, list = FALSE)

chem_imputer <- preProcess(predictors_raw[chem_idx, , drop = FALSE],
                           method = "knnImpute")
predictors_imputed <- predict(chem_imputer, predictors_raw)

nzv_chem <- nearZeroVar(predictors_imputed[chem_idx, , drop = FALSE])
if (length(nzv_chem) > 0) {
  predictors_imputed <- predictors_imputed[, -nzv_chem, drop = FALSE]
}

chem_X_train <- predictors_imputed[chem_idx, , drop = FALSE]
chem_X_test <- predictors_imputed[-chem_idx, , drop = FALSE]
chem_y_train <- yield[chem_idx]
chem_y_test <- yield[-chem_idx]

c(train_rows = nrow(chem_X_train), test_rows = nrow(chem_X_test),
  predictors_after_filtering = ncol(chem_X_train),
  missing_after_imputation = sum(is.na(predictors_imputed)))

##                 train_rows                  test_rows 
##                        132                         44 
## predictors_after_filtering   missing_after_imputation 
##                         56                          0

Tune Linear and Nonlinear Models

The question asks for nonlinear regressions. I tune KNN, MARS, random forest, and radial SVM, and also fit the PLS and elastic-net linear-model approaches from Homework 7. All six models use the same split and 10-fold cross-validation. Comparing the linear and nonlinear models this way makes the predictor-importance comparison meaningful.

set.seed(6240)
chem_folds <- createFolds(chem_y_train, k = 10, returnTrain = TRUE)
chem_ctrl <- trainControl(method = "cv", number = 10, index = chem_folds)

fit_chemical <- function(method, seed, ...) {
  set.seed(seed)
  train(
    x = chem_X_train, y = chem_y_train,
    method = method,
    trControl = chem_ctrl,
    preProcess = c("center", "scale"),
    ...
  )
}

chem_models <- list(
  PLS = fit_chemical("pls", 625, tuneLength = 15),
  ElasticNet = fit_chemical(
    "enet", 626,
    tuneGrid = expand.grid(lambda = c(0.001, 0.01, 0.1),
                           fraction = seq(0.05, 1, length = 10))
  ),
  KNN = fit_chemical("knn", 627, tuneLength = 10),
  MARS = fit_chemical("earth", 628, tuneLength = 10),
  RandomForest = fit_chemical("rf", 629, tuneLength = 5),
  SVM_Radial = fit_chemical("svmRadial", 630, tuneLength = 10)
)

chemical_cv <- bind_rows(lapply(names(chem_models), function(model_name) {
  model <- chem_models[[model_name]]
  best <- model$results[which.min(model$results$RMSE), ]
  data.frame(Model = model_name, CV_RMSE = best$RMSE,
             CV_Rsquared = best$Rsquared)
}))

chemical_test <- bind_rows(lapply(names(chem_models), function(model_name) {
  model <- chem_models[[model_name]]
  performance <- postResample(predict(model, newdata = chem_X_test), chem_y_test)
  data.frame(Model = model_name, Test_RMSE = performance[["RMSE"]],
             Test_Rsquared = performance[["Rsquared"]])
}))

chemical_comparison <- left_join(chemical_cv, chemical_test, by = "Model") |>
  mutate(across(where(is.numeric), ~ round(.x, 4))) |>
  arrange(CV_RMSE)

chemical_comparison

##          Model CV_RMSE CV_Rsquared Test_RMSE Test_Rsquared
## 1 RandomForest  1.0521      0.6932    1.4113        0.5657
## 2   ElasticNet  1.0662      0.6602    1.3191        0.5957
## 3   SVM_Radial  1.1001      0.6407    1.2837        0.6346
## 4         MARS  1.1318      0.6436    1.4700        0.5144
## 5          PLS  1.1544      0.6074    1.6698        0.4582
## 6          KNN  1.2896      0.5100    1.5570        0.4615

linear_names <- c("PLS", "ElasticNet")
nonlinear_names <- c("KNN", "MARS", "RandomForest", "SVM_Radial")

best_linear_name <- chemical_comparison |>
  filter(Model %in% linear_names) |>
  slice_min(CV_RMSE, n = 1, with_ties = FALSE) |>
  pull(Model)

best_nonlinear_name <- chemical_comparison |>
  filter(Model %in% nonlinear_names) |>
  slice_min(CV_RMSE, n = 1, with_ties = FALSE) |>
  pull(Model)

best_test_name <- chemical_comparison |>
  filter(Model %in% nonlinear_names) |>
  slice_min(Test_RMSE, n = 1, with_ties = FALSE) |>
  pull(Model)

best_test_overall_name <- chemical_comparison |>
  slice_min(Test_RMSE, n = 1, with_ties = FALSE) |>
  pull(Model)
best_test_overall_label <- if (best_test_overall_name == "SVM_Radial") {
  "radial SVM"
} else {
  best_test_overall_name
}
chem_test_best <- filter(chemical_comparison, Model == best_test_overall_name)
chem_top3_cv <- chemical_comparison |>
  slice_min(CV_RMSE, n = 3, with_ties = FALSE)
chem_top3_cv_span <- diff(range(chem_top3_cv$CV_RMSE))
chem_rf <- filter(chemical_comparison, Model == "RandomForest")
chem_enet <- filter(chemical_comparison, Model == "ElasticNet")
chem_svm <- filter(chemical_comparison, Model == "SVM_Radial")

chemical_comparison |>
  filter(Model %in% c(best_linear_name, best_nonlinear_name, best_test_name))

##          Model CV_RMSE CV_Rsquared Test_RMSE Test_Rsquared
## 1 RandomForest  1.0521      0.6932    1.4113        0.5657
## 2   ElasticNet  1.0662      0.6602    1.3191        0.5957
## 3   SVM_Radial  1.1001      0.6407    1.2837        0.6346

(a) Which nonlinear model is optimal? Random Forest has the lowest CV RMSE (1.0521, R2 0.6932), narrowly ahead of Elastic Net (1.0662) and radial SVM (1.1001). The three leading CV RMSEs span only 0.048 yield points, so their small ordering differences should not be overinterpreted. On the test set, radial SVM is best overall, with RMSE 1.2837 and R2 0.6346. Elastic Net also beats Random Forest on test RMSE (1.3191 versus 1.4113). Thus CV selects Random Forest, but the test-set ordering favors SVM and Elastic Net; the small sample does not support a confident claim that a nonlinear model is materially better.

Predictor Importance

I compare the ten highest-ranked predictors in the CV-selected nonlinear model with the ten highest-ranked predictors in the CV-selected linear model. For elastic net, importance is the absolute value of the fitted standardized coefficient. For other models, I use caret::varImp() and rank its model-specific or filter-based importance scores. The ranks are useful for comparing membership, but importance scales are not directly comparable between model families.

importance_vector <- function(model, model_name) {
  if (model_name == "ElasticNet") {
    coefficients <- predict(
      model$finalModel,
      s = model$bestTune$fraction,
      type = "coefficients", mode = "fraction"
    )$coefficients
    result <- abs(coefficients[names(coefficients) != "(Intercept)"])
  } else {
    importance <- varImp(model, scale = FALSE)$importance
    result <- importance[[1]]
    names(result) <- rownames(importance)
  }

  result <- result[is.finite(result)]
  sort(result, decreasing = TRUE)
}

linear_importance <- importance_vector(chem_models[[best_linear_name]],
                                       best_linear_name)
nonlinear_importance <- importance_vector(chem_models[[best_nonlinear_name]],
                                          best_nonlinear_name)

top_linear <- head(names(linear_importance), 10)
top_nonlinear <- head(names(nonlinear_importance), 10)

importance_comparison <- full_join(
  data.frame(Predictor = top_linear, Linear_Rank = seq_along(top_linear)),
  data.frame(Predictor = top_nonlinear,
             Nonlinear_Rank = seq_along(top_nonlinear)),
  by = "Predictor"
) |>
  arrange(coalesce(Linear_Rank, Inf), coalesce(Nonlinear_Rank, Inf))

importance_comparison

##                 Predictor Linear_Rank Nonlinear_Rank
## 1  ManufacturingProcess32           1              1
## 2  ManufacturingProcess09           2             10
## 3    BiologicalMaterial06           3              4
## 4  ManufacturingProcess17           4             NA
## 5  ManufacturingProcess06           5             NA
## 6  ManufacturingProcess34           6             NA
## 7  ManufacturingProcess01           7             NA
## 8  ManufacturingProcess39           8             NA
## 9  ManufacturingProcess37           9             NA
## 10 ManufacturingProcess45          10             NA
## 11 ManufacturingProcess31          NA              2
## 12   BiologicalMaterial12          NA              3
## 13   BiologicalMaterial03          NA              5
## 14 ManufacturingProcess13          NA              6
## 15 ManufacturingProcess11          NA              7
## 16   BiologicalMaterial11          NA              8
## 17   BiologicalMaterial04          NA              9

top_nonlinear_types <- data.frame(Predictor = top_nonlinear) |>
  mutate(Type = if_else(grepl("^Biological", Predictor),
                        "Biological", "Process"))

top_linear_biological <- sum(grepl("^Biological", top_linear))
top_nonlinear_biological <- sum(grepl("^Biological", top_nonlinear))
shared_top_predictors <- intersect(top_linear, top_nonlinear)

top_nonlinear_types |>
  count(Type, name = "Predictors")

##         Type Predictors
## 1 Biological          5
## 2    Process          5

(b) Which predictors matter, and does either group dominate? ManufacturingProcess32 (MP32) ranks first in both the Random Forest and Elastic Net top-ten lists. The lists share 3 predictors (ManufacturingProcess32, ManufacturingProcess09, BiologicalMaterial06). Random Forest’s top ten split evenly: 5 biological and 5 process predictors. Elastic Net’s list contains 1 biological and 9 process predictors. Thus, the nonlinear model gives more top-ten weight to biological measurements, while the linear model’s list is dominated by process variables. Process settings are actionable; biological measurements can help screen incoming raw material.

Relationships for Nonlinear-Only Predictors

The plots below focus on predictors that appear in the nonlinear model’s top ten but not in the linear model’s top ten. A loess curve is included to make curvature or changing slopes visible. These are exploratory marginal relationships: the plots do not hold other predictors constant and cannot establish that changing a process setting will cause yield to change.

The plots and marginal-statistics table use all 176 runs for descriptive display only. They are not used to fit or tune the predictive models.

nonlinear_only <- setdiff(top_nonlinear, top_linear)

if (length(nonlinear_only) == 0) {
  cat("All top-ten nonlinear predictors also appear in the linear model's top ten.")
} else {
  predictors_imputed |>
    select(all_of(nonlinear_only)) |>
    mutate(Yield = yield) |>
    pivot_longer(-Yield, names_to = "Predictor", values_to = "Value") |>
    ggplot(aes(x = Value, y = Yield)) +
    geom_point(alpha = 0.6, colour = "steelblue") +
    geom_smooth(method = "loess", se = FALSE, colour = "darkred") +
    facet_wrap(~ Predictor, scales = "free_x", ncol = 3) +
    labs(title = "Yield relationships for nonlinear-only top predictors",
         x = "Imputed, scaled predictor value", y = "Yield (percent)")
}

data.frame(Predictor = nonlinear_only) |>
  mutate(Type = if_else(grepl("^Biological", Predictor),
                        "Biological", "Process"))

##                Predictor       Type
## 1 ManufacturingProcess31    Process
## 2   BiologicalMaterial12 Biological
## 3   BiologicalMaterial03 Biological
## 4 ManufacturingProcess13    Process
## 5 ManufacturingProcess11    Process
## 6   BiologicalMaterial11 Biological
## 7   BiologicalMaterial04 Biological

nonlinear_plot_stats <- lapply(nonlinear_only, function(predictor) {
  value <- predictors_imputed[[predictor]]
  linear_fit <- lm(yield ~ value)
  quadratic_fit <- lm(yield ~ value + I(value^2))

  data.frame(
    Predictor = predictor,
    Pearson = cor(value, yield, method = "pearson"),
    Spearman = cor(value, yield, method = "spearman"),
    Linear_R2 = summary(linear_fit)$r.squared,
    Quadratic_R2 = summary(quadratic_fit)$r.squared,
    Delta_R2 = summary(quadratic_fit)$r.squared - summary(linear_fit)$r.squared,
    Minimum = min(value),
    Cor_with_BiologicalMaterial06 = if (grepl("^Biological", predictor)) {
      cor(value, predictors_imputed$BiologicalMaterial06)
    } else {
      NA_real_
    }
  )
}) |>
  bind_rows()

knitr::kable(
  nonlinear_plot_stats |>
    mutate(Predictor = sub("^ManufacturingProcess", "MP", Predictor),
           Predictor = sub("^BiologicalMaterial", "BM", Predictor)) |>
    mutate(across(where(is.numeric), ~ round(.x, 3))),
  col.names = c("Predictor", "Pearson r", "Spearman rho", "Linear R2",
                "Quadratic R2", "Delta R2", "Min", "r vs BM06"),
  caption = paste("MP = ManufacturingProcess; BM = BiologicalMaterial.",
                  "Marginal statistics for the plotted predictors (all 176 runs).")
)

MP = ManufacturingProcess; BM = BiologicalMaterial. Marginal statistics for the plotted predictors (all 176 runs).

Predictor	Pearson r	Spearman rho	Linear R2	Quadratic R2	Delta R2	Min	r vs BM06
MP31	-0.071	-0.396	0.005	0.145	0.140	-10.949	NA
BM12	0.367	0.425	0.135	0.180	0.045	-2.448	0.813
BM03	0.445	0.491	0.198	0.216	0.018	-2.930	0.872
MP13	-0.504	-0.381	0.254	0.267	0.013	-2.567	NA
MP11	0.347	0.303	0.120	0.121	0.000	-2.601	NA
BM11	0.355	0.373	0.126	0.141	0.015	-2.358	0.776
BM04	0.380	0.441	0.144	0.177	0.033	-1.662	0.651

bio_plot_stats <- nonlinear_plot_stats |>
  filter(grepl("^Biological", Predictor))
bio_pearson_range <- range(bio_plot_stats$Pearson)
bio_max_delta_r2 <- max(bio_plot_stats$Delta_R2)
bio_mp06_text <- paste(paste0(
  sub("^BiologicalMaterial", "BM", bio_plot_stats$Predictor), " = ",
  round(bio_plot_stats$Cor_with_BiologicalMaterial06, 2)
), collapse = ", ")
mp31_stats <- filter(nonlinear_plot_stats, Predictor == "ManufacturingProcess31")
other_max_delta_r2 <- nonlinear_plot_stats |>
  filter(Predictor != "ManufacturingProcess31") |>
  pull(Delta_R2) |>
  max()
mp13_stats <- filter(nonlinear_plot_stats, Predictor == "ManufacturingProcess13")
mp11_stats <- filter(nonlinear_plot_stats, Predictor == "ManufacturingProcess11")

(c) What do the plots suggest? The four biological predictors have positive marginal correlations with yield, ranging from 0.35 to 0.45. Adding a quadratic term raises marginal R2 by at most 0.045 for these predictors. They also correlate with BM06 (BM12 = 0.81, BM03 = 0.87, BM11 = 0.78, BM04 = 0.65), which helps explain why Elastic Net can represent related biological measurements through BM06 while Random Forest distributes importance across several of them; this is not evidence that each has a distinct nonlinear effect.

MP13 has a negative marginal association with yield (Pearson r = -0.5) and MP11 a weaker positive one (r = 0.35). Their quadratic R2 gains are 0.013 and 0, respectively, so the plots and statistics do not show strong curvature for these two process predictors.

MP31 is the exception. Its minimum scaled value is -10.95; its Pearson and Spearman correlations are -0.071 and -0.396. A linear fit has R2 0.005, compared with 0.145 after adding a quadratic term, a gain of 0.14. That is much larger than the maximum gain of 0.045 among the other plotted predictors. The plot is not monotone: the loess curve rises through the sparse far-left tail and then drops around the dense cluster. The large quadratic gain is therefore a real marginal pattern in these data, but it is sensitive to one extreme observation; I would inspect that run and check model stability with and without it before treating the curvature as a reliable process effect.

These are marginal associations and do not hold other predictors fixed. A repeatable pattern in a process setting could motivate a controlled experiment; biological measurements can help screen or stratify raw-material batches. Neither the plots nor the fitted regressions establish that changing a process setting causes yield to change.

Conclusion. On the Friedman simulation, MARS is best on both CV and test and selects all five true predictors, along with X6; KNN performs worst. On the chemical data, Random Forest has the best CV RMSE (1.0521), but radial SVM has the best test RMSE (1.2837), and Elastic Net also beats Random Forest on the test set. The models’ CV scores are tightly clustered, so the results do not establish a material general advantage for nonlinear regression. MP32 is the strongest shared top predictor, and the nonlinear model’s 5/5 biological/process split differs from Elastic Net’s 1/9. MP31 shows the clearest quadratic pattern, but its extreme observation calls for a sensitivity check. The 132-row training set and 44-row test set make all chemical-process conclusions tentative.