---
title: "Foreign Direct Investment and Renewable Energy Consumption in Sudan"
subtitle: "Econometrics Exercise 7 | Out-of-Sample Forecasting with OLS"
author: "Ahmed Mustafa <br> (& Google Gemini"
date: today
lang: en
format:
html:
toc: true
toc-location: left
toc-depth: 3
number-sections: true
theme: cosmo
embed-resources: true
code-fold: show
code-tools: true
execute:
echo: true
message: false
warning: false
fig-align: center
freeze: auto
---
```{css}
:root {
--brand: #AA272F;
--brand-dark: #7F1D24;
--brand-pale: #F7E9EA;
--brand-blue: #1B4F72;
--brand-green: #477A63;
--ink: #292929;
}
body { color: var(--ink); }
h1, h2, h3 { color: var(--brand-dark); }
h1 { border-bottom: 2px solid var(--brand); padding-bottom: .25rem; }
.callout-note { border-left-color: var(--brand) !important; }
.callout-tip { border-left-color: var(--brand-green) !important; }
.callout-warning { border-left-color: #B7791F !important; }
table caption { color: var(--brand-dark); font-weight: 700; }
table thead th { background: var(--brand-dark) !important; color: #fff !important; }
.table > tbody > tr:nth-of-type(odd) > * { background-color: #FBF5F5; }
```
```{r}
#| label: setup
#| include: false
required_packages <- c("dplyr", "ggplot2", "broom", "knitr", "kableExtra", "tidyr")
missing_packages <- required_packages[
!vapply(required_packages, requireNamespace, logical(1), quietly = TRUE)
]
if (length(missing_packages)) {
stop(
"Install the following packages before rendering: ",
paste(missing_packages, collapse = ", ")
)
}
library(dplyr)
library(ggplot2)
library(broom)
library(knitr)
library(kableExtra)
brand <- "#AA272F"
brand_dark <- "#7F1D24"
brand_blue <- "#1B4F72"
brand_green <- "#477A63"
brand_table <- function(x, caption, digits = 3) {
knitr::kable(
x,
format = "html",
digits = digits,
caption = caption,
row.names = FALSE,
escape = TRUE
) |>
kableExtra::kable_styling(
bootstrap_options = c("striped", "hover", "condensed", "responsive"),
full_width = FALSE,
position = "left"
)
}
fmt_p <- function(p) {
ifelse(is.na(p), "not available",
ifelse(p < 0.001, "< 0.001", sprintf("%.3f", p)))
}
```
# Purpose and learning objectives
The research topic remains **the relationship between foreign direct investment (FDI) and renewable energy consumption in Sudan**. The dependent variable is renewable energy consumption as a share of total energy use (`REN_ERG_CONS`). The main explanatory variable is FDI net inflows as a percentage of GDP (`FDI_GDP`). GDP, industry value added, official development assistance and trade openness are included as control variables.
This exercise applies the forecasting methods to the same Sudan dataset in the following steps:
1. Restore the cleaned data prepared in previous exercise.
2. Split the annual observations chronologically into an estimation sample and a holdout test sample.
3. Re-estimate the multiple regression using only the estimation sample.
4. Produce forecasts for the held-out years and inspect forecast errors.
5. Evaluate accuracy using ME, MAE, MSE, RMSE, MAPE and Theil's measures. Compare the regression with a simple benchmark.
6. Construct classical 95% confidence and prediction intervals.
7. Complete the *Malé cvičenie* tasks.
::: {.callout-note}
## Research focus and interpretation
The coefficient on `FDI_GDP` measures its estimated **conditional association** with renewable energy consumption, holding the other included variables constant. Neither the regression nor its forecasts establish a causal effect of FDI.
:::
::: {.callout-warning}
## Important limitation: the holdout sample was examined before
In a fully independent forecasting exercise, the test period should be set aside before exploring the data, selecting regressors or choosing functional forms. Exercise 5 already examined the full 1995–2021 sample. Therefore, the 2016–2021 holdout period in this exercise is not completely untouched: earlier analysis may have influenced the chosen specification. We will still re-estimate the model using only the estimation years, but the forecast results must be described as **illustrative out-of-sample evaluation**, not as an independent confirmation of future predictive accuracy.
:::
# 1. Restore and verify the Sudan dataset
## 1.1 Load the workspace
In the previous exercise, we saved `sudan_clean` in `sudan_exercise5_workspace.RData` and also in `sudan_exercise5_results.RData`. The code checks these files first. If neither is available, it recreates the cleaned dataset from `LDC-Panel-Data.csv` using the same Sudan filter and complete-case rule.
```{r}
#| label: load-sudan-data
workspace_candidates <- c(
"sudan_exercise5_results.RData",
"sudan_exercise5_workspace.RData"
)
workspace_path <- workspace_candidates[file.exists(workspace_candidates)]
if (length(workspace_path) > 0) {
# Load into a separate environment to avoid replacing unrelated objects.
exercise5_env <- new.env(parent = baseenv())
loaded_objects <- load(workspace_path[1], envir = exercise5_env)
if (!exists("sudan_clean", envir = exercise5_env, inherits = FALSE)) {
stop("The selected Exercise 5 workspace does not contain 'sudan_clean'.")
}
sudan_clean <- get("sudan_clean", envir = exercise5_env)
message("Restored sudan_clean from: ", workspace_path[1])
rm(exercise5_env, loaded_objects)
} else {
csv_path <- if (file.exists("udaje/LDC-Panel-Data.csv")) {
"udaje/LDC-Panel-Data.csv"
} else if (file.exists("LDC-Panel-Data.csv")) {
"LDC-Panel-Data.csv"
} else {
stop(
"Cannot find an Exercise 5 workspace or LDC-Panel-Data.csv. ",
"Place the data file in the working directory or in 'udaje/'."
)
}
raw_data <- read.csv(
csv_path,
na.strings = c("", "NA"),
stringsAsFactors = FALSE
)
required_source_vars <- c(
"Country_Code", "Year", "REN_ERG_CONS", "FDI_GDP", "GDP_USD",
"IND_VA_GDP", "NODA_RCV", "Trade_GDP"
)
missing_source_vars <- setdiff(required_source_vars, names(raw_data))
if (length(missing_source_vars) > 0) {
stop("Missing source columns: ", paste(missing_source_vars, collapse = ", "))
}
analysis_vars <- c(
"REN_ERG_CONS", "FDI_GDP", "GDP_billions",
"IND_VA_GDP", "NODA_RCV", "Trade_GDP"
)
sudan_clean <- raw_data |>
filter(Country_Code == "SDN") |>
mutate(GDP_billions = GDP_USD / 1e9) |>
arrange(Year) |>
filter(if_all(all_of(analysis_vars), ~ !is.na(.x)))
rm(raw_data, csv_path, required_source_vars, missing_source_vars)
}
```
## 1.2 Validate and prepare the analysis data
```{r}
#| label: validate-sudan-data
required_vars <- c(
"Year", "REN_ERG_CONS", "FDI_GDP", "GDP_billions",
"IND_VA_GDP", "NODA_RCV", "Trade_GDP"
)
missing_vars <- setdiff(required_vars, names(sudan_clean))
if (length(missing_vars) > 0) {
stop("The cleaned dataset is missing: ", paste(missing_vars, collapse = ", "))
}
udaje7 <- sudan_clean |>
mutate(Year = as.integer(as.character(Year))) |>
arrange(Year) |>
select(all_of(required_vars))
numeric_vars <- setdiff(required_vars, "Year")
if (!all(vapply(udaje7[numeric_vars], is.numeric, logical(1)))) {
stop("All model variables must be numeric.")
}
if (anyNA(udaje7) || any(!is.finite(as.matrix(udaje7)))) {
stop("The analysis data contain missing or non-finite values.")
}
if (anyDuplicated(udaje7$Year)) {
stop("There is more than one observation for at least one year.")
}
if (any(diff(udaje7$Year) <= 0)) {
stop("The observations must be in strictly increasing year order.")
}
if (any(udaje7$GDP_billions <= 0)) {
warning("GDP_billions contains non-positive values; the log-GDP task may not be available.")
}
sample_summary <- data.frame(
Country = "Sudan (SDN)",
First_year = min(udaje7$Year),
Last_year = max(udaje7$Year),
Observations = nrow(udaje7),
GDP_min_billion_USD = min(udaje7$GDP_billions),
GDP_max_billion_USD = max(udaje7$GDP_billions)
)
brand_table(sample_summary, "Verified Sudan analysis sample", digits = 2)
```
# 2. Chronological sample split and model estimation
## 2.1 Baseline model
The baseline specification follows the multiple regression:
$$
REN_t = \beta_0 + \beta_1 FDI_t + \beta_2 GDP_t + \beta_3 Industry_t + \beta_4 ODA_t + \beta_5 Trade_t + u_t
$$
Here, `REN` is renewable energy consumption, `FDI` is FDI net inflows, `GDP` is GDP in billions of USD, `Industry` is industry value added, `ODA` is net official development assistance received, and `Trade` is trade openness. The coefficient \(\beta_1\) is of particular interest because FDI is the main explanatory variable in this research.
## 2.2 Split the data: first 80% for estimation, final 20% for testing
Because the observations are annual, the split must preserve time order. The earlier observations are used to estimate the model and the most recent observations are held out for our evaluation. With 27 observations, this rule gives 21 estimation years and 6 test years (typically 1995–2015 and 2016–2021). The code calculates the split from the actual number of available observations rather than hard-coding those dates.
```{r}
#| label: chronological-split
n <- nrow(udaje7)
n_train <- floor(0.80 * n)
# The baseline has five regressors plus an intercept.
minimum_training_n <- 7L
if (n_train < minimum_training_n || n_train >= n) {
stop("The sample is too small for the requested 80/20 split and baseline model.")
}
train_data <- udaje7[seq_len(n_train), , drop = FALSE]
test_data <- udaje7[seq.int(n_train + 1L, n), , drop = FALSE]
split_table <- data.frame(
Sample = c("Estimation sample", "Test sample", "Total"),
Period = c(
paste0(min(train_data$Year), "–", max(train_data$Year)),
paste0(min(test_data$Year), "–", max(test_data$Year)),
paste0(min(udaje7$Year), "–", max(udaje7$Year))
),
Observations = c(nrow(train_data), nrow(test_data), n),
Share_percent = round(100 * c(nrow(train_data), nrow(test_data), n) / n, 1)
)
brand_table(split_table, "Chronological 80/20 sample split", digits = 1)
```
::: {.callout-note}
## Why not use a random split?
A random split could place later years in the estimation sample and earlier years in the test sample. That would not reflect the forecasting task, where the model is estimated on past information and evaluated on later observations.
:::
## 2.3 Re-estimate the baseline model using only the estimation sample
Here, we re-estimate the same specification using `train_data` only.
```{r}
#| label: estimate-baseline
model7 <- lm(
REN_ERG_CONS ~ FDI_GDP + GDP_billions + IND_VA_GDP + NODA_RCV + Trade_GDP,
data = train_data
)
if (df.residual(model7) <= 0) {
stop("The estimation sample does not leave positive residual degrees of freedom.")
}
baseline_coefficients <- broom::tidy(model7) |>
mutate(
Significance = case_when(
p.value < 0.001 ~ "***",
p.value < 0.01 ~ "**",
p.value < 0.05 ~ "*",
p.value < 0.10 ~ ".",
TRUE ~ ""
)
) |>
select(
Term = term,
Estimate = estimate,
`Std. error` = std.error,
`t statistic` = statistic,
`p-value` = p.value,
Significance
)
brand_table(
baseline_coefficients,
"Baseline OLS estimates using the estimation sample only",
digits = 4
)
```
The FDI coefficient can be interpreted as follows: a one-percentage-point increase in FDI net inflows as a share of GDP is associated with an estimated \(\hat\beta_1\)-percentage-point change in renewable energy consumption, holding the other regressors constant. This is a conditional association, not a causal effect.
```{r}
#| label: fdi-coefficient-interpretation
fdi_row <- baseline_coefficients |>
filter(Term == "FDI_GDP")
fdi_estimate <- fdi_row$Estimate[1]
fdi_p_value <- fdi_row$`p-value`[1]
fdi_summary <- data.frame(
Variable = "FDI_GDP",
Estimate = fdi_estimate,
`p-value` = fdi_p_value,
`5% decision` = ifelse(fdi_p_value < 0.05, "Reject H0: coefficient is zero", "Do not reject H0")
)
brand_table(fdi_summary, "FDI coefficient in the estimation-period model", digits = 4)
```
# 3. Out-of-sample point forecasts
## 3.1 Generate forecasts and forecast errors
For each held-out year, the model produces a fitted value using the estimated coefficients and that year's observed explanatory variables:
\[
\widehat{REN}_t = \hat\beta_0 + \hat\beta_1 FDI_t + \hat\beta_2 GDP_t
+ \hat\beta_3 Industry_t + \hat\beta_4 ODA_t + \hat\beta_5 Trade_t.
\]
The forecast error is \(e_t = REN_t - \widehat{REN}_t\). A positive error means the model underpredicts renewable energy consumption. A negative error means it overpredicts.
```{r}
#| label: generate-forecasts
test_data$Forecast <- as.numeric(predict(model7, newdata = test_data))
test_data$Forecast_Error <- test_data$REN_ERG_CONS - test_data$Forecast
test_data$Absolute_Error <- abs(test_data$Forecast_Error)
forecast_table <- test_data |>
select(
Year,
Actual_REN = REN_ERG_CONS,
FDI_percent_GDP = FDI_GDP,
GDP_billion_USD = GDP_billions,
Forecast,
Forecast_Error,
Absolute_Error
)
brand_table(
forecast_table,
"Actual and forecast renewable energy consumption in the test period",
digits = 3
)
```
::: {.callout-warning}
## Conditional ex-post forecasts
The test-year values of FDI, GDP, industry, ODA and trade are used as inputs to `predict()`. These values would not normally all be known at the start of the forecast period. The exercise therefore evaluates forecasts **conditional on realised explanatory variables**. It does not evaluate a complete forecasting system that also predicts FDI and the control variables.
:::
## 3.2 Plot actual and forecast values
```{r}
#| label: forecast-trajectory
#| fig-cap: "Actual and OLS-forecast renewable energy consumption in the held-out years."
#| fig-width: 9
#| fig-height: 4.8
forecast_plot_data <- test_data |>
select(Year, Actual = REN_ERG_CONS, Forecast) |>
tidyr::pivot_longer(
cols = c("Actual", "Forecast"),
names_to = "Series",
values_to = "Renewable_energy_consumption"
)
ggplot(forecast_plot_data,
aes(x = Year, y = Renewable_energy_consumption,
colour = Series, linetype = Series, shape = Series)) +
geom_line(linewidth = 0.9) +
geom_point(size = 2.7) +
scale_colour_manual(values = c("Actual" = brand_blue, "Forecast" = brand)) +
scale_linetype_manual(values = c("Actual" = "solid", "Forecast" = "dashed")) +
scale_shape_manual(values = c("Actual" = 16, "Forecast" = 17)) +
scale_x_continuous(breaks = test_data$Year) +
labs(
title = "Actual versus forecast renewable energy consumption",
x = "Year",
y = "Renewable energy consumption (% of total)",
colour = NULL, linetype = NULL, shape = NULL
) +
theme_minimal(base_size = 12) +
theme(
plot.title = element_text(colour = brand_dark, face = "bold"),
legend.position = "bottom"
)
```
```{r}
#| label: actual-versus-forecast
#| fig-cap: "Forecasts plotted against actual values; the 45-degree line represents perfect predictions."
#| fig-width: 6.5
#| fig-height: 5.5
plot_limits <- range(c(test_data$REN_ERG_CONS, test_data$Forecast), finite = TRUE)
plot_span <- diff(plot_limits)
if (plot_span == 0) plot_span <- 1
ggplot(test_data, aes(x = REN_ERG_CONS, y = Forecast)) +
geom_abline(intercept = 0, slope = 1, colour = brand, linewidth = 1) +
geom_point(colour = brand_blue, size = 3) +
geom_text(aes(label = Year), nudge_y = 0.02 * plot_span,
colour = brand_dark, size = 3, check_overlap = TRUE) +
coord_equal(xlim = plot_limits, ylim = plot_limits) +
labs(
title = "Actual versus forecast values",
x = "Actual renewable energy consumption (%)",
y = "Forecast renewable energy consumption (%)"
) +
theme_minimal(base_size = 12) +
theme(plot.title = element_text(colour = brand_dark, face = "bold"))
```
::: {.callout-note}
## Reading the forecast plots
Points close to the 45-degree line indicate more accurate forecasts. In the trajectory plot, compare the size and direction of the gaps between the actual and forecast series. The interpretation should follow the rendered plots and forecast errors. Do not assume in advance that the model overpredicts or underpredicts particular years.
:::
# 4. Forecast accuracy
## 4.1 Accuracy measures
Let \(m\) be the number of test observations and \(e_t = y_t - \hat y_t\) the forecast error.
| Measure | Formula | Interpretation |
|---|---|---|
| ME | \(m^{-1}\sum e_t\) | Average signed error; indicates direction of bias. |
| MAE | \(m^{-1}\sum |e_t|\) | Average absolute error, in percentage points. |
| MSE | \(m^{-1}\sum e_t^2\) | Penalises large errors more heavily. |
| RMSE | \(\sqrt{MSE}\) | Error measure in the dependent variable's units. |
| MAPE | \(100m^{-1}\sum |e_t/y_t|\) | Average absolute percentage error; undefined when an actual value is zero. |
```{r}
#| label: metric-functions
calc_metrics <- function(actual, predicted) {
if (length(actual) != length(predicted)) {
stop("Actual values and forecasts must have the same length.")
}
if (length(actual) == 0 || any(!is.finite(actual)) || any(!is.finite(predicted))) {
stop("Actual values and forecasts must be non-empty and finite.")
}
errors <- actual - predicted
mse_value <- mean(errors^2)
u1_denominator <- sqrt(mean(actual^2)) + sqrt(mean(predicted^2))
mape_value <- if (any(actual == 0)) {
NA_real_
} else {
100 * mean(abs(errors / actual))
}
u1_value <- if (u1_denominator == 0) {
NA_real_
} else {
sqrt(mse_value) / u1_denominator
}
data.frame(
ME = mean(errors),
MAE = mean(abs(errors)),
MSE = mse_value,
RMSE = sqrt(mse_value),
MAPE = mape_value,
Theil_U1 = u1_value
)
}
actual_test <- test_data$REN_ERG_CONS
forecast_baseline <- test_data$Forecast
baseline_metrics <- calc_metrics(actual_test, forecast_baseline)
accuracy_table <- data.frame(
Metric = c("ME", "MAE", "MSE", "RMSE", "MAPE", "Theil's U1"),
Value = unlist(baseline_metrics[1, ], use.names = FALSE),
Unit = c("percentage points", "percentage points", "percentage points squared",
"percentage points", "%", "unit-free")
)
brand_table(accuracy_table, "Baseline model: out-of-sample forecast accuracy", digits = 4)
```
::: {.callout-note}
## MAPE caution
MAPE cannot be calculated when any actual test value is zero. The code reports `NA` in that case rather than dividing by zero. MAPE can also become very large when actual values are close to zero, so it should be read alongside MAE and RMSE.
:::
## 4.2 Compare the regression with a simple benchmark
The benchmark forecast is a constant equal to the average renewable energy consumption in the estimation sample. It uses no information from the test-period dependent variable.
```{r}
#| label: benchmark-comparison
benchmark_value <- mean(train_data$REN_ERG_CONS)
forecast_mean_benchmark <- rep(benchmark_value, nrow(test_data))
benchmark_metrics <- calc_metrics(actual_test, forecast_mean_benchmark)
benchmark_comparison <- data.frame(
Model = c("Multiple OLS regression", "Estimation-sample mean benchmark"),
ME = c(baseline_metrics$ME, benchmark_metrics$ME),
MAE = c(baseline_metrics$MAE, benchmark_metrics$MAE),
RMSE = c(baseline_metrics$RMSE, benchmark_metrics$RMSE),
MAPE = c(baseline_metrics$MAPE, benchmark_metrics$MAPE)
)
brand_table(benchmark_comparison, "Regression forecasts versus the constant-mean benchmark", digits = 4)
```
The regression has a lower MAE than the benchmark: **`r ifelse(baseline_metrics$MAE < benchmark_metrics$MAE, "Yes", "No")`**. It has a lower RMSE: **`r ifelse(baseline_metrics$RMSE < benchmark_metrics$RMSE, "Yes", "No")`**. A lower error indicates better performance for that measure on this particular test period.
::: {.callout-tip}
## How to judge the benchmark comparison
The regression is useful for prediction only if it improves on a simple alternative.
Consider MAE and RMSE together: MAE describes the typical absolute miss, while RMSE places more weight on large forecast errors. The results apply only to this short test period.
:::
## 4.3 Theil's inequality coefficient and MSE decomposition
Theil's inequality coefficient is
\[
U_1 = \frac{\sqrt{m^{-1}\sum_{t=1}^{m}(\hat y_t-y_t)^2}}
{\sqrt{m^{-1}\sum_{t=1}^{m}\hat y_t^2}+
\sqrt{m^{-1}\sum_{t=1}^{m}y_t^2}}.
\]
It is zero for perfect forecasts. The MSE can also be split into three proportions:
\[
U^M = \frac{(\bar y-\overline{\hat y})^2}{MSE}, \qquad
U^S = \frac{(s_y-s_{\hat y})^2}{MSE}, \qquad
U^C = \frac{2(s_y s_{\hat y}-\operatorname{Cov}(y,\hat y))}{MSE}.
\]
Here, the standard deviations and covariance use divisor \(m\). The three components sum to one when MSE is positive. They describe the shares associated with differences in mean, spread and co-movement.
```{r}
#| label: theil-decomposition-function
sd_population <- function(x) {
sqrt(mean((x - mean(x))^2))
}
theil_decomposition <- function(actual, predicted) {
if (length(actual) != length(predicted) || length(actual) == 0) {
stop("Actual values and forecasts must have the same positive length.")
}
if (any(!is.finite(actual)) || any(!is.finite(predicted))) {
stop("Actual values and forecasts must be finite.")
}
mse_value <- mean((actual - predicted)^2)
if (mse_value == 0) {
return(data.frame(
Component = c("Bias (means)", "Variance (spread)", "Covariance (co-movement)"),
Share = NA_real_,
Percent = NA_real_
))
}
sd_actual <- sd_population(actual)
sd_predicted <- sd_population(predicted)
covariance_population <- mean(
(actual - mean(actual)) * (predicted - mean(predicted))
)
u_m <- (mean(actual) - mean(predicted))^2 / mse_value
u_s <- (sd_actual - sd_predicted)^2 / mse_value
u_c <- 2 * (sd_actual * sd_predicted - covariance_population) / mse_value
shares <- c(u_m, u_s, u_c)
if (!isTRUE(all.equal(sum(shares), 1, tolerance = 1e-8))) {
stop("The Theil decomposition does not sum to one; check the calculations.")
}
data.frame(
Component = c("Bias (means)", "Variance (spread)", "Covariance (co-movement)"),
Share = shares,
Percent = 100 * shares
)
}
baseline_theil <- theil_decomposition(actual_test, forecast_baseline)
u1_table <- data.frame(Model = "Baseline OLS model", Theil_U1 = baseline_metrics$Theil_U1)
brand_table(u1_table, "Theil's U1 for the baseline model", digits = 5)
brand_table(baseline_theil, "Theil's MSE decomposition: baseline OLS model", digits = 4)
```
```{r}
#| label: theil-decomposition-plot
#| fig-cap: "Shares of the baseline model's out-of-sample MSE."
#| fig-width: 7
#| fig-height: 4
if (all(is.finite(baseline_theil$Percent))) {
theil_plot_data <- baseline_theil |>
mutate(Component = factor(Component, levels = Component))
ggplot(theil_plot_data, aes(x = Component, y = Percent, fill = Component)) +
geom_col(show.legend = FALSE, width = 0.65) +
geom_text(aes(label = sprintf("%.1f%%", Percent)), vjust = -0.35, size = 3.5) +
scale_fill_manual(values = c(brand, brand_blue, brand_green)) +
scale_y_continuous(limits = c(0, max(100, max(theil_plot_data$Percent) * 1.12))) +
labs(
title = "Theil decomposition of forecast MSE",
x = NULL,
y = "Share of MSE (%)"
) +
theme_minimal(base_size = 12) +
theme(
axis.text.x = element_text(angle = 12, hjust = 1),
plot.title = element_text(colour = brand_dark, face = "bold")
)
} else {
cat("The decomposition is undefined because the forecasts are perfect (MSE = 0).")
}
```
The largest component is **`r if (all(is.finite(baseline_theil$Percent))) baseline_theil$Component[which.max(baseline_theil$Percent)] else "not defined"`**.
A larger bias share points to a difference in average levels.
A larger variance share points to a difference in the spread of actual and forecast values.
A larger covariance share reflects imperfect co-movement.
# 5. Confidence and prediction intervals
For each test year, `predict.lm()` produces:
- a **95% confidence interval (CI)** for the conditional mean renewable energy consumption at that year's observed explanatory-variable values;
- a **95% prediction interval (PI)** for an individual annual observation. The PI is wider because it also includes the variation of the disturbance term.
```{r}
#| label: prediction-intervals
ci_baseline <- predict(model7, newdata = test_data,
interval = "confidence", level = 0.95)
pi_baseline <- predict(model7, newdata = test_data,
interval = "prediction", level = 0.95)
intervals_data <- data.frame(
Year = test_data$Year,
Actual = actual_test,
Forecast = forecast_baseline,
CI_lower = ci_baseline[, "lwr"],
CI_upper = ci_baseline[, "upr"],
PI_lower = pi_baseline[, "lwr"],
PI_upper = pi_baseline[, "upr"]
)
brand_table(intervals_data, "Classical 95% confidence and prediction intervals", digits = 2)
pi_coverage <- mean(
actual_test >= intervals_data$PI_lower & actual_test <= intervals_data$PI_upper
)
coverage_table <- data.frame(
Measure = "Observed test values inside the 95% prediction intervals",
Covered = sum(actual_test >= intervals_data$PI_lower & actual_test <= intervals_data$PI_upper),
Test_observations = length(actual_test),
Empirical_coverage_percent = 100 * pi_coverage
)
brand_table(coverage_table, "Empirical coverage of the prediction intervals", digits = 1)
```
::: {.callout-warning}
## Limits of classical OLS intervals for annual macroeconomic data
The standard `predict.lm()` intervals rely on the usual linear-model error assumptions, including independent errors with constant variance and an approximately normal error distribution for exact small-sample inference. Annual macroeconomic data may exhibit serial correlation, heteroscedasticity and structural changes. The nominal 95% level should therefore not be interpreted as a guarantee of 95% coverage in this short test period.
:::
# 6. Interpretation of the baseline forecasting results
The baseline model's test-period MAE is **`r round(baseline_metrics$MAE, 3)` percentage points**, RMSE is **`r round(baseline_metrics$RMSE, 3)` percentage points**, and ME is **`r round(baseline_metrics$ME, 3)` percentage points**. The sign of ME indicates that the model **`r if (baseline_metrics$ME > 0) "underpredicts on average" else if (baseline_metrics$ME < 0) "overpredicts on average" else "has no average signed error"`** in the test period.
Compared with the estimation-sample mean benchmark, the regression has **`r ifelse(baseline_metrics$MAE < benchmark_metrics$MAE, "lower", "higher")` MAE** and **`r ifelse(baseline_metrics$RMSE < benchmark_metrics$RMSE, "lower", "higher")` RMSE**. The result should be assessed using both measures, not a single statistic.
The Theil decomposition helps identify whether forecast errors are more closely associated with a difference in average level, variation or co-movement. These results describe this particular 2016–2021 holdout period and should not be generalised to all future years. In addition, the full sample was examined during beforehand and the test-period regressors are treated as known. These two limitations reduce the strength of any claim about genuine real-time forecasting performance.
# 7. Mini-tasks (Malé cvičenie)
## 7.1 Task 1: Compare the linear-GDP and log-GDP specifications
The alternative specification replaces `GDP_billions` with `log(GDP_billions)`. It is estimated on the same estimation sample and forecasts the same test years. GDP must be positive in both samples for the logarithm to be defined.
```{r}
#| label: task1-log-gdp-model
if (all(train_data$GDP_billions > 0) && all(test_data$GDP_billions > 0)) {
model7_log <- lm(
REN_ERG_CONS ~ FDI_GDP + log(GDP_billions) + IND_VA_GDP + NODA_RCV + Trade_GDP,
data = train_data
)
forecast_log <- as.numeric(predict(model7_log, newdata = test_data))
log_metrics <- calc_metrics(actual_test, forecast_log)
log_model_available <- TRUE
log_coefficients <- broom::tidy(model7_log) |>
select(
Term = term,
Estimate = estimate,
`Std. error` = std.error,
`t statistic` = statistic,
`p-value` = p.value
)
brand_table(log_coefficients, "Log-GDP model estimates on the estimation sample", digits = 4)
model_comparison <- data.frame(
Model = c("Linear GDP", "Log GDP"),
ME = c(baseline_metrics$ME, log_metrics$ME),
MAE = c(baseline_metrics$MAE, log_metrics$MAE),
MSE = c(baseline_metrics$MSE, log_metrics$MSE),
RMSE = c(baseline_metrics$RMSE, log_metrics$RMSE),
MAPE = c(baseline_metrics$MAPE, log_metrics$MAPE),
Theil_U1 = c(baseline_metrics$Theil_U1, log_metrics$Theil_U1)
)
brand_table(model_comparison, "Out-of-sample comparison: linear GDP versus log GDP", digits = 4)
} else {
log_model_available <- FALSE
model7_log <- NULL
forecast_log <- NULL
log_metrics <- NULL
cat("Task 1 cannot be estimated because GDP_billions is not strictly positive in both samples.")
}
```
```{r}
#| label: task1-model-choice
if (log_model_available) {
preferred_by_mae <- ifelse(baseline_metrics$MAE <= log_metrics$MAE,
"Linear-GDP model", "Log-GDP model")
preferred_by_rmse <- ifelse(baseline_metrics$RMSE <= log_metrics$RMSE,
"Linear-GDP model", "Log-GDP model")
cat("Lower MAE: ", preferred_by_mae, ".\n", sep = "")
cat("Lower RMSE: ", preferred_by_rmse, ".", sep = "")
} else {
cat("No comparison is available because the log-GDP model could not be estimated.")
}
```
::: {.callout-tip}
## Task 1 interpretation
Use the MAE and RMSE columns to identify which specification performs better on the test years. Do not assume that the log model is better simply because it is a common transformation. Once the test results are used to choose between specifications, the test sample has become part of the model-selection process and is no longer an independent final test set.
:::
## 7.2 Task 2: Decompose the log-GDP model's MSE
Calculate the same Theil components for the log-GDP forecasts and compare them with the baseline model. This shows whether the two specifications differ mainly in their forecast means, spread or co-movement with actual renewable energy consumption.
```{r}
#| label: task2-log-gdp-theil
#| results: asis
if (log_model_available) {
log_theil <- theil_decomposition(actual_test, forecast_log)
theil_comparison <- data.frame(
Component = baseline_theil$Component,
`Linear-GDP share` = baseline_theil$Share,
`Linear-GDP percent` = baseline_theil$Percent,
`Log-GDP share` = log_theil$Share,
`Log-GDP percent` = log_theil$Percent,
check.names = FALSE
)
print(
brand_table(
theil_comparison,
"Theil's MSE decomposition: linear GDP versus log GDP",
digits = 4
)
)
cat(
"\n\nThe linear-GDP components sum to **", round(sum(baseline_theil$Share), 6),
"**; the log-GDP components sum to **", round(sum(log_theil$Share), 6), "**.",
sep = ""
)
} else {
log_theil <- NULL
cat("Task 2 cannot be completed because the log-GDP model is unavailable.")
}
```
## 7.3 Task 3: Compare confidence-interval and prediction-interval widths for one year
Use the first year in the test sample. The confidence interval estimates the conditional mean, while the prediction interval covers an individual annual outcome and therefore includes an additional source of uncertainty.
```{r}
#| label: task3-interval-width
single_year_intervals <- intervals_data[1, ] |>
mutate(
CI_width = CI_upper - CI_lower,
PI_width = PI_upper - PI_lower,
PI_to_CI_width_ratio = PI_width / CI_width
)
brand_table(
single_year_intervals,
paste0("95% confidence and prediction interval widths: year ", intervals_data$Year[1]),
digits = 3
)
```
For **`r single_year_intervals$Year`**, the prediction interval is **`r round(single_year_intervals$PI_width, 3)` percentage points** wide, while the confidence interval is **`r round(single_year_intervals$CI_width, 3)` percentage points** wide. The prediction interval is wider because it includes both uncertainty about the estimated regression mean and the year-specific disturbance around that mean.
## 7.4 Task 4: Reflect on data leakage and the timing of the split
The chronological estimation/test split should ideally be created before exploratory analysis, variable selection or functional-form decisions. In this project, we already examined the full `1995–2021` Sudan sample beforehand and specified the regressors used here. Consequently, the later test years may indirectly have influenced the model specification.
Re-estimating the model on the earlier years prevents the test-period outcomes from entering the coefficient estimates, but it cannot undo the earlier full-sample exploration. The forecast metrics are therefore useful for a simple demonstration, but they are not a fully independent measure of predictive performance on unseen data.
::: {.callout-warning}
## What would improve the validation?
For a future project, define the final test period before exploring the data, use only earlier years for model development, and consider a rolling-origin evaluation across several forecast origins. A separate forecast of FDI and the control variables would also be needed for genuinely ex-ante forecasts.
:::
# 8. Takeaways
This exercise extends the Sudan analysis from in-sample OLS estimation and diagnostics to a chronological holdout evaluation. It assesses whether the multiple regression containing FDI, GDP, industry, ODA and trade can predict renewable energy consumption better than a simple estimation-sample-mean benchmark for the held-out years. The FDI coefficient remains an important conditional association in the model, but the forecast assessment concerns the predictive performance of the **whole specification**, not the isolated causal effect of FDI.
The conclusions should remain cautious because the test sample is short, the full data were examined beforehand, the forecasts use realised test-period regressors, and classical OLS intervals may not fully reflect time-series uncertainty.
# 9. Save the results
Here, we save the data split, fitted models, forecasts and evaluation results in a new file.
```{r}
#| label: save-exercise7-results
save_objects <- c(
"sudan_clean", "udaje7", "train_data", "test_data", "model7",
"forecast_table", "baseline_metrics", "benchmark_metrics",
"baseline_theil", "intervals_data", "coverage_table"
)
if (log_model_available) {
save_objects <- c(save_objects, "model7_log", "forecast_log", "log_metrics", "log_theil")
}
save(
list = save_objects,
file = "sudan_exercise7_forecasting_results.RData"
)
cat("Saved Exercise 7 results to sudan_exercise7_forecasting_results.RData")
```
# Final checklist
- [ ] The research is concerned with FDI and renewable energy consumption in Sudan.
- [ ] The cleaned dataset is restored from previous exercise.
- [ ] The annual observations are sorted and split chronologically.
- [ ] The baseline model is re-estimated using the estimation sample only.
- [ ] Forecasts and forecast errors are reported for every test year.
- [ ] Forecast performance is compared with the estimation-sample-mean benchmark.
- [ ] ME, MAE, MSE, RMSE, MAPE and Theil's measures are reported with appropriate caveats.
- [ ] The 95% confidence and prediction intervals are reported and compared.
- [ ] All *Malé cvičenie* tasks are completed.
- [ ] Data leakage and the use of realised future regressors are acknowledged.
- [ ] The Exercise 7 results are saved.