Pre-Homogenisation Air Minimum Temperature (TN) Tutorial

Author

Vlad Amihaesei & Sorin Cheval

Introduction

This tutorial walks you through the process of preparing minimum temperature data for homogenization. We will process station metadata, merge multiple data files, and format the dataset for further analysis. The data represents the daily maximum air temperature of 29 weather stations from Romania. The data was downloaded from https://www.ecad.eu/dailydata/. The data is stored as .txt files, each station represents one text file. Also, the data contains a meta data file of the stations. All the files are stored in the ECA_non-blended_custom_RO/ folder. The data homogenisation analysis will be made using climatol library. For more details please see: https://climatol.eu/

# Load required libraries
# Package names
packages <- c("dplyr", "reshape2", "lubridate")

# Install packages not yet installed
installed_packages <- packages %in% rownames(installed.packages())
if (any(installed_packages == FALSE)) {
  install.packages(packages[!installed_packages])
}
# Packages loading
invisible(lapply(packages, library, character.only = TRUE))

Step 1: Set Up Your Environment

Before running the script, make sure to set your working directory and load any required functions. In the

# Set working directory (modify this for your local setup)

setwd("~/D/diverse/")
source("R/function_convert_to_decimals.R")
### or 
### Function to convert DMS to decimal degrees
dms_to_decimal <- function(dms) {
  
  parts <- unlist(strsplit(dms, ":"))  # Split by ":"
  degrees <- as.numeric(parts[1])
  minutes <- as.numeric(parts[2])
  seconds <- as.numeric(parts[3])
  
  # Convert to decimal degrees
  decimal <- abs(degrees) + (minutes / 60) + (seconds / 3600)
  
  # Preserve sign
  if (degrees < 0) {
    decimal <- -decimal
  }
  
  return(decimal)
}

Step 2: Load and Process Metadata

We read the metadata file and convert latitude/longitude from degrees, minutes, and seconds to decimal format.

# Read metadata
meta.tn <- read.table("tabs/ECA_non-blended_custom_RO/_sources_TN.txt", sep = ",", skip = 23, header = T)

# Convert coordinates and clean station names
meta.tn <- meta.tn |> 
        mutate(Latitudine = as.numeric(sapply(LAT, dms_to_decimal)),
               Longitudine = as.numeric(sapply(LON, dms_to_decimal)),
               SOUNAME = gsub(" ", "", SOUNAME, fixed = TRUE)) |> 
        select(Longitudine, Latitudine, HGHT, SOUID, SOUNAME)

# Save processed metadata
write.csv(meta.tn, "tabs1/metadata_TN.csv", row.names = F)

Step 3: Generate a Complete Time Series

We create a complete time series from 1896 to 2024 for all stations.

# Define date range
dates <- seq(as.Date("1896-01-01"), as.Date("2024-12-31"), by = "day")
station_codes <- meta.tn$SOUID

# Create a full time-station data frame
df <- expand.grid(DATE = dates, SOUID = station_codes) |> 
  arrange(DATE, SOUID)

Step 4: Merge Data Files

We read and merge temperature data from multiple files.

# Get list of temperature files
files <- list.files("tabs/ECA_non-blended_custom_RO", full.names = T)
files.tnn <- grep("TN_", files, value = T)

data <- NULL 

# Read and clean temperature data
for(i in seq_along(files.tnn)) {
    tt <- read.table(files.tnn[i], sep = ",", skip = 21, header = T)
    tt <- tt |> mutate(TN = TN * 0.1, 
                        TN = ifelse(TN <= -999, NA, TN), 
                        DATE = as.Date(as.character(DATE), "%Y%m%d")) |> 
              select(SOUID, DATE, TN)
    data <- rbind(data, tt)
}

Step 5: Ensure Data Completeness

We merge the complete time series with the station data to ensure all time steps are present.

data.j <- df |> left_join(data)
Joining with `by = join_by(DATE, SOUID)`
summary(data.j)
      DATE                SOUID              TN        
 Min.   :1896-01-01   Min.   :100682   Min.   :-38.4   
 1st Qu.:1928-04-02   1st Qu.:100696   1st Qu.: -0.8   
 Median :1960-07-02   Median :107487   Median :  5.5   
 Mean   :1960-07-02   Mean   :110561   Mean   :  5.2   
 3rd Qu.:1992-10-01   3rd Qu.:107495   3rd Qu.: 12.3   
 Max.   :2024-12-31   Max.   :254859   Max.   : 27.9   
                                       NA's   :582099  
# Save merged dataset
write.csv(data.j, "tabs1/ECAD_TN_merged.csv", row.names = F)

Step 6: Filter Data for 1990-2022

We extract data only from 2010 to 2022 for further analysis.

data.j <- data.j |> filter(year(DATE) >= 1990 & year(DATE) <= 2022)

Step 7: Reshape Data for Homogenization

We transform the dataset into a wide format where each station is a column.

data_wide <- dcast(data.j, DATE ~ SOUID, value.var = "TN")
data_values <- data_wide[, -1]  # Remove date column

# Ensure station names match metadata
meta.tn$SOUID <- as.character(meta.tn$SOUID)
meta.tn <- meta.tn[match(names(data_values), meta.tn$SOUID),]
identical(meta.tn$SOUID, names(data_values))
[1] TRUE
#cbind(meta.tn$SOUID, names(data_values))

Step 8: Export Data for Homogenization

We export the processed metadata and dataset in the required formats.

write.table(meta.tn, "tn_1990-2022.est", row.names = FALSE, col.names = FALSE)
write(as.matrix(data_values), 'tn_1990-2022.dat')

Start HOMOGENIZATION process

This tutorial guides you through the process of homogenizing daily and monthly temperature data using the climatol package in R. Please uncomment the code lines in order to run. It will take a while. Be patient.

library(climatol)

Step 1: Convert Daily Data to Monthly Averages

To perform homogenization, we first convert daily temperature data into monthly values.

#dd2m('tn', 1990, 2022)

Step 2: Quality Control

Before homogenization, we conduct quality control checks on the data.

#homogen('tn', 1990, 2022, onlyQC = TRUE)

Step 3: Monthly Data Homogenization

We homogenize the monthly temperature data, setting limits for valid temperature values.

#homogen('tn-m', 1990, 2022, vmin = -50, vmax = 50, expl = FALSE)

Step 5: Daily Data Homogenization

Similarly, we homogenize daily temperature data.

#homogen('tn', 1990, 2022, vmin = -50, vmax = 50, expl = FALSE, metad = T)

Conclusion

This tutorial covers the key steps in temperature homogenization using climatol. Adjust parameters as needed for your dataset.

Acknowledgments

Many thanks to Alex Dumitrescu for assisting us in resolving issues encountered during the homogenization process.