Hawaii Restoration Project Data Exploration

Objective

This notebook explores the data from the Hawaii Restoration Project. The goal is to understand the structure of the data, identify any missing values, and summarize and visualize key variables.

Experimental Design

The Hawaii Restoration Project consists of two main experiments: the FOG experiment and the NURSE experiment.

The Fog Experiment

The fog experiment examined the effect of fog capture on the restoration success of native Hawaiian plants. The experiment was set up at two different sites.

The Nurse Plant Experiment

The nurse experiment examined the effect of nurse plants on the restoration success of native Hawaiian plants. The experiment was set up at two different sites.

Load Data

First step is to load the data. The data will be in the data directory. The data is an excel workbook with three sheets: Metadata, FOG, and NURSE.

• Metadata - contains information about the variables and is loose data dictionary for each of the variables
• FOG - contains data from the fog experiment
• NURSE - contains data from the nurse plant experiment

library(readxl)
# Load the FOG data
fog_data <- readxl::read_excel(file.path(data_dir, "KBC_data_SY.xlsx"), sheet = "FOG")
# Load the NURSE data
nurse_data <- readxl::read_excel(file.path(data_dir, "KBC_data_SY.xlsx"), sheet = "NURSE")

Explore the Raw Data

Let’s take a look at the data straight out of the excel file.

Data table structure

# Check the structure of the FOG data

DataExplorer::plot_str(list(FOG=fog_data,
                            NURSE=nurse_data))

NA

Column Name and Type Adjustments

We want to drop the columns 19-24 in both data frames since they are empty. We should drop the volume columns so that we can show our work for reproducibility. We will still use the volume measure in the original data set (a rectangular prism) with in cubic decimeters (Height X Width X Depth X 1/1,000). Unless, we want to present other volumes. However, this is just scaling the volume (HxWxD and has little effect since we will scale all the variables if used as predictors and everything else will just change the beta coefficients). We can also drop the notes columns since they are not needed for the analysis. We should make all column names lower case. We should rename mauka/makai to position so that we don’t have a slash in our column names. I also want to rename plant1 and species1 to just plant and species. We should convert site, replicate, treatment, species, plant and position to factors. Survival should converted to 0 and 1.

# Adjust column names and types for FOG data
fog_data <- fog_data %>%
  select(-c(19:24)) %>%
  select(-c(Volume_T1, Volume_T2)) %>%
  rename_with(tolower) %>%
  rename(position = `mauka/makai`,plant = plant1, species = species1,
         note1 = 'notes 1', note2 = 'notes 2') %>%
  mutate(across(c(site, replicate, treatment, species, plant, position), as.factor),
         survival_t1 = ifelse(survival_t1 == "Y", 1, 0),
         survival_t2 = ifelse(survival_t2 == "Y", 1, 0))


# Adjust column names and types for NURSE data
nurse_data <- nurse_data %>%
  select(-c(18:24)) %>%
  select(-c(Volume_T1, Volume_T2)) %>%
  rename_with(tolower) %>% 
  rename(plant = plant1, note1= 'notes...10',note2= 'notes...15') %>%
  mutate(across(c(site, replicate, treatment, species, plant), as.factor),
         survival_t1 = ifelse(survival_t1 == "y", 1, 0),
         survival_t2 = ifelse(survival_t2 == "y", 1, 0))

# Check the structure again after adjustments
DataExplorer::plot_str(list(fog_data, nurse_data))

FOG Data Exploration

A quick description of the fog data

# Get a quick description of the data
DataExplorer::introduce(fog_data %>% select(-c(note1, note2))) %>% 
  pivot_longer(cols = everything(),
               names_to = "variable", values_to = "value") %>% 
  kbl() %>% 
  kable_minimal(full_width = F)

variable	value
rows	960
columns	14
discrete_columns	6
continuous_columns	8
all_missing_columns	0
total_missing_values	0
complete_rows	960
total_observations	13440
memory_usage	92264

NA

# Visualize the data description for FOG data
DataExplorer::plot_intro(fog_data %>% select(-c(note1, note2)), title = "FOG Data Overview")

Looking at missing values for FOG data

# Visualize missing data for FOG data
DataExplorer::plot_missing(fog_data %>% select(-c(note1, note2)), title = "Missing Data in FOG Data")

We know there’s an issue here the widths and height variables are coded as 0 when they were not present. We will need to address this in data cleaning. For now, this is super groovy!

Quick look at the distribution of the variables in the FOG data

Discrete variables

# Visualize the distribution of discrete variables in FOG data
DataExplorer::plot_bar(fog_data %>% select(-c(note1, note2)), ncol = 3, title = "Distribution of Discrete Variables in FOG Data")

Need to fix mauka (1/18/2022) and to Mauka. THIS NEEDS TO BE FIXED IN THE DATA CLEANING STEP!

# Fix the mauka/makai values in FOG data
fog_data <- fog_data %>%
  mutate(position = recode(position, "mauka (1/18/2022)" = "mauka"))

DataExplorer::plot_bar(fog_data %>% select(-c(note1, note2)), ncol = 3, title = "Distribution of Discrete Variables in FOG Data")

Continuous variables

# Visualize the distribution of continuous variables in FOG data

# Visualize the distribution of continuous variables in FOG data

DataExplorer::plot_histogram(fog_data %>% select(-c(note1, note2)),
                             ncol = 3, title = "Distribution of Continuous Variables in FOG Data")

For the data cleaning we need to address the 0 values in the width and height variables. While looking at this we have options:

1. We can replace the 0 values with NA.
2. We can replace widths and heights were the survival is 0 with NA.

I think for now we will go with option 1. We can always revisit this later if we need to.

NURSE Data Exploration

A quick description of the nurse data

# Get a quick description of the data
DataExplorer::introduce(nurse_data %>% select(-c(note1, note2))) %>% 
  pivot_longer(cols = everything(),
               names_to = "variable", values_to = "value") %>% 
  kbl() %>% 
  kable_minimal(full_width = F)

variable	value
rows	452
columns	13
discrete_columns	5
continuous_columns	8
all_missing_columns	0
total_missing_values	6
complete_rows	446
total_observations	5876
memory_usage	44920

NA

# Visualize the data description for FOG data
DataExplorer::plot_intro(nurse_data %>% select(-c(note1, note2)), title = "NURSE Data Overview")

Looking at missing values for NURSE data

# Visualize missing data for FOG data
DataExplorer::plot_missing(nurse_data %>% select(-c(note1, note2)), title = "Missing Data in FOG Data")

We know there’s an issue here the widths and height variables are coded as 0 when they were not present. We need to determine what is missing in the plant column.

# Print a table of the missing plant number in the NURSE data
nurse_data %>%
  filter(is.na(plant)) %>%
  kbl() %>%
  kable_minimal(full_width = F)

site	replicate	treatment	species	plant	ht_t1	w1_t1	w2_t1	survival_t1	note1	ht_t2	w1_t2	w2_t2	survival_t2	note2
upper	9	nurse upper	aalii	NA	0	0	0	0	nf	0	0	0	0	nf
upper	9	nurse upper	aweoweo	NA	0	0	0	0	nf	0	0	0	0	nf
upper	9	nurse upper	mamane	NA	0	0	0	0	nf	0	0	0	0	nf
upper	9	nurse upper	pawale	NA	0	0	0	0	nf	0	0	0	0	nf
upper	9	nurse upper	pawale	NA	0	0	0	0	nf	0	0	0	0	nf
upper	9	nurse upper	naenae	NA	0	0	0	0	nf	0	0	0	0	nf

Okay these are all plants from the upper site, replicate 9 and have no measurements. Notes indicate not found.

Quick look at the distribution of the variables in the FOG data

Discrete variables

# Visualize the distribution of discrete variables in FOG data
DataExplorer::plot_bar(nurse_data %>% select(-c(note1, note2)), ncol = 3, title = "Distribution of Discrete Variables in NURSE Data")

We probably want to change nurse upper to align with the naming protocol of the lower site. I also need to get rid of the space in the treatment variable value. Nurse k = koa; nurse m = mamane; control = control and nurse = upper = UNSURE!?!.

Continuous variables

# Visualize the distribution of continuous variables in NURSE data

DataExplorer::plot_histogram(nurse_data %>% select(-c(note1, note2)),
                             ncol = 3,
                             title = "Distribution of Continuous Variables in NURSE Data")

For the data cleaning we need to address the 0 values in the width and height variables. While looking at this we have options:

1. We can replace the 0 values with NA.
2. We can replace widths and heights were the survival is 0 with NA.

I think for now we will go with option 1. We can always revisit this later if we need to.

Data issues

FOG Data Issues

Duplicates

The easiest issue to identify the problem is duplicates. Let’s check for duplicates in the FOG data.

# Check for duplicates in FOG data
duplicates_fog <- fog_data %>%
  janitor::get_dupes(site, replicate, treatment, species, plant, position)

writexl::write_xlsx(duplicates_fog, file.path(notes_dir, "FOG_dupes.xlsx"))

duplicates_fog %>%
  kbl() %>%
  kable_minimal(full_width = F)

site	replicate	treatment	species	plant	position	dupe_count	ht_t1	w1_t1	w2_t1	survival_t1	note1	ht_t2	w1_t2	w2_t2	survival_t2	note2
lower	1	0	pawale	1	mauka	2	40.0	66.0	56.0	1	NA	50.0	34.5	39.0	1	NA
lower	1	0	pawale	1	mauka	2	50.5	86.0	63.0	1	NA	0.0	0.0	0.0	0	dead
lower	1	40	aalii	1	makai	2	45.5	13.0	17.0	1	NA	0.0	0.0	0.0	0	NA
lower	1	40	aalii	1	makai	2	0.0	0.0	0.0	0	dead	0.0	0.0	0.0	0	NA
lower	1	40	aalii	1	mauka	2	40.0	10.0	18.5	1	NA	41.0	12.0	16.0	1	NA
lower	1	40	aalii	1	mauka	2	29.0	28.5	28.0	1	3 different stems	33.0	21.0	37.0	1	fruiting
lower	1	40	aalii	2	makai	2	39.0	11.0	15.0	1	NA	0.0	0.0	0.0	0	NA
lower	1	40	aalii	2	makai	2	0.0	0.0	0.0	0	dead	0.0	0.0	0.0	0	NA
lower	1	40	aalii	2	mauka	2	47.0	21.0	10.0	1	2 stems	48.0	18.0	25.0	1	NA
lower	1	40	aalii	2	mauka	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	NA
lower	1	40	aweoweo	1	makai	2	54.0	13.5	14.5	1	NA	59.0	11.0	8.5	1	NA
lower	1	40	aweoweo	1	makai	2	0.0	0.0	0.0	0	broken stem just died	0.0	0.0	0.0	0	NA
lower	1	40	aweoweo	1	mauka	2	13.0	3.0	2.0	1	dead	43.0	34.0	33.0	1	dying/ not found measure 1
lower	1	40	aweoweo	1	mauka	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	nf
lower	1	40	aweoweo	2	makai	2	58.0	40.0	77.5	1	NA	110.0	67.0	47.0	1	NA
lower	1	40	aweoweo	2	makai	2	94.0	55.0	53.0	1	NA	42.0	83.0	65.0	1	NA
lower	1	40	aweoweo	2	mauka	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	nf
lower	1	40	aweoweo	2	mauka	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	NA
lower	1	40	mamane	1	makai	2	7.0	2.5	2.5	1	looks dead coming back	10.0	7.0	9.0	1	NA
lower	1	40	mamane	1	makai	2	30.0	21.0	8.5	1	NA	39.5	14.0	13.5	1	NA
lower	1	40	mamane	1	mauka	2	24.0	9.0	6.5	1	NA	0.0	0.0	0.0	0	nf
lower	1	40	mamane	1	mauka	2	41.0	11.0	13.0	1	NA	0.0	0.0	0.0	0	NA
lower	1	40	mamane	2	makai	2	27.0	7.0	6.0	1	NA	30.5	16.0	13.5	1	NA
lower	1	40	mamane	2	makai	2	32.0	13.0	7.0	1	NA	36.5	16.0	20.0	1	NA
lower	1	40	mamane	2	mauka	2	40.0	10.0	11.0	1	NA	45.0	15.0	16.0	1	NA
lower	1	40	mamane	2	mauka	2	0.0	0.0	0.0	0	dead	0.0	0.0	0.0	0	nf
lower	1	40	naenae	1	makai	2	67.0	32.0	27.0	1	NA	73.0	44.0	35.0	1	big mama
lower	1	40	naenae	1	makai	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	NA
lower	1	40	naenae	1	mauka	2	33.5	19.0	23.5	1	NA	27.5	11.0	9.0	1	NA
lower	1	40	naenae	1	mauka	2	42.5	20.0	28.0	1	pro	52.0	69.0	45.0	1	flowering
lower	1	40	pawale	1	makai	2	37.0	39.0	34.0	1	NA	51.5	27.0	25.0	1	NA
lower	1	40	pawale	1	makai	2	70.0	70.5	62.0	1	NA	65.0	73.0	70.0	1	NA
lower	1	40	pawale	1	mauka	2	32.0	43.5	33.0	1	NA	37.0	38.0	72.0	1	NA
lower	1	40	pawale	1	mauka	2	62.0	96.0	31.0	1	NA	52.0	38.0	59.0	1	NA
lower	1	40	pawale	2	makai	2	30.5	26.0	35.0	1	NA	22.0	26.5	15.0	1	NA
lower	1	40	pawale	2	makai	2	63.0	37.0	44.5	1	NA	84.0	37.0	46.0	1	NA
lower	1	40	pawale	2	mauka	2	41.0	63.0	70.0	1	NA	61.0	37.0	36.0	1	NA
lower	1	40	pawale	2	mauka	2	81.0	55.0	57.0	1	NA	63.0	70.0	102.0	1	mostly dead, 3 green leaves
lower	2	0	aalii	1	mauka	2	45.0	0.0	0.0	0	just died	0.0	0.0	0.0	0	NA
lower	2	0	aalii	1	mauka	2	34.0	0.0	0.0	0	just died	0.0	0.0	0.0	0	dead
lower	2	0	pawale	1	mauka	2	51.0	30.0	18.0	1	NA	35.0	15.0	28.0	1	NA
lower	2	0	pawale	1	mauka	2	66.0	87.0	80.0	1	NA	87.0	102.0	93.0	1	NA
lower	3	40	mamane	1	mauka	2	41.0	11.0	4.0	1	NA	44.0	10.5	9.0	1	NA
lower	3	40	mamane	1	mauka	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	NA
lower	4	100	aalii	1	control	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	NA
lower	4	100	aalii	1	control	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	NA
lower	4	100	aweoweo	1	control	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	NA
lower	4	100	aweoweo	1	control	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	NA
lower	4	100	mamane	1	control	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	NA
lower	4	100	mamane	1	control	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	NA
lower	4	100	pawale	1	control	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	NA
lower	4	100	pawale	1	control	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	NA
lower	5	40	aalii	1	mauka	2	37.0	0.0	0.0	0	just died	0.0	0.0	0.0	0	dead
lower	5	40	aalii	1	mauka	2	28.0	0.0	0.0	0	just died	0.0	0.0	0.0	0	dead
lower	5	63	pawale	1	makai	2	29.0	32.0	28.0	1	NA	44.0	29.0	13.0	1	NA
lower	5	63	pawale	1	makai	2	29.0	32.0	28.0	1	NA	44.0	29.0	13.0	1	NA
upper	1	0	mamane	1	mauka	2	21.5	3.0	2.0	1	NA	0.0	0.0	0.0	0	dead
upper	1	0	mamane	1	mauka	2	0.0	0.0	0.0	0	dead	0.0	0.0	0.0	0	dead
upper	1	40	aweoweo	1	mauka	2	59.0	25.0	42.0	1	NA	35.0	28.0	33.0	1	next to mullen, pro
upper	1	40	aweoweo	1	mauka	2	88.0	42.0	20.0	1	NA	123.0	62.0	52.0	1	big boy
upper	2	63	aweoweo	1	mauka	2	13.0	3.0	2.0	1	dead	87.0	50.0	70.0	1	fence in way, not found on first measurement
upper	2	63	aweoweo	1	mauka	2	61.0	32.0	45.0	1	NA	148.0	152.0	142.0	1	NA
upper	2	63	mamane	1	mauka	2	38.0	12.0	9.0	1	NA	51.0	18.0	12.0	1	NA
upper	2	63	mamane	1	mauka	2	0.0	0.0	0.0	0	dead	0.0	0.0	0.0	0	NA
upper	4	63	aweoweo	1	makai	2	17.0	21.0	28.0	1	dead	0.0	0.0	0.0	0	dead
upper	4	63	aweoweo	1	makai	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	NA
upper	4	100	mamane	1	control	2	4.0	1.0	1.5	1	dead	0.0	0.0	0.0	0	dead
upper	4	100	mamane	1	control	2	4.0	1.0	1.0	1	NA	0.0	0.0	0.0	0	dead
upper	6	100	aalii	1	control	2	26.0	12.0	15.0	1	NA	33.0	3.0	3.0	1	NA
upper	6	100	aalii	1	control	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	dead
upper	7	40	mamane	1	mauka	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	NA
upper	7	40	mamane	1	mauka	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	NA
upper	8	0	mamane	1	makai	2	0.0	0.0	0.0	0	dead	0.0	0.0	0.0	0	NA
upper	8	0	mamane	1	makai	2	0.0	0.0	0.0	0	dead	0.0	0.0	0.0	0	NA
upper	8	100	aweoweo	1	control	2	23.0	19.0	15.0	1	NA	46.0	4.0	3.0	1	NA
upper	8	100	aweoweo	1	control	2	54.0	17.0	20.0	1	NA	32.0	6.0	6.0	1	NA
upper	8	100	mamane	1	control	2	4.0	3.0	8.0	1	NA	0.0	0.0	0.0	0	dead
upper	8	100	mamane	1	control	2	11.0	7.0	2.0	1	NA	0.0	0.0	0.0	0	dead
upper	9	40	aalii	1	makai	2	28.0	7.0	4.0	1	NA	0.0	0.0	0.0	0	NA
upper	9	40	aalii	1	makai	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	NA
upper	9	63	aweoweo	2	mauka	2	49.0	82.5	39.5	1	NA	106.0	174.0	75.0	1	NA
upper	9	63	aweoweo	2	mauka	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	nf
upper	10	0	mamane	1	mauka	2	2.0	3.0	1.0	1	NA	0.0	0.0	0.0	0	NA
upper	10	0	mamane	1	mauka	2	0.0	0.0	0.0	0	NA	0.0	0.0	0.0	0	NA

We have 84 duplicates in the FOG data. We will need to address these in the data cleaning step. For now, we will just note them.

T1 Measurements

Survivors with no measurement

Let’s check for survivors that have no measurements. This could indicate a problem with the data collection or entry.

#' Filter survivors with missing or zero measurements
#'
#' This function filters a data frame or tibble to include only rows where the
#' specified survival column equals 1 and at least one of the specified measurement
#' columns is either zero or (optionally) NA. It returns a subset of selected columns.
#'
#' @param df A data frame or tibble containing the data.
#' @param survival_col A string specifying the name of the survival column.
#'    Default is `"survival_t1"`.
#' @param measurement_cols A character vector of column names to check
#'    for zero or missing values.
#'                         Default is `c("ht_t1", "w1_t1", "w2_t1")`.
#' @param select_cols A character vector of column names to include in the output.
#'                    Default includes:
#'                      site, replicate, treatment, species, plant, position.
#' @param treat_na_as_missing Logical. If `TRUE`,
#'    NA values in measurement columns are treated as missing
#'                            and included in the filter. Default is `TRUE`.
#'
#' @return A tibble containing the filtered rows and selected columns.
#' @examples
#' filter_survivors_no_measurement(fog_data)
#' filter_survivors_no_measurement(fog_data, treat_na_as_missing = FALSE)
#' filter_survivors_no_measurement(fog_data,
#'                                 survival_col = "alive_t1",
#'                                 measurement_cols = c("height_t1", "width1_t1", "width2_t1"),
#'                                 select_cols = c("site", "species", "plant", "height_t1", "width1_t1", "width2_t1"))
filter_survivors_no_measurement <- function(df,
                                            survival_col = "survival_t1",
                                            measurement_cols =
                                              c("ht_t1", "w1_t1", "w2_t1"),
                                            select_cols =
                                              c("site",
                                                "replicate",
                                                "treatment",
                                                "species",
                                                "plant",
                                                "position"),
                                            treat_na_as_missing = TRUE) {
  # Check for required columns
  required <- c(survival_col, measurement_cols, select_cols)
  missing <- setdiff(required, names(df))
  if (length(missing) > 0) {
    stop("Missing required columns: ", paste(missing, collapse = ", "))
  }
  
  # Build dynamic filter expression
  zero_or_na_exprs <- lapply(measurement_cols, function(col) {
    if (treat_na_as_missing) {
      rlang::expr((!!rlang::sym(col) == 0) | is.na(!!rlang::sym(col)))
    } else {
      rlang::expr(!!rlang::sym(col) == 0)
    }
  })
  
  combined_measurement_expr <- purrr::reduce(zero_or_na_exprs, function(x, y)
    rlang::expr(!!x | !!y))
  
  full_filter_expr <- rlang::expr(!!rlang::sym(survival_col) == 1 &
                                    (!!combined_measurement_expr))
  
  df %>%
    filter(!!full_filter_expr) %>%
    select(all_of(select_cols))
}

# Check for survivors at t1 with no measurements in FOG data
filter_survivors_no_measurement(
  fog_data,
  survival_col = "survival_t1",
  measurement_cols = c("ht_t1", "w1_t1", "w2_t1"),
  select_cols = c("site", "replicate", "treatment",
                  "species", "plant", "position",
                  "ht_t1", "w1_t1", "w2_t1")) %>%
  kbl() %>%
  kable_minimal(full_width = F)

site	replicate	treatment	species	plant	position	ht_t1	w1_t1	w2_t1
lower	4	63	aalii	1	mauka	51	0	0
upper	5	40	aweoweo	1	makai	61	10	0
upper	4	63	pawale	1	makai	0	0	0

Just 3, we can address these in the data cleaning step. But we need to choose how we want to handle these. We can either remove them or replace the 0 values with NA or replace all the measurements with NA. For now, we will just note them.

lower-4-63-aalii-mauka-plant1: Note says just died. I vote we code survival as 0 and measurements as NA.

upper-5-40-aweoweo-plant1: Note says eaten. No leaves. I vote we code as 0 and measurements as NA. It is dead at the next time step.

upper-4-63-pawale-plant1: Note says not found. It was found at time step 2. I vote I NA the measurements but code as 1. I’ll Drop.NA before analysis so it won’t be included in any analyses with measurements.

Died but with measurements

Let’s check for plants that died but have measurements. This could indicate a problem with the data collection or entry or could be a valid observation if dead plants were measured.

#' Filter non-survivors with non-zero measurements
#'
#' This function filters a data frame or tibble to include only rows where the
#' specified survival column equals 0 and at least one of the specified measurement
#' columns is greater than zero or (optionally) not NA. It returns a subset of selected columns.
#'
#' @param df A data frame or tibble containing the data.
#' @param survival_col A string specifying the name of the survival column. Default is `"survival_t1"`.
#' @param measurement_cols A character vector of column names to check for non-zero or non-missing values.
#'                         Default is `c("ht_t1", "w1_t1", "w2_t1")`.
#' @param select_cols A character vector of column names to include in the output.
#'                    Default includes site, replicate, treatment, species, plant, position, and measurements.
#' @param treat_na_as_present Logical. If `TRUE`, NA values in measurement columns are treated as present
#'                            and included in the filter. Default is `FALSE`.
#'
#' @return A tibble containing the filtered rows and selected columns.
#' @examples
#' filter_non_survivors_with_measurement(fog_data)
#' filter_non_survivors_with_measurement(fog_data, treat_na_as_present = TRUE)
#' filter_non_survivors_with_measurement(fog_data,
#'                                       survival_col = "alive_t1",
#'                                       measurement_cols = c("height_t1", "width1_t1", "width2_t1"),
#'                                       select_cols = c("site", "species", "plant", "height_t1", "width1_t1", "width2_t1"))
filter_non_survivors_with_measurement <- function(df,
                                                  survival_col = "survival_t1",
                                                  measurement_cols = c("ht_t1", "w1_t1", "w2_t1"),
                                                  select_cols = c("site", "replicate", "treatment", "species", "plant", "position",
                                                                  "ht_t1", "w1_t1", "w2_t1"),
                                                  treat_na_as_present = FALSE) {
  # Check for required columns
  required <- c(survival_col, measurement_cols, select_cols)
  missing <- setdiff(required, names(df))
  if (length(missing) > 0) {
    stop("Missing required columns: ", paste(missing, collapse = ", "))
  }

  # Build dynamic filter expression
  present_exprs <- lapply(measurement_cols, function(col) {
    if (treat_na_as_present) {
      rlang::expr((!!rlang::sym(col) > 0) | is.na(!!rlang::sym(col)))
    } else {
      rlang::expr(!!rlang::sym(col) > 0)
    }
  })

  combined_measurement_expr <- purrr::reduce(present_exprs, function(x, y) rlang::expr(!!x | !!y))

  full_filter_expr <- rlang::expr(
    !!rlang::sym(survival_col) == 0 & (!!combined_measurement_expr)
  )

  df %>%
    filter(!!full_filter_expr) %>%
    select(all_of(select_cols))
}

# Check for plants that died but have measurements in FOG data
filter_non_survivors_with_measurement(
  fog_data,
  survival_col = "survival_t1",
  measurement_cols = c("ht_t1", "w1_t1", "w2_t1"),
  select_cols = c("site", "replicate", "treatment",
                  "species", "plant", "position",
                  "ht_t1", "w1_t1", "w2_t1")) %>%
  kbl() %>%
  kable_minimal(full_width = F)

site	replicate	treatment	species	plant	position	ht_t1	w1_t1	w2_t1
lower	2	0	aalii	1	mauka	45	0	0
lower	2	0	aalii	1	mauka	34	0	0
lower	5	0	aalii	1	mauka	28	0	0
lower	5	40	aalii	1	mauka	37	0	0
lower	5	40	aalii	1	mauka	28	0	0
lower	5	100	aalii	1	control	17	0	0

So 6 observations with 2 duplicate observations. We can address these in the data cleaning step. But we need to choose how we want to handle these. We can either remove them or replace the 0 values with NA, replace all the measurements with NA or leave them but then how do we want to deal with them in the analysis. For now, we will just note them. ### T2 Measurements

lower-2-0-aalii-mauka-p1: Both entries are recorded as just died in notes. I vote they be coded as 0 with measurements NA. We can keep the duplicate records if we can confirm that there are two plants and that plant2 was just miss entered. Or we can delete one of them if we are unsure.

lower-5-0-aalii-mauka-p1: Recorded as just died in notes. I vote code as 0 with measurements, NA.

lower-5-40-aalii-mauka-p1: Both, entries are recorded as just died in notes. I vote code as 0 with measurements, NA. We can keep the duplicate records if we can confirm that there are two plants and that plant2 was just miss entered. Or we can delete one of them if we are unsure.

lower-5-100-aalii-control-p1: Recorded as just died. I vote coded as 0 with measurements, NA.

Survivors with no measurement

# Check for survivors at t2 with no measurements in FOG data
filter_survivors_no_measurement(
  fog_data,
  survival_col = "survival_t2",
  measurement_cols = c("ht_t2", "w1_t2", "w2_t2"),
  select_cols = c("site", "species", "plant", "ht_t2", "w1_t2", "w2_t2")) %>%
  kbl() %>%
  kable_minimal(full_width = F)

site	species	plant	ht_t2	w1_t2	w2_t2

None! that’s great! We can move on to the next step.

Died but with measurements

Let’s check for plants that died but have measurements. This could indicate a problem with the data collection or entry or could be a valid observation if dead plants were measured.

# Check for plants that died but have measurements in FOG data
# Check for plants that died but have measurements in FOG data
filter_non_survivors_with_measurement(
  fog_data,
  survival_col = "survival_t2",
  measurement_cols = c("ht_t2", "w1_t2", "w2_t2"),
  select_cols = c("site", "replicate", "treatment",
                  "species", "plant", "position",
                  "ht_t2", "w1_t2", "w2_t2")) %>%
  kbl() %>%
  kable_minimal(full_width = F)

site	replicate	treatment	species	plant	position	ht_t2	w1_t2	w2_t2

None! that’s great! We can move on to the next step.

Lazurus Plants

Let’s check for plants that were dead in T1 but were alive in T2. This could indicate a problem with the data collection or entry or could be a valid observation if plants were added between T1 and T2.

# Check for lazarus plants in FOG data
lazarus_plants_fog <- fog_data %>%
  filter(survival_t1 == 0 & survival_t2 == 1) %>%
  select(site, replicate, treatment, species, plant, position)

lazarus_plants_fog %>%
  kbl() %>%
  kable_minimal(full_width = F)

site	replicate	treatment	species	plant	position

They have NOT risen!! We have no lazarus plants in the FOG data. That’s great! We can move on to the next step.

NURSE Data Issues

Duplicates

The easiest issue to identify problem is duplicates. Let’s check for duplicates in the FOG data.

# Check for duplicates in FOG data
nurse_duplicates <- nurse_data %>%
  janitor::get_dupes(site, replicate, treatment, species, plant)

writexl::write_xlsx(nurse_duplicates, file.path(notes_dir, "NURSE_dupes.xlsx"))

nurse_duplicates %>% 
  kbl() %>%
  kable_minimal(full_width = F)

site	replicate	treatment	species	plant	dupe_count	ht_t1	w1_t1	w2_t1	survival_t1	note1	ht_t2	w1_t2	w2_t2	survival_t2	note2
lower	1	control	aalii	1	2	33	7	8.0	1	NA	45	6	5	1	NA
lower	1	control	aalii	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	1	control	aalii	2	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	1	control	aalii	2	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	1	control	aweoweo	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	1	control	aweoweo	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	1	control	aweoweo	2	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	1	control	aweoweo	2	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	1	control	mamane	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	1	control	mamane	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	1	control	mamane	2	2	10	2	2.5	1	NA	17	3	3	1	NA
lower	1	control	mamane	2	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	1	control	naenae	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	1	control	naenae	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	1	control	pawale	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	1	control	pawale	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	1	control	pawale	2	2	0	0	0.0	0	NA	0	0	0	0	NA
lower	1	control	pawale	2	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	2	nurse m	mamane	2	2	26	6	5.0	1	NA	27	13	9	1	NA
lower	2	nurse m	mamane	2	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	3	nurse m	naenae	1	2	37	20	12.0	1	lower slope	48	27	18	1	NA
lower	3	nurse m	naenae	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	8	control	aalii	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	8	control	aalii	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	8	control	aalii	2	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	8	control	aalii	2	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	8	control	aweoweo	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	8	control	aweoweo	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	8	control	aweoweo	2	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	8	control	aweoweo	2	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	8	control	mamane	1	2	10	2	2.5	1	nf	23	7	4	1	not found on first measurement
lower	8	control	mamane	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	8	control	mamane	2	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	8	control	mamane	2	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	8	control	naenae	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	8	control	naenae	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	8	control	pawale	1	2	0	0	0.0	0	dead	0	0	0	0	dead
lower	8	control	pawale	1	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	8	control	pawale	2	2	0	0	0.0	0	dead	0	0	0	0	dead
lower	8	control	pawale	2	2	0	0	0.0	0	nf	0	0	0	0	nf
lower	9	nurse k	naenae	1	2	17	10	6.0	1	NA	11	7	5	1	pro
lower	9	nurse k	naenae	1	2	0	0	0.0	0	nf	0	0	0	0	nf
upper	9	nurse upper	pawale	NA	2	0	0	0.0	0	nf	0	0	0	0	nf
upper	9	nurse upper	pawale	NA	2	0	0	0.0	0	nf	0	0	0	0	nf

We have 44 duplicates in the NURSE data. We will need to address these in the data cleaning step. For now, we will just note them.

T1 Measurements

Survivors with no measurement

Let’s check for survivors that have no measurements. This could indicate a problem with the data collection or entry.

# Check for survivors at t1 with no measurements in FOG data
filter_survivors_no_measurement(
  nurse_data,
  survival_col = "survival_t1",
  measurement_cols = c("ht_t1", "w1_t1", "w2_t1"),
  select_cols = c("site", "replicate", "treatment", "species", "plant",
                  "ht_t1", "w1_t1", "w2_t1")) %>%
  kbl() %>%
  kable_minimal(full_width = F)

site	replicate	treatment	species	plant	ht_t1	w1_t1	w2_t1
lower	6	nurse m	aweoweo	1	67	12	0

No Notes for this record at T1. At T2, eaten no foliage. measured stem.

Died but with measurements

Let’s check for plants that died but have measurements. This could indicate a problem with the data collection or entry or could be a valid observation if dead plants were measured.

# Check for plants that died but have measurements in FOG data
filter_non_survivors_with_measurement(
  nurse_data,
  survival_col = "survival_t1",
  measurement_cols = c("ht_t1", "w1_t1", "w2_t1"),
  select_cols = c("site", "replicate", "treatment", "species", "plant",
                  "ht_t1", "w1_t1", "w2_t1", "survival_t1")) %>%
  arrange(site, replicate, treatment, species, plant) %>%
  kbl() %>%
  kable_minimal(full_width = F)

site	replicate	treatment	species	plant	ht_t1	w1_t1	w2_t1	survival_t1
upper	1	control	aalii	2	32.5	11	5	0

Quite a few. We can address these in the data cleaning step. But we need to choose how we want to handle these. We can either remove them or replace the 0 values with NA, replace all the measurements with NA or leave them but then how do we want to deal with them in the analysis. For now, we will just note them. ### T2 Measurements

upper-1-control-aalii-2: No notes.

Survivors with no measurement

# Check for survivors at t2 with no measurements in FOG data
filter_survivors_no_measurement(
  nurse_data,
  survival_col = "survival_t2",
  measurement_cols = c("ht_t2", "w1_t2", "w2_t2"),
  select_cols = c("site", "replicate", "treatment", "species", "plant",
                  "ht_t2", "w1_t2", "w2_t2")) %>%
  kbl() %>%
  kable_minimal(full_width = F)

site	replicate	treatment	species	plant	ht_t2	w1_t2	w2_t2

None! that’s great! We can move on to the next step.

Died but with measurements

Let’s check for plants that died but have measurements. This could indicate a problem with the data collection or entry or could be a valid observation if dead plants were measured.

# Check for plants that died but have measurements in FOG data
# Check for plants that died but have measurements in FOG data
filter_non_survivors_with_measurement(
  nurse_data,
  survival_col = "survival_t2",
  measurement_cols = c("ht_t2", "w1_t2", "w2_t2"),
  select_cols = c("site", "replicate", "treatment", "species", "plant",
                  "ht_t2", "w1_t2", "w2_t2", "survival_t2")) %>%
  arrange(site, replicate, treatment, species, plant) %>%
  kbl() %>%
  kable_minimal(full_width = F)

site	replicate	treatment	species	plant	ht_t2	w1_t2	w2_t2	survival_t2
lower	6	nurse m	aweoweo	1	48	0	0	0

Quite a few. We can address these in the data cleaning step. But we need to choose how we want to handle these. We can either remove them or replace the 0 values with NA, replace all the measurements with NA or leave them but then how do we want to deal with them in the analysis. For now, we will just note them.

lower-6-nurse m-aweoweo-plant1: No notes.

Lazurus Plants

Let’s check for plants that were dead in T1 but were alive in T2. This could indicate a problem with the data collection or entry or could be a valid observation if plants were added between T1 and T2.

# Check for lazarus plants in FOG data
lazarus_plants_fog <- nurse_data %>%
  filter(survival_t1 == 0 & survival_t2 == 1) %>%
  select("site", "replicate", "treatment", "species", "plant")

lazarus_plants_fog %>%
  kbl() %>%
  kable_minimal(full_width = F)

site	replicate	treatment	species	plant
upper	10	nurse upper	pawale	1
upper	9	nurse upper	aalii	1

We have two Lazarus plants in the Nurse data.

upper-10-nurse upper-pawale-plant1: No notes.

upper-9-nurse upper-aalii-plant1: Note: not found on measurement 1.