Glue Problem

The task

Which is two while loops, one nested in the other. This does not work, for a couple of reasons, and I would like you to take a look at it and see what you think about:

The design of a double-nested while loop

• How information flows around the function
• Any key errors you can see
• Any ideas you have for improving the function

Original Code:

glue_func <- function(dir = "", outfolder = "", sitelist){
  
  # dir is the input directory for the files
  
  # outfolder should be directory to put the new files
  
  # sitelist is a list of the Site names
  
  
  length(sitelist) -> site_n
  
  site_counter = 0
  
  
  while(site_counter < site_n){
    
    site_interest <- sitelist[site_counter+1]
    
    
    filepath <- paste0(dir, "\\", site_interest)
    
    
    file_list <- list.files(filepath, pattern = ".csv", full.names = TRUE)
    
    # List out all of the files in that directory
    
    n_files <- length(file_list) # Tells me how many files there are
    
    
    output <- read.csv(file_list[1])
    
    file_counter = 1
    
    
    while(file_counter < n_files){
      
      to_bind <- read.csv(file_list[file_counter+1])
      
      output <<- rbind(output, to_bind)
      
      file_counter = file_counter + 1
      
    }
    
    
    
    outfule <- paste0(outfolder, site_interest, ".csv")
    
    write.csv(output, outfile, row.names = F)
    
    message("Site Complete")
    
    
    site_counter = site_counter + 1
    
  }
  
  
  message("All Sites Complete")
  
}

Improvement suggestions

Notes from looking at code:

Fix typos

# outfule <- paste0(outfolder, site_interest, ".csv")
outfile <- paste0(outfolder, site_interest, ".csv")

Assignment direction

# length(sitelist) -> site_n
site_n <- length(sitelist)

Notes from looking resources online:

R Indices - start with 1 not 0 so could cause problem?

  site_counter = 0
  
  while(site_counter < site_n){
    
    site_interest <- sitelist[site_counter+1]

File path - for making more standard use this function?

file.path(dir, site_interest)

Suggestion in Advanced R book, to use least flexible solution to a problem: Replacing nested loops with lapply and for loops (for is less flexible than while)

Avoid the global assignment operator <<- . Is it possible to assign locally instead, otherwise it might overwrite outside function?

Any way to handle wrong/empty directories or CSVs?

Suggested actions, what I would do next:

1. investigate further how to use for instead of while loops and write try code for this
2. research how to use lapply() for this code and try
3. explore further binding and rbind()

Further comments

Non-vectorized approach

# Read files one by one and combine them
output <- read.csv(file_list[1])
file_counter = 1

while(file_counter < n_files){
  to_bind <- read.csv(file_list[file_counter+1])
  output <- rbind(output, to_bind)  # This creates a NEW data frame each time!
  file_counter = file_counter + 1
}

Problems:

Each rbind() creates a completely new data frame R has to copy ALL the existing data plus the new data With 10 files, you create 9 unnecessary intermediate data frames

Vectorized approach:

# Read ALL files at once, then combine ALL at once
all_data <- lapply(file_list, read.csv)  # Apply read.csv to each file
output <- do.call(rbind, all_data)       # Combine everything in one operation

Current work 21.8.25

• Explore if while loop can be optimised / fixed.
• Maybe substitute for loop, do we know exactly how many iterations? nsites
• also explore vectorized approach (using lapply instead of loops) and try on test data. compare both data.

Suggested Code (work in progress)

glue_func_fixed <- function(dir = "", outfolder = "", sitelist){
  
  # dir is the input directory for the files
  # outfolder should be directory to put the new files
  # sitelist is a list of the Site names
  
  length(sitelist) -> site_n
  site_counter = 0
  
  while(site_counter < site_n){
    
    site_interest <- sitelist[site_counter+1]
    
    # change 1: Use file.path() instead of paste0 with \\
    # This works on Windows, Mac, and Linux
    filepath <- file.path(dir, site_interest)  # CHANGED from paste0(dir, "\\", site_interest)
    
    file_list <- list.files(filepath, pattern = ".csv", full.names = TRUE)
    
    # change 2: Check if we found any files (avoid crashes) - needed? unsure
    if(length(file_list) == 0) {
      warning("No CSV files found for site: ", site_interest)
      site_counter = site_counter + 1  # increment
      next  # Skip to next site
    }
    
    # List out all of the files in that directory
    n_files <- length(file_list) # Tells me how many files there are
    
    output <- read.csv(file_list[1])
    file_counter = 1
    
    while(file_counter < n_files){
      
      to_bind <- read.csv(file_list[file_counter+1])
      
      # change 3: Remove <<- operator
      output <- rbind(output, to_bind)  # CHANGED from output <<- rbind(output, to_bind)
      
      file_counter = file_counter + 1
      
    }
    
    # change 4: Fix the typo - outfule should be outfile
    outfile <- paste0(outfolder, site_interest, ".csv")  # CHANGED from outfule
    
    write.csv(output, outfile, row.names = F)
    
    message("Site Complete: ", site_interest)  # Added site name for clarity
    
    site_counter = site_counter + 1
    
  }
  
  message("All Sites Complete")
  
}