SOP for instrument data processing using the R package occ
revised May 2019
‘occ’ package installation
The occ package is uploaded to GitHub and can be downloaded directly in R.
- Open the devtools library.
library(devtools)- To download the occ package, use the
install_githubcommand. The repository name is in the form “username/repo”. For the occ package, the username is “hannahbarkley” and the repo is “occ”. This install only need to occur once; however, the occ package will need to be reinstalled if there are updates to the package (which is likely).
install_github("hannahbarkley/occ")- Once installed, load the occ package.
library(occ)CTD profiles - RBR
Process in R
RBR CTD data are downloaded in Ruskin as a RSK file containing multiple profiles within a single time series.
- Specify the path to the downloaded RSK file.
ctd.file = "T:/Oceanography/R code/occ_SOP/201280_20190522_1626.rsk"- Specify the CTD cast ID and starting cast number. The cast ID should contain the cruise, region, island, and CTD serial number and should match the ShallowCTDID recorded for the cast in the CTD database. The starting cast number is the cast number that will be assigned to the first cast in the time series, and subsequent cast numbers will be assigned sequentially from this number.
CastID = "HA1801_AmSamoa_TUT_101280"
CastNum = 1- Run the
processCTD_RBRfunction to separate, process, and save individual CTD profiles. Specify the ctd file path (ctd.file), the CTD cast ID (CastID), and the starting cast number (CastNum).
processCTD_RBR(ctd.file, CastID, CastNum)- Once processing is complete, the function will output a table of the individual profiles identified in the time series file, their assigned ShallowCTDID (combined cast ID and cast number), and the UTC start time of the cast. Use this table to compare with the field data sheet and make sure that all casts are accounted for. This table will also be saved as a CSV file.
## ShallowCTDID UTCTimeStart
## 1 HA1801_AmSamoa_TUT_101280_001 2019-05-15 20:18:40
## 2 HA1801_AmSamoa_TUT_101280_002 2019-05-15 21:23:42
## 3 HA1801_AmSamoa_TUT_101280_003 2019-05-15 23:02:30
## 4 HA1801_AmSamoa_TUT_101280_004 2019-05-17 19:42:03
## 5 HA1801_AmSamoa_TUT_101280_005 2019-05-17 20:43:48The processCTD_RBR function will also save a CSV file and PNG image of each profile.
head(read.csv("T:/Oceanography/R code/occ_SOP/HA1801_AmSamoa_TUT_101280_001.csv"))## ShallowCTDID Pressure_db Depth_m Temp_DegC Cond_S_per_m
## 1 HA1801_AmSamoa_TUT_101280_001 1.007971 1 26.44982 5.390411
## 2 HA1801_AmSamoa_TUT_101280_001 2.016216 2 26.44780 5.390265
## 3 HA1801_AmSamoa_TUT_101280_001 3.024462 3 26.44003 5.389381
## 4 HA1801_AmSamoa_TUT_101280_001 4.032707 4 26.43059 5.388332
## 5 HA1801_AmSamoa_TUT_101280_001 5.040953 5 26.42749 5.388173
## 6 HA1801_AmSamoa_TUT_101280_001 6.049198 6 26.40427 5.385307
## Saln_PSU Density_Sigmat
## 1 34.50446 22.52312
## 2 34.50465 22.52824
## 3 34.50379 22.53437
## 4 34.50300 22.54108
## 5 34.50386 22.54704
## 6 34.50030 22.55599
CTD profiles - SBE
Process in SBE Data Processing
Profiles are downloaded from the CTD as individual files in HEX format. These HEX files need to be converted to CNV in the SeaTerm software using the appropriate configuration files. This processing step must be run separately for each CTD.
In the SBE Data Processing program, select “Run” > “1. Data Conversion” to open the Data Conversion window.
In the “File Setup” tab: if you have a setup file already created, open it under “Program setup file” (.psa). Here, we are using “DataCnv_profile.psa”, which already has the correct parameters and variables saved for CTD profiles. If this file doesn’t exist, we’ll create it in the next few steps.
At the “Instrument configuration file”, section choose the correct configuration file (.xmlcon).
In the “Input Directory” section, select the CTD casts to process. The processing will not run if the configuration file and CTD casts are not from the same CTD.
- In the “Data Setup” tab, make sure:
- “Process scans to end of file” is checked
- “Scans to skip over” is 0
- “Output format” is “ASCII output”
- “Convert data from” is “Downcast”
- “Create file types” is “Create converted data (.CNV) file only”
- Click the “Select Output Variables…” button to select which variables are exported. Make sure the variables match the image below.
Click “OK” to exit, and then “Start Process” to run the data conversion. Each HEX file should now be converted to CNV.
Save the program steup file when prompted upon exiting if it hasn’t already been created. Open this setup file next time you process CTD profiles.
Process in R
- Specify the path to the folder containing all of the CNV CTD profiles.
profiles = "T:/Cruise/CruiseData/HA1801_AmSamoa/Project Group/OceanClimateChange/CTD Profiles/Shallow/TUT/"- In this example, the path to the folder is called by the variable name profiles. Check the contents of the profiles folder using the
list.filesfunction. Here we’re looking for all CNV files by specifying the pattern.
list.files(profiles, pattern = "*.cnv")## [1] "HA1801_AmSamoa_TUT_SN4282_001.cnv"
## [2] "HA1801_AmSamoa_TUT_SN4282_002.cnv"
## [3] "HA1801_AmSamoa_TUT_SN4282_003.cnv"
## [4] "HA1801_AmSamoa_TUT_SN4282_004.cnv"
## [5] "HA1801_AmSamoa_TUT_SN4282_005.cnv"
## [6] "HA1801_AmSamoa_TUT_SN4282_006.cnv"
## [7] "HA1801_AmSamoa_TUT_SN4282_007.cnv"
## [8] "HA1801_AmSamoa_TUT_SN4282_008.cnv"
## [9] "HA1801_AmSamoa_TUT_SN4282_009.cnv"
## [10] "HA1801_AmSamoa_TUT_SN4282_010.cnv"
## [11] "HA1801_AmSamoa_TUT_SN4282_011.cnv"
## [12] "HA1801_AmSamoa_TUT_SN4282_012.cnv"
## [13] "HA1801_AmSamoa_TUT_SN4282_013.cnv"
## [14] "HA1801_AmSamoa_TUT_SN4282_014.cnv"
## [15] "HA1801_AmSamoa_TUT_SN4282_015.cnv"
## [16] "HA1801_AmSamoa_TUT_SN4282_016.cnv"
## [17] "HA1801_AmSamoa_TUT_SN4282_017.cnv"
## [18] "HA1801_AmSamoa_TUT_SN4282_018.cnv"
## [19] "HA1801_AmSamoa_TUT_SN4282_019.cnv"
## [20] "HA1801_AmSamoa_TUT_SN4282_021.cnv"
## [21] "HA1801_AmSamoa_TUT_SN4282_022.cnv"
## [22] "HA1801_AmSamoa_TUT_SN4818_001.cnv"
## [23] "HA1801_AmSamoa_TUT_SN4818_002.cnv"
## [24] "HA1801_AmSamoa_TUT_SN4818_003.cnv"
## [25] "HA1801_AmSamoa_TUT_SN4818_004.cnv"
## [26] "HA1801_AmSamoa_TUT_SN4818_005.cnv"
## [27] "HA1801_AmSamoa_TUT_SN4818_006.cnv"
## [28] "HA1801_AmSamoa_TUT_SN4818_007.cnv"
## [29] "HA1801_AmSamoa_TUT_SN4818_008.cnv"- Process all CNV files in the folder using the
processCTD_profilefunction. This function will batch process each CNV file in the folder and convert to CSV.
processCTD_profile(profiles)## [1] "1 of 29 casts processed"
## [1] "2 of 29 casts processed"
## [1] "3 of 29 casts processed"
## [1] "4 of 29 casts processed"
## [1] "5 of 29 casts processed"
## [1] "6 of 29 casts processed"
## [1] "7 of 29 casts processed"
## [1] "8 of 29 casts processed"
## [1] "9 of 29 casts processed"
## [1] "10 of 29 casts processed"
## [1] "11 of 29 casts processed"
## [1] "12 of 29 casts processed"
## [1] "13 of 29 casts processed"
## [1] "14 of 29 casts processed"
## [1] "15 of 29 casts processed"
## [1] "16 of 29 casts processed"
## [1] "17 of 29 casts processed"
## [1] "18 of 29 casts processed"
## [1] "19 of 29 casts processed"
## [1] "20 of 29 casts processed"
## [1] "21 of 29 casts processed"
## [1] "22 of 29 casts processed"
## [1] "23 of 29 casts processed"
## [1] "24 of 29 casts processed"
## [1] "25 of 29 casts processed"
## [1] "26 of 29 casts processed"
## [1] "27 of 29 casts processed"
## [1] "28 of 29 casts processed"
## [1] "29 of 29 casts processed"- Check the profiles folder or use the
list.filesfunction to make sure that CSV files were created. It’s good practice to open the files to make sure that they include data.
list.files(profiles, pattern = "*.csv")## [1] "HA1801_AmSamoa_TUT_SN4282_001.csv"
## [2] "HA1801_AmSamoa_TUT_SN4282_002.csv"
## [3] "HA1801_AmSamoa_TUT_SN4282_003.csv"
## [4] "HA1801_AmSamoa_TUT_SN4282_004.csv"
## [5] "HA1801_AmSamoa_TUT_SN4282_005.csv"
## [6] "HA1801_AmSamoa_TUT_SN4282_006.csv"
## [7] "HA1801_AmSamoa_TUT_SN4282_007.csv"
## [8] "HA1801_AmSamoa_TUT_SN4282_008.csv"
## [9] "HA1801_AmSamoa_TUT_SN4282_009.csv"
## [10] "HA1801_AmSamoa_TUT_SN4282_010.csv"
## [11] "HA1801_AmSamoa_TUT_SN4282_011.csv"
## [12] "HA1801_AmSamoa_TUT_SN4282_012.csv"
## [13] "HA1801_AmSamoa_TUT_SN4282_013.csv"
## [14] "HA1801_AmSamoa_TUT_SN4282_014.csv"
## [15] "HA1801_AmSamoa_TUT_SN4282_015.csv"
## [16] "HA1801_AmSamoa_TUT_SN4282_016.csv"
## [17] "HA1801_AmSamoa_TUT_SN4282_017.csv"
## [18] "HA1801_AmSamoa_TUT_SN4282_018.csv"
## [19] "HA1801_AmSamoa_TUT_SN4282_019.csv"
## [20] "HA1801_AmSamoa_TUT_SN4282_021.csv"
## [21] "HA1801_AmSamoa_TUT_SN4282_022.csv"
## [22] "HA1801_AmSamoa_TUT_SN4818_001.csv"
## [23] "HA1801_AmSamoa_TUT_SN4818_002.csv"
## [24] "HA1801_AmSamoa_TUT_SN4818_003.csv"
## [25] "HA1801_AmSamoa_TUT_SN4818_004.csv"
## [26] "HA1801_AmSamoa_TUT_SN4818_005.csv"
## [27] "HA1801_AmSamoa_TUT_SN4818_006.csv"
## [28] "HA1801_AmSamoa_TUT_SN4818_007.csv"
## [29] "HA1801_AmSamoa_TUT_SN4818_008.csv"- Here’s what the top of each CSV file should look like (this code is just a quick way of opening the first CSV file in the folder and reading the first few lines). This is the correct format for ingestion into the CTD database.
head(read.csv(paste0(profiles,list.files(profiles, pattern = "*.csv")[1])))## ShallowCTDID Pressure_db Depth_m Temp_DegC Cond_S_per_m
## 1 HA1801_AmSamoa_TUT_SN4282_001 1.006183 1 28.73536 5.660754
## 2 HA1801_AmSamoa_TUT_SN4282_001 2.012354 2 28.70631 5.663484
## 3 HA1801_AmSamoa_TUT_SN4282_001 3.018525 3 28.82981 5.688413
## 4 HA1801_AmSamoa_TUT_SN4282_001 4.024696 4 28.82829 5.688306
## 5 HA1801_AmSamoa_TUT_SN4282_001 5.030868 5 28.82872 5.688810
## 6 HA1801_AmSamoa_TUT_SN4282_001 6.037039 6 28.82857 5.688552
## Saln_PSU Density_Sigmat
## 1 34.72248 21.94814
## 2 34.76233 21.99202
## 3 34.84387 22.01652
## 4 34.84396 22.02140
## 5 34.84683 22.02772
## 6 34.84489 22.03062At the end of the cruise, all CTD profiles from each region will need to be merged into a single spreadsheet for ingestion into the database.
- Use the ‘mergeCTD’ function, specifying the parent directory dir with all of the casts to be grouped, the name of the cruise (e.g. HA1801), and the group (e.g. AmSamoa). Don’t forget any shipboard casts!
dir = 'T:/Cruise/CruiseData/HA1801_AmSamoa/Project Group/OceanClimateChange/CTD Profiles/'
cruise = 'HA1801'
region = 'AmSamoa'
mergeCTD(dir,cruise,region)## [1] "Casts from cruise HA1801 and region AmSamoa merged in T:/Cruise/CruiseData/HA1801_AmSamoa/Project Group/OceanClimateChange/CTD Profiles/"STR time series
Process in Seabird V2
The beginning and end of all STR time series need to be trimmed to remove any periods when the instrument was above water.
- After downloading the time series from the logger, export the data file to CSV format in the Seabird V2 software. To do so, select the “Export” button in the bottom right corner.
- An “Export Data” window will appear. Make sure:
- File type is “.csv”
- Desired file is selected in the “Upload Data File” section
Date format is “yyyy-mm-dd”
Then click “Export”.
Process in R
- Specify the path to the raw STR CSV file.
str.file = "T:/Cruise/CruiseData/HA1801_AmSamoa/Project Group/OceanClimateChange/Moorings/TUT/SBE05604724_2018-07-02.csv"- Specify the path to a metadata CSV file for all STRs on a cruise. The exact form and contents of this spreadsheet are not standard. However, the spreadsheet must contain serial numbers (InstrumentSerialNum) for each instrument, and ideally transect IDs (TransectID), site IDs (SiteID), and depths (Depth_ft) as well. Including this information will ensure that the instrument serial number in the data file name matches the serial number in the database, and the processing script will autopopulate values for the rest of the metadata fields. It is not critical that this metadata file be included, but it is highly recommended. Note: In the example below, transect and site IDs do not conform to the new naming convention established for 2019.
meta.file = "T:/Oceanography/R code/occ_SOP/ASRAMP2018_STR_metadata.csv"
head(read.csv(meta.file), 3)## InstrumentSerialNum TransectID SiteID Depth_ft
## 1 5604726 OFU-06 OFU_OCEAN_011 83
## 2 5604720 OFU-06 OFU_OCEAN_012 49
## 3 5604729 OFU-09 OFU_OCEAN_015 49- Run the
processSTRfunction to trim the time series. TheprocessSTRfunction will open a Shiny app in a new window. Specify the STR data file path str.file, the metadata file path meta.file, and the instrument type model. It may take a minute to load if the file is large.
processSTR(str.file, model = "SBE", meta.file)
- Enter the serial number for the instrument time series you are processing in the popup window. If you have uploaded a metadata file, the pull-down menu will automatically populate with all of the possible STRs recovered. Click the “OK” button when ready.
- Once the serial number is entered, a second window will appear. Enter the transect ID, site ID, depth (in feet), and date retrieved for the instrument. If using a metadata spreadsheet, these values should automatically populate each field as the default. Check that they are correct. The default date retrieved is the current date. Click the “Trim STR data” button to proceed and the popup window will disappear.
8. The new filename for the data set will now appear at the top of the window. You can change the serial number of metadata by clicking the “Change STR serial number” or “Change STR metadata” buttons.
9. Check the entire raw time series, displayed at the top, to make sure that it looks ok and that there are no issues beyond bad data at the beginning and end.
- There are two figures beneath the entire time series. The left figure, “Select START point” shows the first month on the time series. The right figure, “Select END point”, shows the last month. Use the slider at the bottom of each figure to adjust the window if this view is not good.
- To select start and end dates, click directly on the time series at the appropriate points. Once the app registers the click, it will display the selected date and time beneath each figure.
- Once both start and end dates have been selected, the trimmed time series will be displayed at the bottom. Make sure it looks good. If the selected start and end dates are not right, keep re-clicking on the start and end figures until correct.
- Once the trimmed time series looks good, click the “SAVE” button. A “File saved” message will appear when the save is successful.
- This app will save the trimmed time series as a CDP file (for the database) and a CSV file (better for further processing in R) using the filename specified at the top of the window. A PNG image of the time series figure will also be saved, shown below.
- Click the “Stop app” button to close the app or hit the escape key in R. The app must be closed in order to continue to use R.
Diel suite
Moored CTD
Process in SBE Data Processing
Moored CTD time series are downloaded from the CTD in HEX format. These HEX files need to be converted to CNV in the SeaTerm software using the appropriate configuration files.
In the SBE Data Processing program, select “Run” > “1. Data Conversion” to open the Data Conversion window.
In the “File Setup” tab: if you have a setup file already created, open it under “Program setup file” (.psa). Here, we are using “DataCnv_moored.psa”, which already has the correct parameters and variables saved for a moored CTD time series. If this file doesn’t exist, we’ll create it in the next few steps.
Under the “Instrument configuration file” section, choose the correct configuration file (.xmlcon).
In the “Input Directory” section, select the CTD time series to process. The configuration file and data file must be from the same CTD.
- The moored CTD time series requires a different configuration file than profiles. Once a configuration file is selected, click “Modify” to view. Make sure than the “Mode” is “Moored”.
- In the “Data Setup” tab, make sure:
- “Process scans to end of file” is checked
- “Scans to skip over” is 0
- “Output format” is “ASCII output”
- “Convert data from” is “Upcast and downcast”
- “Create file types” is “Create converted data (.CNV) file only”
- Click the “Select Output Variables…” button to select which variables are exported. Make sure the variables in the left panel match the image below (only include oxygen if the SBE-43 was included in the deployment package). Also, make sure that a time variable is included, although the name of the time variable may differ from what’s below based on the CTD. Here, the time variable is “Time, Elapsed [seconds]”.
8. Click “OK” to exit, and then “Start Process” to run the data conversion. Each HEX file should now be converted to CNV.
9. Save the program steup file when prompted upon exiting if it hasn’t already been created. Open this setup file next time you process moored CTD data.
Process in R
- Specify the moored CTD CNV file.
moored.file = "T:/Cruise/CruiseData/HA1801_AmSamoa/Project Group/OceanClimateChange/Diurnal Suites/TUT-ARMS-74/CTD-Moored/HA1801_AmSamoa_TUT_7744_DS001.cnv"- Run the
processCTD_mooredfunction. Ignore “unrecognized SBE name” warning if it appears (this just means that the function that loads the CTD data doesn’t recognize the name of the time variable - the function knows how to deal with this). Optional arguments to the function include tz.in (the time zone of deployment, if not UTC), tz.out (the desired output time zone, if not UTC), write.csv (specify FALSE to suppress CSV output), and SN3029 (specify TRUE if using CTD SN 3029; this is for historical data, SN 3029 should not be used for diel suite). These only need to be specified if you don’t want the default.
mooredctd = processCTD_moored(moored.file) - The
processCTD_mooredfunction produces one data frame in R that is added to the workspace as mooredctd. The function also saves two CSV spreadsheets. The first (“_processed.csv“) is the processed moored CTD time series. The second (”_SEAFET.csv“) is specifically formatted to temperature- and salinity-correct the SeaFET pH time series (instructions below).
head(mooredctd)## ShallowCTDID UTCDateTime Pressure_db Depth_m
## 1 HA1801_AmSamoa_TUT_7744_DS001 2018-06-23 23:00:01 15.124 15.031
## 2 HA1801_AmSamoa_TUT_7744_DS001 2018-06-23 23:05:00 14.994 14.902
## 3 HA1801_AmSamoa_TUT_7744_DS001 2018-06-23 23:10:00 15.115 15.022
## 4 HA1801_AmSamoa_TUT_7744_DS001 2018-06-23 23:15:01 15.012 14.920
## 5 HA1801_AmSamoa_TUT_7744_DS001 2018-06-23 23:20:01 15.113 15.020
## 6 HA1801_AmSamoa_TUT_7744_DS001 2018-06-23 23:25:00 15.260 15.166
## Temp_DegC Saln_PSU Density_Sigmat Cond_S_per_m Oxygen_mgL
## 1 29.1216 34.7826 21.8602 5.710930 5.2095
## 2 29.1164 34.7828 21.8621 5.710407 5.1963
## 3 29.1372 34.7957 21.8648 5.714493 5.2499
## 4 29.1480 34.8093 21.8714 5.717616 5.2055
## 5 29.1517 34.8115 21.8718 5.718336 5.2526
## 6 29.1495 34.8157 21.8757 5.718723 5.2221ADCP
Process in R
Raw current data is downloaded from the ADCP as a PRF file.
- To process, first specify the PRF file.
adcp.file = "T:/Cruise/CruiseData/HA1801_AmSamoa/Project Group/OceanClimateChange/Diurnal Suites/TUT-ARMS-74/ADCP/HA1801_AmSamoa_TUT_ADCP.prf"- The
processADCPfunction takes the raw ADCP data and converts to speed and direction. The function can output the data in long (every observation is its own row) or wide (each height bin is a column) form. Choose long form for working in R. Optional arguments to the function include tz.in (the time zone of deployment, if not UTC) and write.csv (specify FALSE to suppress CSV output). These only need to be specified if you don’t want the default.
adcp = processADCP(adcp.file, format = "long")
head(adcp)## UTCDateTime HeightBin_m Speed_ms Direction_deg
## 1 2018-06-23 23:00:00 1.41 0.08896629 225.9240
## 6879 2018-06-23 23:00:00 2.41 0.07701299 244.4400
## 13757 2018-06-23 23:00:00 3.41 0.13644413 210.7453
## 20635 2018-06-23 23:00:00 4.41 0.09702577 207.3874
## 27513 2018-06-23 23:00:00 5.41 0.15269250 232.4543
## 34391 2018-06-23 23:00:00 6.41 0.06981404 246.0375- In long form, the
processADCPfunction saves two CSV files: “_speeddirection.csv" and “_pressure.csv“. Only the speeddirection data frame is stored within the variable adcp in the workspace.
head(adcp)## UTCDateTime HeightBin_m Speed_ms Direction_deg
## 1 2018-06-23 23:00:00 1.41 0.08896629 225.9240
## 6879 2018-06-23 23:00:00 2.41 0.07701299 244.4400
## 13757 2018-06-23 23:00:00 3.41 0.13644413 210.7453
## 20635 2018-06-23 23:00:00 4.41 0.09702577 207.3874
## 27513 2018-06-23 23:00:00 5.41 0.15269250 232.4543
## 34391 2018-06-23 23:00:00 6.41 0.06981404 246.0375- The wide form of the ADCP data set creates a separate CSV file for speed, direction, and pressure. Only speed and direction are saved to the workspace (we prefer to use pressure data from the CTD), and can be accessed as speed and direction components of the variable adcp with rows as each time step and column names as the depth bins.
adcp.wide = processADCP(adcp.file, format = "wide")
head(adcp.wide$speed) ## UTCDateTime 1.41 2.41 3.41 4.41
## 1 2018-06-23 23:00:00 0.08896629 0.07701299 0.13644413 0.09702577
## 2 2018-06-23 23:05:00 0.07167984 0.08324062 0.07905694 0.08108021
## 3 2018-06-23 23:10:00 0.09917661 0.05266878 0.04549725 0.08449852
## 4 2018-06-23 23:15:00 0.06447480 0.02733130 0.03536948 0.09849365
## 5 2018-06-23 23:20:00 0.07323933 0.05679789 0.05202884 0.03863936
## 6 2018-06-23 23:25:00 0.05088222 0.04364631 0.08888757 0.07682448
## 5.41 6.41 7.41 8.41 9.41 10.41
## 1 0.15269250 0.06981404 0.08949860 0.13421624 0.13029582 0.19725364
## 2 0.09064767 0.10303883 0.05081338 0.08794316 0.11057577 0.10959015
## 3 0.05793962 0.06206448 0.08495293 0.10852649 0.09982986 0.09036039
## 4 0.08300000 0.02714774 0.08895504 0.05049752 0.13082049 0.14039231
## 5 0.07144928 0.09454628 0.08161495 0.11909660 0.07284230 0.12236421
## 6 0.07529276 0.08295179 0.10926573 0.14047776 0.15172673 0.14258682
## 11.41 12.41 13.41 14.41 15.41
## 1 0.17829470 0.06074537 0.3382987 0.7648111 NA
## 2 0.11674331 0.10673800 0.3605066 0.6920094 NA
## 3 0.08611620 0.10945775 0.3778267 0.7309795 NA
## 4 0.07487323 0.11009541 0.3289757 0.6816282 NA
## 5 0.12342204 0.13303759 0.3308610 0.7067524 NA
## 6 0.13624977 0.11194642 0.3294936 0.6216341 NAhead(adcp.wide$direction)## UTCDateTime 1.41 2.41 3.41 4.41 5.41
## 1 2018-06-23 23:00:00 225.9240 244.4400 210.7453 207.3874 232.4543
## 2 2018-06-23 23:05:00 210.9638 209.8355 235.3048 234.5445 225.0000
## 3 2018-06-23 23:10:00 225.0000 243.9246 304.9920 267.2073 248.7495
## 4 2018-06-23 23:15:00 283.1340 217.4054 322.1250 272.3373 295.2405
## 5 2018-06-23 23:20:00 265.2364 273.0665 280.0080 291.2505 234.9406
## 6 2018-06-23 23:25:00 220.1009 246.9745 230.9374 213.6901 189.3350
## 6.41 7.41 8.41 9.41 10.41 11.41 12.41 13.41
## 1 246.0375 251.7675 230.1398 212.7087 222.7259 221.7948 220.4860 240.0240
## 2 190.7580 220.9717 188.9017 223.5121 204.8874 223.2643 202.8906 242.5820
## 3 235.7843 191.1738 210.8721 217.6736 225.4511 215.5377 210.6852 238.5921
## 4 289.1790 224.5416 178.8542 225.3097 221.5318 232.0578 218.9910 246.7099
## 5 248.7820 227.4896 226.3639 198.4349 205.3014 213.4326 171.8699 256.7198
## 6 190.6197 220.5444 229.6199 206.9061 219.8572 227.3859 230.1944 257.0184
## 14.41 15.41
## 1 233.3998 NA
## 2 231.1592 NA
## 3 229.8833 NA
## 4 235.0975 NA
## 5 232.0112 NA
## 6 230.8793 NASeaFET
Process in SeaFET software
SeaFET raw data is downloaded from the instrument as a CSV. Before processing in R, the raw pH data need to be salinity and temperature-corrected in the SeaFET software using the CTD time series.
In the “SeaFET Data Processing Dashboard” pane, select the SeaFET data file in the “SeaFET FULL ASCII Data Files” section.
In the “Processing Options”, make sure “Enable Raw Data Checksum Validation” is checked and “Coefficients from SeaFET Data File Header” is selected.
In the “Specify Temperature Salinity Data” section, choose the CTD file under “Temperature-Salinity External File”. The correct file is the moored CTD CSV produced above with “_SEAFET" at the end of the file name. In “Temperature Options”, select “Temperature from External File”. In the “Salinity Options”, select “Salinity from External File”.
4. Click the “Process Selected File(s)” button. This may take several minutes depending on file size, but after processing is complete the new time series should appear in the “Processed pH Data Viewer pane”. The processed file will be saved in the same folder as the raw data with “_pro" appended to the file name.
Process in R
- The SeaFET data are then reformatted using the
processSEAFETfunction. First, define the path to the CSV file.
seafet.file = "T:/Cruise/CruiseData/HA1801_AmSamoa/Project Group/OceanClimateChange/Diurnal Suites/TUT-ARMS-74/SeaFET/HA1801_AmSamoa_TUT_SeaFET_pro.csv"- Run the
processSEAFETfunction. Optional arguments to the function include tz.in (the time zone of deployment, if not UTC), tz.out (the desired output time zone, if not UTC), average (specify TRUE if the data were collected in bursts and need to be averaged in one-minute bins, default is FALSE) and write.csv (specify FALSE to suppress CSV output). These only need to be specified if you don’t want the default. It’s possible to run theprocessSEAFETfunction on an uncorrected SeaFET data set by adding the optional arg “corrected = FALSE”.
seafet = processSEAFET(seafet.file)- The processed data will be added to the workspace as the variable seafet and saved as a CSV file in the same location as the raw file. The processed dataframe will look like this:
head(seafet)## # A tibble: 6 x 2
## UTCDateTime pH
## <dttm> <dbl>
## 1 2018-02-08 23:36:00 5.82
## 2 2018-02-08 23:38:00 5.82
## 3 2018-02-08 23:40:00 5.82
## 4 2018-02-08 23:42:00 5.82
## 5 2018-02-08 23:44:00 5.82
## 6 2018-02-08 23:46:00 5.82PUC samples
Process in R
PUC water samples for TA and DIC are processed by PMEL. However, we need to provide temperature, salinity, and pressure data for each sample to PMEL prior to TA and DIC analysis. These data come from from the moored CTD time series.
The function processPUC uses the time stamp of the start of the PUC sample to pair to CTD data averaged over a 45-minute window (assuming the PUCs take 45 minutes to fill).
Specify the CSV file that contains PUC sample metadata (puc.file) and the moored CTD CSV file already processed using the processCTD_moored function (ctd.file). The PUC metadata spreadsheet must include a column called “DateTimeDeploy” that has a concatenated deployment date/time for each sample time.
Optional arguments to the processPUC function include tz.in (the time zone of “DateTimeDeploy”, if not UTC), tz.out (the desired output time zone, if not UTC), and write.csv (specify FALSE to suppress CSV output). These only need to be specified if you don’t want the default.
This will save as a CSV file in the same folder as the original spreadsheet with “_processed" appended to the filename. Make sure that the temperature, salinity, and pressure columns are now populated with data.
puc.file = "T:/Oceanography/R code/occ_SOP/HA1801_TUT74_PUCs.csv"
ctd.file = "T:/Cruise/CruiseData/HA1801_AmSamoa/Project Group/OceanClimateChange/Diurnal Suites/TUT-ARMS-74/CTD-Moored/HA1801_AmSamoa_TUT_7744_DS001_processed.csv"puc.samples = processPUC(puc.file, ctd.file)Trim all diel suite time series
Once all of the individual instrument data have been processed, the next step is to trim all of the time series so that the out of water signal is removed. The is done using the trimDS function. This function looks at the pressure and heading data collected by the ADCP to establish when the instrument package was deployed and recovered and then uses these time stamps to trim the CTD and SeaFET data.
To run the trimDS function, specify the ADCP, SeaFET, and CTD data files (CSV) or variable names. Following the workflow above, these variables should already exist in the workspace and can be called directly in the function (specify read.csv = FALSE). If the variables aren’t stored, define the path to each CSV file and specify read.csv = TRUE. Specify the output directory output.dir where the trimmed data files should be saved, and the filename to use in the form YYYY_SiteID_diel.
output.dir = "T:/Cruise/CruiseData/HA1801_AmSamoa/Project Group/OceanClimateChange/Diurnal Suites/TUT-ARMS-74/Processed CSV/"
trim = trimDS(
adcp = adcp,
seafet = seafet,
ctd = mooredctd,
output.dir = output.dir,
read.csv = FALSE,
filename = '2018_TUT74_diel'
)Thr trimDS creates a ADCP, SeaFET, and CTD CSV files. These data sets are also stored in the workspace as adcp, seafet, and ctd within the variable trim.
Create a combined diel suite plot
Finally, plot the entire diel suite together using the plotDS function (not including PUC data because TA/DIC values aren’t available immediately post cruise).
Again, following the workflow above, these variable names should already exist in the workspace and can be called in the function (specify read.csv = FALSE). If the variables aren’t stored, define the path to each CSV file and specify read.csv = TRUE. The sal.range, temp.range, and pH.range can be defined if the default isn’t good. Change the time.step and time.format based on the length of deployment.
plot = plotDS(adcp = trim$adcp,
seafet = trim$seafet,
ctd = trim$ctd,
oxygen = TRUE,
read.csv = FALSE,
time.step = "2 day",
time.format = "%m/%d"
)View the combined plot by calling plot$all.ds and save.
plot$all.ds
ggsave(plot$all.ds, file = "T:/Cruise/CruiseData/HA1801_AmSamoa/Project Group/OceanClimateChange/Diurnal Suites/TUT-ARMS-74/TUT74_DielSuite_2018.pdf", width = 11, height = 8.5)Diel suite work flow
Summarizing the procedure above, below is an example of a R workflow for a typical diel suite (after raw data have been appropriately processed in SeaBird software).
# Moored CTD CNV file
moored.file = "T:/Cruise/CruiseData/HA1801_AmSamoa/Project Group/OceanClimateChange/Diurnal Suites/TUT-ARMS-74/CTD-Moored/HA1801_AmSamoa_TUT_7744_DS001.cnv"
# ADCP PRF file
adcp.file = "T:/Cruise/CruiseData/HA1801_AmSamoa/Project Group/OceanClimateChange/Diurnal Suites/TUT-ARMS-74/ADCP/HA1801_AmSamoa_TUT_ADCP.prf"
# SeaFET temperature and salinity corrected file
seafet.file = "T:/Cruise/CruiseData/HA1801_AmSamoa/Project Group/OceanClimateChange/Diurnal Suites/TUT-ARMS-74/SeaFET/HA1801_AmSamoa_TUT_SeaFET_pro.csv"
# Process all three data sets
mooredctd = processCTD_moored(moored.file)
adcp = processADCP(adcp.file, format = "long")
seafet = processSEAFET(seafet.file)
# Specify output directory for trimmed data
output.dir = "T:/Cruise/CruiseData/HA1801_AmSamoa/Project Group/OceanClimateChange/Diurnal Suites/TUT-ARMS-74/Processed CSV/"
# Trim ADCP, SeaFET, CTD data
trim = trimDS(
adcp = adcp,
seafet = seafet,
ctd = mooredctd,
output.dir = output.dir,
read.csv = FALSE,
filename = '2018_TUT74_diel'
)
# Plot all time series together
plot = plotDS(adcp = trim$adcp,
seafet = trim$seafet,
ctd = trim$ctd,
oxygen = TRUE,
read.csv = FALSE,
time.step = "2 day",
time.format = "%m/%d"
)
# View combined plot
plot$all.ds
# Save combined plot figure
ggsave(plot$all.ds, file = "T:/Cruise/CruiseData/HA1801_AmSamoa/Project Group/OceanClimateChange/Diurnal Suites/TUT-ARMS-74/TUT74_DielSuite_2018.pdf", width = 11, height = 8.5)
# Add CTD data to PUC samples for PMEL
puc.file = "T:/Oceanography/R code/occ_SOP/HA1801_TUT74_PUCs.csv"
ctd.file = "T:/Cruise/CruiseData/HA1801_AmSamoa/Project Group/OceanClimateChange/Diurnal Suites/TUT-ARMS-74/CTD-Moored/HA1801_AmSamoa_TUT_7744_DS001_processed.csv"
puc.samples = processPUC(puc.file, ctd.file)