The HIEv R package is not available on CRAN, and must be installed with the devtools package as follows. Windows users must have Rtools installed.
library(devtools)
install_bitbucket("remkoduursma/HIEv")Also use this command to update the HIEv R package, which I recommend you do every so often.
Then load the package as usual,
library(HIEv)To be able to communicate with the HIEv, you have to set the API token. This token can be found by logging into the HIEv, click on your email address in the top-right, then ‘Settings’, and the API token will be shown. You can set this token in a few different ways, but I strongly recommend you store it in a separate text file, not include it in your scripts. You must take care to not share the token publicly.
Paste the token in a text file, and set the token every time you use the HIEv R package, like so:
# If the token is stored in a text file in the current working directory
setToken(tokenfile="mytokenfile.txt")
# On Windows, if you store your token in 'c:/hiev/token.txt', you can just do:
setToken()To search for files on the HIEv, we use the searchHIEv function, which has many options. A few of them are shown in the examples below, but more are available from the help page (?searchHIEv).
The default way to search for files is by any part of a filename. For example,
# Files from the WTC_TEMP experiment
searchHIEv("WTC_TEMP")## Your search returned 98 records.
##
## Filenames (max.10): Permission:
## 1 WTC_TEMP_CM_LEAFCARB-GIRDLE_20140513-20140525_R.csv Yes
## 2 WTC_TEMP_CM_LEAFCARB-DIURNAL_20140220-20140326_R.csv Yes
## 3 WTC_TEMP_CM_LI2200_20130514-20140227_L2.csv Yes
## 4 WTC_TEMP_CM_BARK-OSMOTPOT_20140525_R.csv Yes
## 5 WTC_TEMP_CM_LEAF-CN-ACi-TEMP_20130917-20140507_L2.csv Yes
## 6 WTC_TEMP_CM_GMES_20131026-20140421_L2.csv Yes
## 7 WTC_TEMP_CM_BARKCARB-GIRDLE_20140513-20140525_R.csv Yes
## 8 WTC_TEMP_CM_LEAF-CN_20130515-20140402_L2.csv Yes
## 9 WTC_TEMP_CM_LEAFCARB_20130515-20140402_L2.csv Yes
## 10 WTC_TEMP_CM_SAPFLUX-15MIN-GIRDLE_20140501-20140527.csv YesBy default, only the first 10 results are shown. However, all search results can be stored if the search is saved into an object.
mysearch <- searchHIEv("WTC_TEMP", quiet=TRUE)
nrow(mysearch)## [1] 98head(mysearch$filename)## [1] "WTC_TEMP_CM_LEAFCARB-GIRDLE_20140513-20140525_R.csv"
## [2] "WTC_TEMP_CM_LEAFCARB-DIURNAL_20140220-20140326_R.csv"
## [3] "WTC_TEMP_CM_LI2200_20130514-20140227_L2.csv"
## [4] "WTC_TEMP_CM_BARK-OSMOTPOT_20140525_R.csv"
## [5] "WTC_TEMP_CM_LEAF-CN-ACi-TEMP_20130917-20140507_L2.csv"
## [6] "WTC_TEMP_CM_GMES_20131026-20140421_L2.csv"Where the mysearch object is a dataframe, containing all variables returned by the HIEv.
Note : if you search for term that matches most or all filenames on HIEv, the searchHIEv functio will be very slow.
When uploading files to the HIEv, the facility and experiment always have to be set. We can also search for files from any particular experiment or facility. It may be useful to list the facilities on HIEv, like so:
showFacilities()## Admin Facility
## EucFACE
## HFE Common Garden
## HIE Laboratory
## Mini Rainout Shelters
## ROS Weather Station
## Whole Tree Chambers
## Cumberland Plains Eddy Flux Tower (CUP)
## Glasshouse S39
## HFE diversity plantations
## Irrigation x Fertilisation Plots
## Rainout Shelters
## S30 GlasshouseAnd similarly, for experiments (results hidden),
showExperiments()We can now search for data from some experiment, for example,
searchHIEv(experiment_name = "EucFACE_P0017_Andrew_ insect_herbivore")## Found the following experiment(s):
## EucFACE_P0017_Andrew_ insect_herbivore (EucFACE)
##
## Your search returned 4 records.
##
## Filenames (max.10): Permission:
## 1 FACE_P0017_RA_Litter_20121001-20131231-R.csv Yes
## 2 FACE_P0017_RA_FRASSFALL_L2_20120914-20150209.csv Yes
## 3 FACE_P0017_RA_FRASSCHEMISTRY_L2_20121112-20141016.csv Yes
## 4 FACE_P0017_RA_Litter_20140101-20141016-R.csv YesYou can use the facility_name argument to searchHIEv in a similar way to find facility data.
We can also search by (any part of) uploader name or email address,
searchHIEv(uploader_name="Remko")## Found the following user(s):
## Remko Duursma (r.duursma@uws.edu.au)
## Remko Duursma (remkoduursma@gmail.com)
##
## Your search returned 110 records.
##
## Filenames (max.10): Permission:
## 1 WTC_TEMP_CM_LI2200_20130514-20140227_L2.csv Yes
## 2 WTC_CO2DROUGHT_CM_DATABASE_20090401_L2.ZIP Yes
## 3 FACE_RA_P0037_PARAGG_20121016-20150301_L2.csv Yes
## 4 FACE_RA_P0037_LEAFLITTER_20121009-20140814_L2.csv Yes
## 5 FACE_RA_P0037_GAPFRACLAI_20121026-20150225_L2.csv Yes
## 6 FACE_RA_P0037_PHOTOGAPFRAC_20121020-20131022_L2.csv Yes
## 7 FACE_RA_P0037_DAILYMET_20110619-20151026_L2.csv Yes
## 8 FACE_RA_P0037_SOLVARS_20120601-20200101_L2.csv Yes
## 9 FACE_P0037_RA_GAPFRACLAI_OPEN_L2.dat Yes
## 10 WTC_TEMPCO2_CM_EVENTLOG_20100324-20121204_L1.csv Yeswhich also finds all files from users that have (part of) that name, even if uploaded under multiple user accounts. All users of the HIEv can be listed with the showUsers function.
It is also possible to use regular expressions to find files on the HIEv. For example,
# Find all files that look like 'ROS_1_Table..', 'ROS_5_Table...' and so on.
rostables <- searchHIEv(filename="ROS_[0-9]{1}_Table")## Your search returned 84 records.
##
## Filenames (max.10): Permission:
## 1 ROS_3_Table2_20130430.dat Yes
## 2 ROS_1_Table2_20130430.dat Yes
## 3 ROS_2_Table2_20130430.dat Yes
## 4 ROS_5_Table2_20130430.dat Yes
## 5 ROS_4_Table2_20130430.dat Yes
## 6 ROS_6_Table2_20130430.dat Yes
## 7 ROS_4_Table2_20130630.dat Yes
## 8 ROS_1_Table2_20130630.dat Yes
## 9 ROS_3_Table2_20130630.dat Yes
## 10 ROS_2_Table2_20130630.dat YesAnd to combine terms (all terms are to be found),
searchHIEv(c("WTC_TEMP","LEAF"))## Your search returned 11 records.
##
## Filenames (max.10): Permission:
## 1 WTC_TEMP_CM_LEAFCARB-GIRDLE_20140513-20140525_R.csv Yes
## 2 WTC_TEMP_CM_LEAFCARB-DIURNAL_20140220-20140326_R.csv Yes
## 3 WTC_TEMP_CM_LEAF-CN-ACi-TEMP_20130917-20140507_L2.csv Yes
## 4 WTC_TEMP_CM_LEAF-CN_20130515-20140402_L2.csv Yes
## 5 WTC_TEMP_CM_LEAFCARB_20130515-20140402_L2.csv Yes
## 6 WTC_TEMP_CM_PETIOLEGIRDLE-LEAFMASS-AREA-CARB_20140507-20140512_L2.CSV Yes
## 7 WTC_TEMP_CM_BIOMASS-LEAFCOUNT_20140226_L1.csv Yes
## 8 WTC_TEMP_CM_LEAFLITTER_20130913-20140528_L1.csv Yes
## 9 WTC_TEMP_CM_leaf-sample-record-archive_20121209-20140630.csv Yes
## 10 WTC_TEMP_CM_BIOMASS-LEAFCOUNT_20130909_L1.csv YesFor files that have set the ‘start time’ and ‘end time’ fields on the HIEv, it is useful to use the startDate and endDate arguments to searchHIEv, as the example below. This is particularly useful for automated logger data, which are normally uploaded in monthly chunks.
searchHIEv("FACE_R1_B1_AirVars", startDate="2015-10-1")## Your search returned 4 records.
##
## Filenames (max.10): Permission:
## 1 FACE_R1_B1_AirVars_20160131.dat Yes
## 2 FACE_R1_B1_AirVars_20151031.dat Yes
## 3 FACE_R1_B1_AirVars_20151130.dat Yes
## 4 FACE_R1_B1_AirVars_20151231.dat YesTo download files from the HIEv, it is necessary to first perform the search. It is also important to set the path to which files are to be downloaded. If you do nothing, it will be the current working directory, which is probably not a good idea if you want to keep your files organized.
# Set download path for the current session
setToPath("c:/temp")
# I prefer to keep files in a subdirectory of the current working directory,
setToPath("cache")Note that the directory has to exist, it will not be created for you.
To download a single file, we first do a search and store it in an object,
s <- searchHIEv("WTC_TEMP_CM_LEAFCARB_20130515-20140402_L2.csv")## Your search returned 1 records.
##
## Filenames (max.10): Permission:
## 1 WTC_TEMP_CM_LEAFCARB_20130515-20140402_L2.csv Yesand then download the files returned by the search,
downloadHIEv(s)## Downloading 0 Kb.
##
## Skipping WTC_TEMP_CM_LEAFCARB_20130515-20140402_L2.csv - file exists.
## Done.## Files were downloaded into : C:/repos/hiev/vignettes
## Some files were skipped. To redownload these files, delete the local copy.If multiple files are found by the search, all will be downloaded. Alternatively, set the which argument to specify which files of the search to download.
Note : the downloadHIEv function does not download more than 50 files unless the user requests this. This is to avoid accidental very large downloads. To override, use the command downloadHIEv(mysearch, maxnfiles = 1000).
For CSV (comma-separated values) and TOA5 (automated logger output) files, the HIEv R package has two convenience functions that search, download and read the files from HIEv. For example,
leafcarb <- downloadCSV("WTC_TEMP_CM_LEAFCARB_20130515-20140402_L2.csv") # CSV## Your search returned 1 records.
##
## Filenames (max.10): Permission:
## 1 WTC_TEMP_CM_LEAFCARB_20130515-20140402_L2.csv Yes
##
##
## Downloading 0 Kb.
##
## Skipping WTC_TEMP_CM_LEAFCARB_20130515-20140402_L2.csv - file exists.
## Done.## Files were downloaded into : C:/repos/hiev/vignettes
## Some files were skipped. To redownload these files, delete the local copy.
## Reading files...done.faceair <- downloadTOA5("FACE_R1_B1_AirVars_20160131.dat") # TOA5## Your search returned 1 records.
##
## Filenames (max.10): Permission:
## 1 FACE_R1_B1_AirVars_20160131.dat Yes
##
##
## Downloading 2.04 MB.
##
## ---=== Downloading file 1 of 1 (2.04 MB) -- FACE_R1_B1_AirVars_20160131.dat
## Done.## Files were downloaded into : C:/repos/hiev/vignettes
##
## Reading files...done.
## Combining dataframes...done.When multiple files match the search string, both functions attempt to row-bind the results into one dataframe. This is most useful for timeseries data from automated loggers. For example,
airvar <- downloadTOA5("FACE_R1_B1_AirVars", startDate="2015-10-1", endDate="2016-1-15")## Your search returned 4 records.
##
## Filenames (max.10): Permission:
## 1 FACE_R1_B1_AirVars_20160131.dat Yes
## 2 FACE_R1_B1_AirVars_20151031.dat Yes
## 3 FACE_R1_B1_AirVars_20151130.dat Yes
## 4 FACE_R1_B1_AirVars_20151231.dat Yes
##
##
## Downloading 8.93 MB.
##
## Skipping FACE_R1_B1_AirVars_20160131.dat - file exists.
## ---=== Downloading file 2 of 4 (3 MB) -- FACE_R1_B1_AirVars_20151031.dat
## ---=== Downloading file 3 of 4 (2.91 MB) -- FACE_R1_B1_AirVars_20151130.dat
## ---=== Downloading file 4 of 4 (3.01 MB) -- FACE_R1_B1_AirVars_20151231.dat
## Done.## Files were downloaded into : C:/repos/hiev/vignettes
## Some files were skipped. To redownload these files, delete the local copy.
##
## Reading files...done.
## Combining dataframes...done.Both functions will look for a DateTime variable, which will be converted to a POSIXct class with standard methods (or see datetimevar and posixctFormat arguments). This makes life easy,
# Plot not shown
with(airvar, plot(DateTime, PAR_Den_2_Avg, type='l'))