Data Import
Chemduino Dataset
This section describes the raw data import and reshaping from the low cost sensor module.
Chemduino measurements for the entire measurement period are loaded manually in csv format, this can also be achieved using a suitable API.
chemduino<-read.csv("feeds8.csv",header=TRUE)
names(chemduino)<-c("Time","ID","C.Temp","C.Humidity","C.PM10","C.PM2.5","C.NO2","C.CO","C.Batt")
chemduino$Time<-as.POSIXct(strptime(as.character(chemduino$Time),format="%Y-%m-%d %H:%M:%S"))The data set is split into contiguous sub-sets, split by periods of inactivity (splitting period given by cutoff parameter in seconds). Periods shorter than an optional threshold (in days) are discarded.
##Declare utility functions
return_windows<-function(x,cutoff=21600, threshold=0){
indices<-x$Time
l<-c(0)
for(i in 1:(length(indices)-1)){
init<-indices[[i]]
sec<-indices[[i+1]]
if(sec-cutoff>init){
l<-c(l,i)
}
}
l<-c(l,length(indices))
windowed.data<-list()
no.windows<-length(l)
if(no.windows>0){
for(j in 1:(no.windows-1)){
temp<-x[c((l[j]+1):(l[j+1])),]
windowed.data[[j]]<-temp
}
}
if(threshold){
indices<-lapply(windowed.data,function(x) {difftime( x[length(x[,1]),1],x[1,1],units="days")})>threshold
windowed.data<-windowed.data[indices]
}
windowed.data
}
return_resistance<-function(x,vin=5,raux=22000,scale_max=1023){
m<-((1/((x*vin/scale_max)/(5*raux)))-raux)
}
find_max_CCF<- function(a,b)
{
d <- ccf(a, b, plot = FALSE,na.action=na.approx.default,max=1000)
cor = d$acf[,,1]
lag = d$lag[,,1]
res = data.frame(cor,lag)
res_max = res[which.max(res$cor),]
return(res_max)
}
##Applied to dataset
chemduino<-return_windows(chemduino,21600,6)
summary(chemduino)## Length Class Mode
## [1,] 9 data.frame list
## [2,] 9 data.frame listCalibrant Dataset
Reference data is sourced from the London Air Quality Network, using openair3 R package. Reference location is chosen as LAQN London-Bloomsbury Site due to proximity (site code BL0). Wind speed and direction data are also collected to aid analysis. A summary plot of selected pollutants and Temperature is given below. Chemduino data labels are prepended with “C.”
The data is then coerced into “zoo” class and merged with the windowed time series from Chemduino. Imputation can be done using spline or linear approximations. Spline method is chosen to allow smooth transitions between data points with low resolution After the merge, the object is reverted back into a time series object.
library(openair)
ccl2<-importKCL(site="BL0",year=2016:2017,met=TRUE)## NOTE - mass units are used
## ug/m3 for NOx, NO2, SO2, O3; mg/m3 for CO
## PM10_raw is raw data multiplied by 1.3 lastday<-as.character(as.Date(chemduino[[length(chemduino)]][length(chemduino[[length(chemduino)]][,1]),1]))
firstday<-as.character(as.Date(chemduino[[1]][1,1]))
ccl2<-selectByDate(ccl2,start=firstday,end=lastday)
cclZoo<-read.zoo(ccl2)
mergedData_<-data.frame()
for(i in chemduino){
lastday_<-as.character(as.Date(i[length(i[,1]),1]))
firstday_<-as.character(as.Date(i[1,1]))
temp_<-selectByDate(ccl2,start=firstday_,end=lastday_)
temp_$site<-NULL;temp_$code<-NULL;temp_$solar<-NULL;temp_$so2<-NULL;temp_$v10<-NULL;temp_$v2.5<-NULL;temp_$nv10<-NULL;
temp_$nv2.5<-NULL; temp_$nox<-NULL;
tempZoo<-read.zoo(temp_)
tempi<-read.zoo(i)
mergedData_<-rbind(mergedData_,(as.data.frame(window(na.spline(merge(tempi,tempZoo)),index(tempi)))))
}
date<-as.POSIXct(as.character(rownames(mergedData_)))
mergedData_<-cbind(date,mergedData_)
mergedData_<-mergedData_[,c("date","wd","ws","o3","C.CO","C.PM10","pm10","C.PM2.5","pm25","C.NO2","no2","C.Temp","C.Humidity")]
summaryPlot(mergedData_,print.datacap=FALSE,period="months",type="density")## date1 date2 wd ws o3 C.CO
## "POSIXct" "POSIXt" "numeric" "numeric" "numeric" "numeric"
## C.PM10 pm10 C.PM2.5 pm25 C.NO2 no2
## "numeric" "numeric" "numeric" "numeric" "numeric" "numeric"
## C.Temp C.Humidity
## "numeric" "numeric"
LAQN Data Over Collection Period and Density Summary
There seems to be no correlation between NO2 figures from both data sets, calibration is required to remove the confounding effects of Temperature, Humidity and input voltage ripple.
Calibrating Chemduino NO2 Data 
Calibrating Chemduino NO2 Data 
Calibrating Chemduino NO2 Data 
Calibrating Chemduino NO2 Data 
Binned Performance and CCF 
Binned Performance and CCF 
