Processing data report for JEFF station

Introduction

Data about pollution measured by this station has been provided by Prof Ouarzazi in 2013. It accounts for Co, NO2, Wind Speed, Temperature, PM10, SO2, Solar Radiation and Ozone hourly based.

setwd("~/git/ouarzazi")
suppressPackageStartupMessages(library(googleVis))
suppressPackageStartupMessages(library(xtable))
suppressPackageStartupMessages(library(Peaks))
suppressPackageStartupMessages(library(magic))
suppressPackageStartupMessages(library(segmented))
suppressPackageStartupMessages(library(fftw))
suppressPackageStartupMessages(library(FITSio))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(utils))
suppressPackageStartupMessages(library(e1071))
suppressPackageStartupMessages(library(randomForest))
suppressPackageStartupMessages(library(rpart))
suppressPackageStartupMessages(library(nnet))
suppressPackageStartupMessages(library(party))
suppressPackageStartupMessages(library(partykit))

if (file.exists("JEF.RData")) {
    load("JEF.RData")
} else {
    JEF <- read.csv(file = "JEF_data.csv", sep = ";", dec = ",", header = TRUE)
    DJEF <- JEF[!is.na(as.Date(as.character(JEF[, 1]))), 1:11]
    newc <- paste(as.character(DJEF[, 1]), paste(DJEF[, 2], ":00:00", sep = ""), 
        sep = " ")
    newd <- strptime(newc, "%d/%m/%y %H:%M:%S")
    antes <- newd - 3600
    NJEF <- DJEF[, -2]
    NJEF[, 1] <- as.data.frame(newd)
    save(JEF, DJEF, NJEF, file = "JEF.RData")
}
rm(JEF)

Numerical treatment will be performed by using the well known open source statistical environment R (http://www.r-project.org).

Processing

In order to compare with Prof Ouarzazi's results (corr = 0.84) for a local based model O3 ~ remaining variables at the same period, we will use several technologies.

Basic methodology will be:

• To apply cross correlation learning validation as it becomes more robust that the fixed approach 70%,15%,15%
• To apply full validation to all dataset, after selecting the best model, as Prof Ouarzazi did.
• The hourly based moted was selected as for learning what it is possible to do, even when \( O_3 \) should be accounted by its maximum per day and/or the dosage by 8h periods, depending on the specific regulation.
• Uncertainty about future predictors was removed as we were no predicting Ozone with any lag.

Linear approach as reference

A linear model is considered as reference, for comparison of results in order to evaluate the degree of linearity

if (file.exists("JEF_lm.RData")) {
    load("JEF_lm.RData")
} else {
    objJEF <- tune(lm, O3.JEF ~ ., data = NJEF[, -1], tunecontrol = tune.control(sampling = "cross", 
        cross = 10))
    JEF.lm <- objJEF
    pO3 <- predict(JEF.lm$best.model, NJEF[, -c(1, 4)])
    cr.lm <- cor(as.numeric(NJEF[, 4]), as.numeric(pO3))
    save(JEF.lm, NJEF, cr.lm, file = "JEF_lm.RData")
}
pO3 <- predict(JEF.lm$best.model, NJEF[, -c(1, 4)])
cr.lm <- cor(as.numeric(NJEF[, 4]), as.numeric(pO3))
par(mgp = c(2.2, 0.45, 0), tcl = 0.4, mar = c(3.3, 3.6, 1.1, 1.1))
plot(as.numeric(NJEF[, 4]), as.numeric(pO3), main = expression(O[3] ~ Prediction. ~ 
    JEF ~ station), xlab = expression(O[3] ~ real), ylab = expression(O[3] ~ 
    LM ~ predicted))
lines(c(0, 200), c(0, 200), col = 2)

The results found account for a correlation of 0.7644. It will considered as a reference.

SVM approach

A wrapper for SVM based regressors is applied looking for best parameters of learning.

The results found account for a correlation of 0.9509 which outperforms the initial proposal carried out by Prof. Ouarzazi.

RandomForest

Let's test the randomForest technology.

The results found account for a correlation of 0.9829.

FFNN: MLP

Let's test backpropagation trained multilayer perceptron type neural network do their work.

The results found account for a correlation of 0.8329.

CART solution

Now we will use classification and regression trees to have a look at their capabilities for this particular problem.

The results found account for a correlation of 0.8329.

Conclusions

After this short analysis we can conclude that:

From the figures, it is clear that RF produces some kind of understimation of higher values, probably because the data set is density imbalanced. Regarding this particular factor it exhibits a pretty nice performance the SVM technology.

In a global view we can conclude that the best fit was scored for RF method with a corrlation factor of 0.9829