Lecture 4 - data Tutorial

Author

Jan C Greyling

Loading the data

Create a new script for this class and call it Lecture 4. Download the Desmoines data from the Teams site and save it in the Data folder of this project. Load the Desmoines data, call it dat, and view it

Code

remove(list=ls()) #clear all from memory
require("data.table")
require("ggplot2")
require("lubridate")
require("tidyverse")

dat <- fread("Data/Weather_DesMoines.csv")
str(dat)

Classes 'data.table' and 'data.frame':  366 obs. of  28 variables:
 $ WBANNO                 : int  54902 54902 54902 54902 54902 54902 54902 54902 54902 54902 ...
 $ LST_DATE               : int  20200101 20200102 20200103 20200104 20200105 20200106 20200107 20200108 20200109 20200110 ...
 $ CRX_VN                 : num  2.62 2.62 2.62 2.62 2.62 ...
 $ LONGITUDE              : num  -93.3 -93.3 -93.3 -93.3 -93.3 ...
 $ LATITUDE               : num  41.6 41.6 41.6 41.6 41.6 ...
 $ T_DAILY_MAX            : num  9.8 9.5 2.5 0.2 7.4 6.7 5.9 -2.4 15.6 -1.6 ...
 $ T_DAILY_MIN            : num  -5.6 0.1 -0.9 -5.7 -2.9 -6.1 -6.4 -10.9 -2.9 -7.8 ...
 $ T_DAILY_MEAN           : num  2.1 4.8 0.8 -2.8 2.2 0.3 -0.3 -6.7 6.3 -4.7 ...
 $ T_DAILY_AVG            : num  2.3 3.6 0.2 -2.9 2.8 -0.3 -0.8 -6.3 5.6 -4.1 ...
 $ P_DAILY_CALC           : num  0 0 0 0.4 0 0 0 0 0 12.4 ...
 $ SOLARAD_DAILY          : num  6.81 7.23 1.49 3.17 7.75 5.87 7.21 8.88 3.86 0.93 ...
 $ SUR_TEMP_DAILY_TYPE    : chr  "C" "C" "C" "C" ...
 $ SUR_TEMP_DAILY_MAX     : num  11.2 12.2 5 2.8 8.3 10.8 10.7 0.1 16 -0.3 ...
 $ SUR_TEMP_DAILY_MIN     : num  -6.7 -2.5 -0.5 -5.5 -5.1 -8.8 -7.5 -11.7 -3.9 -8.2 ...
 $ SUR_TEMP_DAILY_AVG     : num  1.2 3.4 0.6 -3 1.6 -0.9 -1.5 -5.8 4.3 -3.9 ...
 $ RH_DAILY_MAX           : num  90.4 93.9 92.1 97.6 86.4 93.1 90.1 69.5 89.5 94.6 ...
 $ RH_DAILY_MIN           : num  47.9 60.8 77 74.7 47 22 44.1 38.1 39.5 78.4 ...
 $ RH_DAILY_AVG           : num  69.4 82 87.2 87.4 66.4 67.7 72.9 55.2 63.1 89 ...
 $ SOIL_MOISTURE_5_DAILY  : num  0.378 0.38 0.38 0.378 0.376 0.375 0.374 0.372 0.37 0.37 ...
 $ SOIL_MOISTURE_10_DAILY : num  0.405 0.405 0.405 0.403 0.402 0.401 0.4 0.399 0.399 0.399 ...
 $ SOIL_MOISTURE_20_DAILY : num  0.421 0.419 0.418 0.417 0.416 0.416 0.416 0.414 0.415 0.414 ...
 $ SOIL_MOISTURE_50_DAILY : num  0.443 0.442 0.44 0.44 0.44 0.44 0.439 0.439 0.439 0.437 ...
 $ SOIL_MOISTURE_100_DAILY: num  0.494 0.494 0.494 0.494 0.493 0.493 0.494 0.494 0.493 0.494 ...
 $ SOIL_TEMP_5_DAILY      : num  1.4 2.4 2.7 2.1 2.1 1.9 1.9 1.3 1.4 1.6 ...
 $ SOIL_TEMP_10_DAILY     : num  1.9 2.4 2.7 2.4 2.2 2.2 2.1 1.7 1.5 1.7 ...
 $ SOIL_TEMP_20_DAILY     : num  2.9 3.1 3.3 3.2 3 3 2.9 2.7 2.4 2.5 ...
 $ SOIL_TEMP_50_DAILY     : num  4.5 4.4 4.4 4.5 4.4 4.3 4.3 4.2 4.1 4 ...
 $ SOIL_TEMP_100_DAILY    : num  5.4 5.3 5.2 5.2 5.2 5.2 5.1 5.1 5 4.9 ...
 - attr(*, ".internal.selfref")=<externalptr>

Exploring the data

Ok, so there is a lot of variables. What are we working with?

Code

summary(dat)

     WBANNO         LST_DATE            CRX_VN        LONGITUDE     
 Min.   :54902   Min.   :20200101   Min.   :2.622   Min.   :-93.29  
 1st Qu.:54902   1st Qu.:20200401   1st Qu.:2.622   1st Qu.:-93.29  
 Median :54902   Median :20200702   Median :2.622   Median :-93.29  
 Mean   :54902   Mean   :20200667   Mean   :2.622   Mean   :-93.29  
 3rd Qu.:54902   3rd Qu.:20200983   3rd Qu.:2.622   3rd Qu.:-93.29  
 Max.   :54902   Max.   :20201231   Max.   :2.622   Max.   :-93.29  
    LATITUDE      T_DAILY_MAX         T_DAILY_MIN        T_DAILY_MEAN   
 Min.   :41.56   Min.   :-9999.000   Min.   :-9999.00   Min.   :-18.80  
 1st Qu.:41.56   1st Qu.:    6.025   1st Qu.:   -3.75   1st Qu.:  1.60  
 Median :41.56   Median :   17.050   Median :    3.45   Median : 10.10  
 Mean   :41.56   Mean   :  -11.286   Mean   : -214.24   Mean   : 10.42  
 3rd Qu.:41.56   3rd Qu.:   26.700   3rd Qu.:   12.97   3rd Qu.: 20.07  
 Max.   :41.56   Max.   :   35.100   Max.   :   23.80   Max.   : 28.20  
  T_DAILY_AVG       P_DAILY_CALC    SOLARAD_DAILY    SUR_TEMP_DAILY_TYPE
 Min.   :-19.000   Min.   : 0.000   Min.   : 0.140   Length:366         
 1st Qu.:  1.125   1st Qu.: 0.000   1st Qu.: 7.173   Class :character   
 Median : 10.200   Median : 0.000   Median :13.260   Mode  :character   
 Mean   : 10.161   Mean   : 2.129   Mean   :13.735                      
 3rd Qu.: 20.000   3rd Qu.: 0.400   3rd Qu.:20.058                      
 Max.   : 27.900   Max.   :75.300   Max.   :30.050                      
 SUR_TEMP_DAILY_MAX SUR_TEMP_DAILY_MIN SUR_TEMP_DAILY_AVG  RH_DAILY_MAX   
 Min.   :-12.70     Min.   :-25.900    Min.   :-19.000    Min.   : 55.40  
 1st Qu.:  8.85     1st Qu.: -5.175    1st Qu.:  0.825    1st Qu.: 88.40  
 Median : 20.65     Median :  2.700    Median :  9.750    Median : 94.40  
 Mean   : 18.26     Mean   :  2.886    Mean   :  9.722    Mean   : 91.63  
 3rd Qu.: 28.68     3rd Qu.: 11.775    3rd Qu.: 19.475    3rd Qu.: 97.40  
 Max.   : 37.70     Max.   : 23.100    Max.   : 27.200    Max.   :100.00  
  RH_DAILY_MIN    RH_DAILY_AVG   SOIL_MOISTURE_5_DAILY SOIL_MOISTURE_10_DAILY
 Min.   :17.20   Min.   :35.30   Min.   :-99.0000      Min.   :-99.0000      
 1st Qu.:39.65   1st Qu.:65.55   1st Qu.:  0.2102      1st Qu.:  0.2422      
 Median :49.85   Median :74.00   Median :  0.3060      Median :  0.3345      
 Mean   :52.00   Mean   :73.67   Mean   : -8.6550      Mean   : -8.6252      
 3rd Qu.:63.35   3rd Qu.:82.62   3rd Qu.:  0.3680      3rd Qu.:  0.3970      
 Max.   :97.90   Max.   :99.70   Max.   :  0.4280      Max.   :  0.4410      
 SOIL_MOISTURE_20_DAILY SOIL_MOISTURE_50_DAILY SOIL_MOISTURE_100_DAILY
 Min.   :-99.0000       Min.   :-99.0000       Min.   :-99.0000       
 1st Qu.:  0.3320       1st Qu.:  0.3892       1st Qu.:  0.4537       
 Median :  0.3835       Median :  0.4345       Median :  0.4900       
 Mean   : -3.9722       Mean   : -4.4716       Mean   : -3.3280       
 3rd Qu.:  0.4170       3rd Qu.:  0.4490       3rd Qu.:  0.5000       
 Max.   :  0.4710       Max.   :  0.4870       Max.   :  0.5260       
 SOIL_TEMP_5_DAILY  SOIL_TEMP_10_DAILY SOIL_TEMP_20_DAILY SOIL_TEMP_50_DAILY
 Min.   :-9999.00   Min.   :-9999.00   Min.   :-9999.00   Min.   :-9999.00  
 1st Qu.:    1.00   1st Qu.:    0.90   1st Qu.:    2.10   1st Qu.:    3.00  
 Median :    6.95   Median :    6.75   Median :    7.45   Median :    7.25  
 Mean   : -837.34   Mean   :-1056.32   Mean   : -892.04   Mean   :-1165.39  
 3rd Qu.:   17.90   3rd Qu.:   17.30   3rd Qu.:   17.43   3rd Qu.:   17.00  
 Max.   :   24.00   Max.   :   23.30   Max.   :   22.60   Max.   :   21.10  
 SOIL_TEMP_100_DAILY
 Min.   :-9999.00   
 1st Qu.:    4.20   
 Median :    8.05   
 Mean   : -974.07   
 3rd Qu.:   15.88   
 Max.   :   20.00

WBANNO is the location code of the measurements.

LST_DATE is the measurement data ranging from 2020/01/01 to 2020/12/31.

LONGITUDE and LATITUDE shows the location of the measurements, all are at the same location.

T_DAILY_MAX,T_DAILY_MIN,T_DAILY_MEAN,T_DAILY_AVG,P_DAILY_CALC are all temperature variables

SOLARAD_DAILY represents daily solar radiation

RH_ is relative humidity

SOIL_ shows soil variables

Subsetting

We are not interested in all of these variables, lets make a copy of the data and call it d.OR and create a new version of dat wherein we keep the variables of interest as T_DAILY_MEAN and SOLARAD_DAILY.

Code

d.OR <- dat
dat <- dat[, .(LST_DATE,T_DAILY_MEAN,SOLARAD_DAILY)]

Variable names

Now change the names of the variables.

Code

setnames(dat,c("LST_DATE","T_DAILY_MEAN","SOLARAD_DAILY"),c("date","tas","rsds"))

Working with dates

At the momement, the variable LST_DATE is an integer meaning that it is a whole number. Change it to a date using the lubridate package.

Code

dat[, date := ymd(date)]
str(dat$date)

 Date[1:366], format: "2020-01-01" "2020-01-02" "2020-01-03" "2020-01-04" "2020-01-05" ...

Assignment 4

Plot the daily mean temperature (tas) using the techniques that you learned in the previous lectures
Build a regression model to that you can use to predict daily mean temperature (tas) using solar radiation (rsds). Then to do the following

Discuss the goodness of fit of your model
Predict tas and call it tas_pred
Calculate your residuals, i.e. the differnce between the predicted and actual values, by Subtracting tas_pred from tas and call it tad_div
Plot your tad_div and tas_pred as a scatter plot, fit a linear trendline and make the plot neat.