AS2-2 Detecting Flu Epidemics via Search Engine Query Data

edX assignment link: http://bit.ly/2tXZc4F

Flu epidemics constitute a major public health concern causing respiratory illnesses, hospitalizations, and deaths. According to the National Vital Statistics Reports published in October 2012, influenza ranked as the eighth leading cause of death in 2011 in the United States. Each year, 250,000 to 500,000 deaths are attributed to influenza related diseases throughout the world.

The U.S. Centers for Disease Control and Prevention (CDC) and the European Influenza Surveillance Scheme (EISS) detect influenza activity through virologic and clinical data, including Influenza-like Illness (ILI) physician visits. Reporting national and regional data, however, are published with a 1-2 week lag.

The Google Flu Trends project was initiated to see if faster reporting can be made possible by considering flu-related online search queries – data that is available almost immediately.

---

Section 1 - Understanding the Data

1.1

We would like to estimate influenza-like illness (ILI) activity using Google web search logs. Fortunately, one can easily access this data online:

ILI Data - The CDC publishes on its website the official regional and state-level percentage of patient visits to healthcare providers for ILI purposes on a weekly basis.

Google Search Queries - Google Trends allows public retrieval of weekly counts for every query searched by users around the world. For each location, the counts are normalized by dividing the count for each query in a particular week by the total number of online search queries submitted in that location during the week. Then, the values are adjusted to be between 0 and 1.

The csv file FluTrain.csv aggregates this data from January 1, 2004 until December 31, 2011 as follows:

“Week” - The range of dates represented by this observation, in year/month/day format.

“ILI” - This column lists the percentage of ILI-related physician visits for the corresponding week.

“Queries” - This column lists the fraction of queries that are ILI-related for the corresponding week, adjusted to be between 0 and 1 (higher values correspond to more ILI-related search queries).

Before applying analytics tools on the training set, we first need to understand the data at hand. Load “FluTrain.csv” into a data frame called FluTrain. Looking at the time period 2004-2011, which week corresponds to the highest percentage of ILI-related physician visits? Select the day of the month corresponding to the start of this week.

TR=read.csv("C:/bussiness analytics/data/0711data/FluTrain.csv")
# 417

Which week corresponds to the highest percentage of ILI-related query fraction?

TR[which.max(TR$ILI),]

TR[which.max(TR$Queries),]

1.2

Let us now understand the data at an aggregate level. Plot the histogram of the dependent variable, ILI. What best describes the distribution of values of ILI?

• Most of the ILI values are small, with a relatively small number of much larger values (in statistics, this sort of data is called “skew right”).

• The ILI values are balanced, with equal numbers of unusually large and unusually small values.

• Most of the ILI values are large, with a relatively small number of much smaller values (in statistics, this sort of data is called “skew left”).

hist(TR$ILI,20)  # right skew

1.3

When handling a skewed dependent variable, it is often useful to predict the logarithm of the dependent variable instead of the dependent variable itself – this prevents the small number of unusually large or small observations from having an undue influence on the sum of squared errors of predictive models. In this problem, we will predict the natural log of the ILI variable, which can be computed in R using the log() function.

Plot the natural logarithm of ILI versus Queries. What does the plot suggest?.

• There is a negative, linear relationship between log(ILI) and Queries.
• There is no apparent linear relationship between log(ILI) and Queries.
• There is a positive, linear relationship between log(ILI) and Queries.

plot(TR$Queries, log(TR$ILI))  # linear

Section 2 - Linear Regression Model

2.1

Based on the plot we just made, it seems that a linear regression model could be a good modeling choice. Based on our understanding of the data from the previous subproblem, which model best describes our estimation problem?

• ILI = intercept + coefficient x Queries, where the coefficient is negative
• Queries = intercept + coefficient x ILI, where the coefficient is negative
• ILI = intercept + coefficient x Queries, where the coefficient is positive
• Queries = intercept + coefficient x ILI, where the coefficient is positive
• log(ILI) = intercept + coefficient x Queries, where the coefficient is negative
• Queries = intercept + coefficient x log(ILI), where the coefficient is negative
• log(ILI) = intercept + coefficient x Queries, where the coefficient is positive
• Queries = intercept + coefficient x log(ILI), where the coeffcient is positive

# log(ILI) = b0 + b1*Queries 

2.2

Let’s call the regression model from the previous problem (Problem 2.1) FluTrend1 and run it in R. Hint: to take the logarithm of a variable Var in a regression equation, you simply use log(Var) when specifying the formula to the lm() function.

What is the training set R-squared value for FluTrend1 model (the “Multiple R-squared”)?

FluTrend1 = lm(log(ILI)~Queries, TR)
summary(FluTrend1) 

Call:
lm(formula = log(ILI) ~ Queries, data = TR)

Residuals:
     Min       1Q   Median       3Q      Max 
-0.76003 -0.19696 -0.01657  0.18685  1.06450 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.49934    0.03041  -16.42   <2e-16 ***
Queries      2.96129    0.09312   31.80   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.2995 on 415 degrees of freedom
Multiple R-squared:  0.709, Adjusted R-squared:  0.7083 
F-statistic:  1011 on 1 and 415 DF,  p-value: < 2.2e-16

# (Intercept)  -0.4993     0.0304   -16.4   <2e-16 ***
# Queries       2.9613     0.0931    31.8   <2e-16 ***
# Multiple R-squared:  0.709

2.3

For a single variable linear regression model, there is a direct relationship between the R-squared and the correlation between the independent and the dependent variables. What is the relationship we infer from our problem? (Don’t forget that you can use the cor function to compute the correlation between two variables.)

• R-squared = Correlation^2
• R-squared = log(1/Correlation)
• R-squared = exp(-0.5*Correlation)

cor(log(TR$ILI), TR$Queries)^2 # 0.70902

[1] 0.7090201

Note that the “exp” function stands for the exponential function. The exponential can be computed in R using the function exp().

Section 3 - Performance on the Test Set

3.1

The csv file FluTest.csv provides the 2012 weekly data of the ILI-related search queries and the observed weekly percentage of ILI-related physician visits. Load this data into a data frame called FluTest.

Normally, we would obtain test-set predictions from the model FluTrend1 using the code

PredTest1 = predict(FluTrend1, newdata=FluTest)

However, the dependent variable in our model is log(ILI), so PredTest1 would contain predictions of the log(ILI) value. We are instead interested in obtaining predictions of the ILI value. We can convert from predictions of log(ILI) to predictions of ILI via exponentiation, or the exp() function. The new code, which predicts the ILI value, is

PredTest1 = exp(predict(FluTrend1, newdata=FluTest))

What is our estimate for the percentage of ILI-related physician visits for the week of March 11, 2012? (HINT: You can either just output FluTest$Week to find which element corresponds to March 11, 2012, or you can use the “which” function in R. To learn more about the which function, type ?which in your R console.)

TS=read.csv("C:/bussiness analytics/data/0711data/FluTest.csv")
# 3663
pred1 = exp(predict(FluTrend1, newdata=TS))
i = which(substr(TS$Week, 1, 10) == "2012-03-11")
pred1[i]  # 2.1874

      11 
2.187378 

3.2

What is the relative error betweeen the estimate (our prediction) and the observed value for the week of March 11, 2012? Note that the relative error is calculated as

(Observed ILI - Estimated ILI)/Observed ILI

(TS$ILI[i] - pred1[i])/TS$ILI[i]  # 0.046238

        11 
0.04623827 

3.3

What is the Root Mean Square Error (RMSE) between our estimates and the actual observations for the percentage of ILI-related physician visits, on the test set?

(SSE = sum( (TS$ILI - pred1)^2 )) # 29.177

[1] 29.17708

(RMSE = sqrt( SSE/nrow(TS) ) )    # 0.74906

[1] 0.7490645

Section 4 - Training a Time Series Model

4.1

The observations in this dataset are consecutive weekly measurements of the dependent and independent variables. This sort of dataset is called a “time series.” Often, statistical models can be improved by predicting the current value of the dependent variable using the value of the dependent variable from earlier weeks. In our models, this means we will predict the ILI variable in the current week using values of the ILI variable from previous weeks.

First, we need to decide the amount of time to lag the observations. Because the ILI variable is reported with a 1- or 2-week lag, a decision maker cannot rely on the previous week’s ILI value to predict the current week’s value. Instead, the decision maker will only have data available from 2 or more weeks ago. We will build a variable called ILILag2 that contains the ILI value from 2 weeks before the current observation.

To do so, we will use the “zoo” package, which provides a number of helpful methods for time series models. While many functions are built into R, you need to add new packages to use some functions. New packages can be installed and loaded easily in R, and we will do this many times in this class. Run the following two commands to install and load the zoo package. In the first command, you will be prompted to select a CRAN mirror to use for your download. Select a mirror near you geographically.

install.packages("zoo")

library(zoo)

After installing and loading the zoo package, run the following commands to create the ILILag2 variable in the training set:

ILILag2 = lag(zoo(FluTrain$ILI), -2, na.pad=TRUE)

FluTrain$ILILag2 = coredata(ILILag2)

In these commands, the value of -2 passed to lag means to return 2 observations before the current one; a positive value would have returned future observations. The parameter na.pad=TRUE means to add missing values for the first two weeks of our dataset, where we can’t compute the data from 2 weeks earlier.

How many values are missing in the new ILILag2 variable?

# library(zoo)
# TR$ILILag2 = coredata(lag(zoo(TR$ILI), -2, na.pad=TRUE))
# is.na(TR$ILILag2) %>% sum    # 2
TR$ILILag2 = dplyr::lag(TR$ILI,2)
is.na(TR$ILILag2) %>% sum    # 2

Error in is.na(TR$ILILag2) %>% sum : could not find function "%>%"

4.2

Use the plot() function to plot the log of ILILag2 against the log of ILI. Which best describes the relationship between these two variables?

• There is a strong negative relationship between log(ILILag2) and log(ILI).
• This is a weak or no relationship between log(ILILag2) and log(ILI)
• There is a strong positive relationship between log(ILILag2) and log(ILI).

plot(log(TR$ILILag2), log(TR$ILI))  # strong positive

4.3

Train a linear regression model on the FluTrain dataset to predict the log of the ILI variable using the Queries variable as well as the log of the ILILag2 variable. Call this model FluTrend2.

Which coefficients are significant at the p=0.05 level in this regression model? (Select all that apply.)

• Intercept
• Queries
• log(ILILag2)

FluTrend2 = lm(log(ILI) ~ Queries + log(ILILag2), TR)
summary(FluTrend2)

Call:
lm(formula = log(ILI) ~ Queries + log(ILILag2), data = TR)

Residuals:
     Min       1Q   Median       3Q      Max 
-0.52209 -0.11082 -0.01819  0.08143  0.76785 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept)  -0.24064    0.01953  -12.32   <2e-16 ***
Queries       1.25578    0.07910   15.88   <2e-16 ***
log(ILILag2)  0.65569    0.02251   29.14   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.1703 on 412 degrees of freedom
  (2 observations deleted due to missingness)
Multiple R-squared:  0.9063,    Adjusted R-squared:  0.9059 
F-statistic:  1993 on 2 and 412 DF,  p-value: < 2.2e-16

# (Intercept)   -0.2406     0.0195   -12.3   <2e-16 ***
# Queries        1.2558     0.0791    15.9   <2e-16 ***
# log(ILILag2)   0.6557     0.0225    29.1   <2e-16 ***
# Multiple R-squared:  0.906

What is the R^2 value of the FluTrend2 model?

# Multiple R-squared:  0.906

4.4

On the basis of R-squared value and significance of coefficients, which statement is the most accurate?

• Due to overfitting, FluTrend2 is a weaker model then FluTrend1 on the training set.
• FluTrend2 is about the same quality as FluTrend1 on the training set.
• FluTrend2 is a stronger model than FluTrend1 on the training set.

# FluTrend2 is stronger

Section 5 - Evaluating the Time Series Model in the Test Set

5.1

So far, we have only added the ILILag2 variable to the FluTrain data frame. To make predictions with our FluTrend2 model, we will also need to add ILILag2 to the FluTest data frame (note that adding variables before splitting into a training and testing set can prevent this duplication of effort).

Modify the code from the previous subproblem to add an ILILag2 variable to the FluTest data frame. How many missing values are there in this new variable?

TS$ILILag2 = dplyr::lag(TS$ILI,2)
is.na(TS$ILILag2) %>% sum    # 2

Error in is.na(TS$ILILag2) %>% sum : could not find function "%>%"

5.2

In this problem, the training and testing sets are split sequentially – the training set contains all observations from 2004-2011 and the testing set contains all observations from 2012. There is no time gap between the two datasets, meaning the first observation in FluTest was recorded one week after the last observation in FluTrain. From this, we can identify how to fill in the missing values for the ILILag2 variable in FluTest.

Which value should be used to fill in the ILILag2 variable for the first observation in FluTest?

• The ILI value of the second-to-last observation in the FluTrain data frame.
• The ILI value of the last observation in the FluTrain data frame.
• The ILI value of the first observation in the FluTest data frame.
• The ILI value of the second observation in the FluTest data frame.

#The ILI value of the second-to-last observation in the FluTrain data frame

Which value should be used to fill in the ILILag2 variable for the second observation in FluTest?

• The ILI value of the second-to-last observation in the FluTrain data frame.
• The ILI value of the last observation in the FluTrain data frame.
• The ILI value of the first observation in the FluTest data frame.
• The ILI value of the second observation in the FluTest data frame.

# The ILI value of the last observation in the FluTrain data frame.

5.3

Fill in the missing values for ILILag2 in FluTest. In terms of syntax, you could set the value of ILILag2 in row “x” of the FluTest data frame to the value of ILI in row “y” of the FluTrain data frame with “FluTest\(ILILag2[x] = FluTrain\)ILI[y]”. Use the answer to the previous questions to determine the appropriate values of “x” and “y”. It may be helpful to check the total number of rows in FluTrain using str(FluTrain) or nrow(FluTrain).

What is the new value of the ILILag2 variable in the first row of FluTest?

TS$ILILag2[1:2] = TR$ILI[416:417]
TS$ILILag2[1:2]  # 1.8527, 2.1241

[1] 1.852736 2.124130

What is the new value of the ILILag2 variable in the second row of FluTest?

#2.1241

5.4

Obtain test set predictions of the ILI variable from the FluTrend2 model, again remembering to call the exp() function on the result of the predict() function to obtain predictions for ILI instead of log(ILI).

What is the test-set RMSE of the FluTrend2 model?

pred2 = exp(predict(FluTrend2,TS))
(SSE = sum( (TS$ILI - pred2)^2 )) # 4.5009

[1] 4.500877

(RMSE = sqrt( SSE/nrow(TS) ) )    # 0.2942

[1] 0.2942029

5.5

Which model obtained the best test-set RMSE?

# FluTrend2

In this problem, we used a simple time series model with a single lag term. ARIMA models are a more general form of the model we built, which can include multiple lag terms as well as more complicated combinations of previous values of the dependent variable. If you’re interested in learning more, check out ?arima or the available online tutorials for these sorts of models.