Logistic regression for a simple DCE
Michael Choi
22/09/2020
Introduction
This report is a supplement to the honors project ‘Designing a mobile phone visual messaging intervention to support healthy behavior change among patients undergoing pulmonary and cardiovascular rehabilitation: a discrete choice experiment’. # What is a discrete choice experiment? A discrete choice experiment (DCE) is an attribute-based survey method which measures a participant’s preferences. The surveys are comprised of several choice sets. Each choice set contains options to choose from. Respondents choose the option they prefer. Each option is differentiated from each other by certain attributes which vary along levels. Levels are simply the range over which attributes can vary. When a respondent chooses an option, their preference for the option’s underlying attributes is revealed. This is the underlying assumption of the DCE; that individuals derive utility from the attributes of a product rather than the product itself. Therefore, an individuals preferences for certain attributes can be estimated from their choices. # Aim The aim of this report is two-fold; 1) to generate a suitable design for the discrete choice experiment, 2) To analyze the results from the discrete choice experiment. # 1: Generating a suitable design ## 1.1 Initial setup using Rstudio The ‘idefix’ is a R package which implements a D-efficient approach in generating optimal designs for discrete choice experiments. We want to install this package from: https://cran.r-project.org/web/packages/idefix/index.html 1. install.packages(“idefix”, repos = “http://cran.us.r-project.org”) 2. install.packages(“stringi”) Now we want to load the ‘idefix’ package into R.
library("idefix")## Loading required package: shiny1.2 Specifying the design of the DCE
This DCE will employ 3 attributes with 4, 2 and 2 levels respectively. This means that the total number of choices is 4 x 2 x 2 = 16. We want to use a full factorial design for this DCE so we will construct 16 choice sets. Each choice set will consist of two options. Each option has a different attribute-level combination. ## 1.3 Creating profiles to match the specified design of the DCE {#1.3} In the vector at.lvls, the number of elements indicates the number of attributes (here we have 3 elements for the 3 attributes). The numeric values of this vector indicate the number of levels each attribute contains. In this example, there are three attributes, where the first one has four, the second one has two and the last one has two levels.
at.lvls <- c(4, 2, 2)Now we need to specify the type of coding for each attribute. In the vector c.type, one character is assigned for each attribute. Attributes can be effects coded “E”, dummy coded “D” or treated as a continuous variable “C”. We chose effects coding because it is widely accepted as superior method to dummy coding when you have categorical variables.
c.type <- c("E", "E", "E")Now that we have specified our attributes, levels and type of coding, we can generate profiles i.e. combinations of attribute levels. The Profiles function will be used. It has two arguments: 1. ‘lvls’ - This refers to the number of levels and we set this as the attribute levels as specified in the vector ‘at.lvls’ 2. ‘coding’ - This refers to the type of coding and we set this as the effects coding as specified in the vector ‘c.type’
cs <- Profiles(lvls=at.lvls, coding = c.type)1.4 Multivariate prior distribution {#1.4}
A multivariate normal distribution is an extension of a univariate normal distribution to two or more variables. In this study we have more than two variables. For every multivariate normal distribution, two parameters can be defined: a mean vector ‘µ’ and a covariance matrix ‘sigma’.
The mean vector mu, contains 5 elements. The first three are the parameters for the first attribute, the fourth is the parameter for the second attribute and the fifth is the parameter for the third attribute.
mu <- c(0, 0, 0, 0, 0)The covariance matrix sigma, is based off the mean vector.
sigma <- diag(length(mu)) The set.seed function is used for pseudo-random number generation. The starting point is set at 123, a random number.
set.seed(123)The multivariate normal distribution is simulated by by the function mvrnorm which has three arguments. 1. ‘n’ - This refers to the number of iterations and we set it at 500. 2. ‘mu’ - This refers to the mean vector and we set it to the ‘mu’ vector we have coded before. 3. ‘Sigma’ - This refers to the covariance matrix which we set to the ‘sigma’ vector.
M <- MASS::mvrnorm(n = 500, mu = mu, Sigma = sigma)1.5 Producing the design matrix {#1.5}
The design matrix is generated by the ‘Modfed’ function (idefix package) which has 3 arguments: 1. ‘cand.set’ - This refers to design matrix and we set it the ‘cs’ matrix we have crafted previously [#1.3] 2. ‘n.sets’ - This refers to the number of choice sets and since we are employing a full factorial design, we set this at 16 3. ‘par.draws’ - This specifies a matrix in which each row is a draw from a multivariate prior distribution We set this to the ‘M’ matrix as specified earlier [#1.4]
D <- Modfed(cand.set = cs, n.sets = 16, n.alts = 2, par.draws = M)By default, the Modfed funciton will swap profiles infinitely until the design with the lowest D(B) error is reached. At this point the Modfed function stops. The output of the Modefed function consists of four items. 1. ‘design’ - This contains the optimised design that resulted in the lowest D(B) error. It is a matrix where each row is a single alternative. 2. ‘error’ - This specifies the value of the lowest D(B) error. 3. ‘inf.error’ - This specifies the percentage of draws for which the design resulted in an infinite D-error. This percentage should be as close to zero. 4. ‘probs’ - This shows the average probabilities for each alternative in each choice set given the sample from the prior preference distribution in ‘par.draws’. Now we run the Modfed function. By default, the algorithm will swap profiles infinitely until the design with the lowest D(B) error is reached. At this point the Modfed function stops. The output of the Modefed function consists of four items. 1. ‘design’ - This contains the optimised design that resulted in the lowest D(B) error. It is a matrix where each row is a single alternative. 2. ‘error’ - This specifies the value of the lowest D(B) error. 3. ‘inf.error’ - This specifies the percentage of draws for which the design resulted in an infinite D-error. This percentage should be as close to zero. 4. ‘probs’ - This shows the average probabilities for each alternative in each choice set given the sample from the prior preference distribution in ‘par.draws’.
head(D)## $design
## Var11 Var12 Var13 Var21 Var31
## set1.alt1 0 0 1 -1 1
## set1.alt2 1 0 0 1 -1
## set2.alt1 0 1 0 1 -1
## set2.alt2 1 0 0 -1 1
## set3.alt1 0 1 0 1 1
## set3.alt2 1 0 0 -1 -1
## set4.alt1 0 0 1 -1 -1
## set4.alt2 0 1 0 -1 -1
## set5.alt1 -1 -1 -1 1 1
## set5.alt2 0 0 1 -1 1
## set6.alt1 1 0 0 -1 -1
## set6.alt2 0 0 1 -1 -1
## set7.alt1 1 0 0 -1 1
## set7.alt2 0 1 0 -1 -1
## set8.alt1 0 0 1 1 1
## set8.alt2 -1 -1 -1 -1 -1
## set9.alt1 1 0 0 1 1
## set9.alt2 0 0 1 -1 -1
## set10.alt1 0 1 0 -1 -1
## set10.alt2 -1 -1 -1 1 1
## set11.alt1 0 0 1 1 -1
## set11.alt2 -1 -1 -1 -1 1
## set12.alt1 -1 -1 -1 -1 1
## set12.alt2 0 1 0 1 1
## set13.alt1 0 0 1 1 -1
## set13.alt2 0 1 0 -1 -1
## set14.alt1 -1 -1 -1 1 -1
## set14.alt2 1 0 0 1 -1
## set15.alt1 0 1 0 -1 1
## set15.alt2 1 0 0 -1 -1
## set16.alt1 -1 -1 -1 1 -1
## set16.alt2 0 1 0 -1 1
##
## $error
## [1] 0.4640716
##
## $inf.error
## [1] 0
##
## $probs
## Pr(alt1) Pr(alt2)
## set1 0.5112707 0.4887293
## set2 0.5057882 0.4942118
## set3 0.5207536 0.4792464
## set4 0.4919768 0.5080232
## set5 0.4994358 0.5005642
## set6 0.4817120 0.5182880
## set7 0.4958301 0.5041699
## set8 0.5049350 0.4950650
## set9 0.4917687 0.5082313
## set10 0.5062209 0.4937791
## set11 0.5013411 0.4986589
## set12 0.4848646 0.5151354
## set13 0.4896827 0.5103173
## set14 0.5052289 0.4947711
## set15 0.5224816 0.4775184
## set16 0.4935506 0.50644941.6 Making the design matrix readable
We specify all attribute levels in ‘lvls’. So the first attribute is image purpose with four different levels: positive gain, negative loss, educational, no purpose. The second attribute is the image realism with two different levels: real image and not a real image. The third attribute represents whether the image offers a link to an external website: with a link and without a link.
lvls <- list(c("Gain", "Loss", "Edu", "No"), c("Real", "Not"), c("With", "W/out"))To make the design matrix [#1.5] readable, as they would appear in a real survey, we use the Decode function. The Decode function has three arguments: 1. ‘des’ - This is where we specify the optimized design that resulted in the lowest D(B) error from the output of the Modfed function. 2. ‘lvl.names’ - This is where we relate the attribute names to the optimal design. 3. ‘coding’ - Here we specify the same type of coding used to generate the design matrix (1.1) 4. ‘n.alts’ - Here we specify the number of options.
DD <- Decode (des=D$design, lvl.names = lvls, coding = c.type, n.alts=2)The output of Decode has two components: 1. ‘design’ - Shows the design matrix with the labels attached. 2. ‘lvl.balance’ - This shows the frequency of each attribute level in the design.
DD## $design
## V1 V2 V3
## set1.alt1 Edu Not With
## set1.alt2 Gain Real W/out
## set2.alt1 Loss Real W/out
## set2.alt2 Gain Not With
## set3.alt1 Loss Real With
## set3.alt2 Gain Not W/out
## set4.alt1 Edu Not W/out
## set4.alt2 Loss Not W/out
## set5.alt1 No Real With
## set5.alt2 Edu Not With
## set6.alt1 Gain Not W/out
## set6.alt2 Edu Not W/out
## set7.alt1 Gain Not With
## set7.alt2 Loss Not W/out
## set8.alt1 Edu Real With
## set8.alt2 No Not W/out
## set9.alt1 Gain Real With
## set9.alt2 Edu Not W/out
## set10.alt1 Loss Not W/out
## set10.alt2 No Real With
## set11.alt1 Edu Real W/out
## set11.alt2 No Not With
## set12.alt1 No Not With
## set12.alt2 Loss Real With
## set13.alt1 Edu Real W/out
## set13.alt2 Loss Not W/out
## set14.alt1 No Real W/out
## set14.alt2 Gain Real W/out
## set15.alt1 Loss Not With
## set15.alt2 Gain Not W/out
## set16.alt1 No Real W/out
## set16.alt2 Loss Not With
##
## $lvl.balance
## $lvl.balance$`attribute 1 `
##
## Edu Gain Loss No
## 8 8 9 7
##
## $lvl.balance$`attribute 2 `
##
## Not Real
## 19 13
##
## $lvl.balance$`attribute 3 `
##
## W/out With
## 18 142 Analysing the data
We have captured participant choices using an online platform called REDcap. This data was exported to an excel file where the data was cleaned. The data was then combined with the labeled design matrix in excel. This new file was then imported into R as a .csv file to undergo data analysis.
2.1 Export the design of the experiment
We extract the labeled design and encode into a separate .csv file
codedesign <- DD$design
write.csv(codedesign, 'codedesign.csv')The ‘codedesign.csv’ file is then combined with the participant data in excel. (not shown here)
2.2 Load the data file
We want to read in the .csv file which contains the choice data and the survey design
surveydata <- read.csv("surveydata-1.csv")Let’s have a look at the rows and columns of surveydata. Here we can see that we have 6 columns. From left to right: - ID: the participant number - set.alt: the alternative. (there are two alternatives for each set and there are 16 sets) - V1: variable 1 is the image purpose with three levels (education, gain frame, loss frame) - V2: variable 2 is the image type with two levels (not real, real) - V3: variable 3 is the whether the image has a URL link (with, without) - choice: this is a binary vector which indicates whether this option was chosen.
There are 32 observations for 41 participants. 41 * 32 = 1312 In total there are 1312 rows.
head(surveydata)## ID set.alt V1 V2 V3 choice
## 1 1 set1.alt1 Edu Not With 0
## 2 1 set1.alt2 Gain Real W/out 1
## 3 1 set2.alt1 Loss Real W/out 0
## 4 1 set2.alt2 Gain Not With 1
## 5 1 set3.alt1 Loss Real With 0
## 6 1 set3.alt2 Gain Not W/out 12.3 Setup of Data analysis
We need three packages to conduct a logistic regression: 1. install.packages(“mlogit”) 2. install.packages(“aod”) 3. install.packages(“ggplot2”) Then we will load the packages into the library.
library(mlogit)## Loading required package: dfidx##
## Attaching package: 'dfidx'## The following object is masked from 'package:stats':
##
## filterlibrary(aod)
library(ggplot2)2.4 Logistic regression
We need to change the ‘V1’ attribute from a character type variable to a factor type variable. The reason why we need to do this is because we need to tell R which level is the reference variable. In this case, the ‘No’ level is the reference level.
surveydata$V1 = as.factor(surveydata$V1)
surveydata$V1 <- relevel(surveydata$V1, ref = "No")
surveydata$V2 = as.factor(surveydata$V2)
surveydata$V2 <- relevel(surveydata$V2, ref = "Not")
surveydata$V3 = as.factor(surveydata$V3)
surveydata$V3 <- relevel(surveydata$V3, ref = "W/out")
str(surveydata)## 'data.frame': 1312 obs. of 6 variables:
## $ ID : int 1 1 1 1 1 1 1 1 1 1 ...
## $ set.alt: chr "set1.alt1" "set1.alt2" "set2.alt1" "set2.alt2" ...
## $ V1 : Factor w/ 4 levels "No","Edu","Gain",..: 2 3 4 3 4 3 2 4 1 2 ...
## $ V2 : Factor w/ 2 levels "Not","Real": 1 2 2 1 2 1 1 1 2 1 ...
## $ V3 : Factor w/ 2 levels "W/out","With": 2 1 1 2 2 1 1 1 2 2 ...
## $ choice : int 0 1 0 1 0 1 1 0 1 0 ...levels(surveydata$V1)## [1] "No" "Edu" "Gain" "Loss"levels(surveydata$V2)## [1] "Not" "Real"levels(surveydata$V3)## [1] "W/out" "With"Now lets perform the logistic regression using the generalised linear model (glm) function which has three arguments: 1. The model - ‘choice’ is the variable we are interested in and how it is affected by V1, V2 and V3 2. ‘data’ - Here we read in the ‘surveydata’ data frame. 3. ‘family’ - Here we set it as binomial - not really sure why
logit1 <- glm(choice ~ V1 + V2 + V3, data=surveydata, family = "binomial")
summary(logit1)##
## Call:
## glm(formula = choice ~ V1 + V2 + V3, family = "binomial", data = surveydata)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -1.72511 -1.12528 -0.01313 1.10121 1.68822
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) -0.2761 0.1529 -1.805 0.0710 .
## V1Edu 0.1522 0.1660 0.917 0.3592
## V1Gain 0.9144 0.1736 5.266 1.39e-07 ***
## V1Loss -0.8739 0.1678 -5.209 1.90e-07 ***
## V2Real 0.3058 0.1202 2.543 0.0110 *
## V3With 0.2880 0.1191 2.418 0.0156 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 1818.8 on 1311 degrees of freedom
## Residual deviance: 1681.7 on 1306 degrees of freedom
## AIC: 1693.7
##
## Number of Fisher Scoring iterations: 4In the output above, we see a number of things. 1. ‘Call’ - This refers to what model we ran. Its simply a reminder of what we specified. 2. ‘Deviance Residuals’ - This is a measure of model fit and will be discussed later. 3. ‘Coefficients’ - This shows the coefficients, their standard errors, the z-statistic and the associated p-values. As indicated by the asterisks, ‘V1Gain’, ‘V1Loss, ’V2Real’ andn ‘V3With’ are statistically significant. So the coefficients refer to the change in the log odds of the outcome for a one unit increase in the predictor variable. For example, for every one unit change in ‘V1Gain’, the log odds of choice increases by 0.91. 4. ‘Deviance values’ - These include the null and deviance residulas and the AIC. These will be discussed later to help assess model fit.
##2.5 Confidence intervals We can use the ‘confint’ function to obtain confidence intervals for the coefficient estimates produced above. We can use two methods to obtain the confidence intervals; 1) using profiled log-likelihood function, 2) using standard errors (default)
confint(logit1)## Waiting for profiling to be done...## 2.5 % 97.5 %
## (Intercept) -0.57660654 0.02326005
## V1Edu -0.17295600 0.47806244
## V1Gain 0.57596039 1.25700387
## V1Loss -1.20458155 -0.54648102
## V2Real 0.07043758 0.54189821
## V3With 0.05498813 0.52210133And confidence intervals with standard errors
confint.default(logit1)## 2.5 % 97.5 %
## (Intercept) -0.57574876 0.02361892
## V1Edu -0.17314819 0.47751055
## V1Gain 0.57405672 1.25465474
## V1Loss -1.20272770 -0.54504238
## V2Real 0.07013047 0.54140876
## V3With 0.05452285 0.52145501##2.6 Testing for overall effect for V1 Test for an overall effect of V1 using the ‘wald.test’ function from the ‘aod’ library. This function contains three arguments: 1. ‘b’ - This refers to the coefficients from the logistic regression 2. ‘Sigma’ - This refers to the variance covariance matrix of the error terms which was generated by the logistic regression 3. ‘Terms’ - This tells R which terms in the model are to be tested. In this case, terms 1, 2, 3, 4 (V1No, V1Gain, V1Loss, V1Edu) are the 4 levels of V1.
wald.test(b = coef(logit1), Sigma = vcov(logit1), Terms = 1:4)## Wald test:
## ----------
##
## Chi-squared test:
## X2 = 118.2, df = 4, P(> X2) = 0.0The chi-squared statistic of 118.2, with 4 degrees of freedom is associated with a p-value of 0.0 indicating that the overall effect of V1 is statistically significant.
We can also test whether differences in the coefficients within V1 are statistically significant. Here we test whether the coefficient for V1Gain is equal to the coefficient for V1Loss. First, we specify a vector ‘l’ that defines the test we want to perform. In this case, we want to test the difference (subtraction) of the terms for V1Gain and V1Loss (the 4th and 5th terms in the model). To contrast these two terms, we multiply one of them by 1 and the other by -1. The other terms in the model are not involved in the test, so they are multipled by 0.
l <- cbind(0, 0, 0, 1, -1, 0)Then, we use L=1 as the third argument in the wald.test function to tell R that we wish to base the test on the vector l rather than using the Terms option as we did before.
wald.test(b = coef(logit1), Sigma = vcov(logit1), L = l)## Wald test:
## ----------
##
## Chi-squared test:
## X2 = 37.8, df = 1, P(> X2) = 7.9e-10The chi-squared test statistic of 37.8 with 1 degree of freedom is associated with a p-value of 7.9e-10, indicating that the difference between the coefficient for V1Gain and the coefficient for V1Loss is statistically significant. ##2.7 Odds Ratio We can also exponentiate the coefficients and interpret them as odds-ratio. We use the function ‘exp’ and tell R that the object that we want to exponentiate is called coefficients and that its a part of your logistic regression.
exp(coef(logit1))## (Intercept) V1Edu V1Gain V1Loss V2Real V3With
## 0.7587637 1.1643712 2.4951672 0.4173271 1.3576695 1.3337425We can display this in one table using the ‘cbind’ function. So we have the coefficients and the confidence-intervals side by side.
exp(cbind(OR = coef(logit1), confint(logit1)))## Waiting for profiling to be done...## OR 2.5 % 97.5 %
## (Intercept) 0.7587637 0.5618016 1.0235327
## V1Edu 1.1643712 0.8411746 1.6129462
## V1Gain 2.4951672 1.7788381 3.5148747
## V1Loss 0.4173271 0.2998174 0.5789837
## V2Real 1.3576695 1.0729776 1.7192673
## V3With 1.3337425 1.0565281 1.6855659We can say that for every image that is ‘gain-framed’, the odds of a participant selecting that image (versus not selecting that image) increase by a factor of 2.49. ##2.8 Predicted probabilities We can also use predicted probabilities to help us understand the model. In order to do this we need to create a new data frame with the values we want as independent variables.
We will start by calculating the predicted probability of choice at each value of V1, varying V2 and V3
predict1 <- with(surveydata, data.frame(V2, V3, V1 ))
predict1## V2 V3 V1
## 1 Not With Edu
## 2 Real W/out Gain
## 3 Real W/out Loss
## 4 Not With Gain
## 5 Real With Loss
## 6 Not W/out Gain
## 7 Not W/out Edu
## 8 Not W/out Loss
## 9 Real With No
## 10 Not With Edu
## 11 Not W/out Gain
## 12 Not W/out Edu
## 13 Not With Gain
## 14 Not W/out Loss
## 15 Real With Edu
## 16 Not W/out No
## 17 Real With Gain
## 18 Not W/out Edu
## 19 Not W/out Loss
## 20 Real With No
## 21 Real W/out Edu
## 22 Not With No
## 23 Not With No
## 24 Real With Loss
## 25 Real W/out Edu
## 26 Not W/out Loss
## 27 Real W/out No
## 28 Real W/out Gain
## 29 Not With Loss
## 30 Not W/out Gain
## 31 Real W/out No
## 32 Not With Loss
## 33 Not With Edu
## 34 Real W/out Gain
## 35 Real W/out Loss
## 36 Not With Gain
## 37 Real With Loss
## 38 Not W/out Gain
## 39 Not W/out Edu
## 40 Not W/out Loss
## 41 Real With No
## 42 Not With Edu
## 43 Not W/out Gain
## 44 Not W/out Edu
## 45 Not With Gain
## 46 Not W/out Loss
## 47 Real With Edu
## 48 Not W/out No
## 49 Real With Gain
## 50 Not W/out Edu
## 51 Not W/out Loss
## 52 Real With No
## 53 Real W/out Edu
## 54 Not With No
## 55 Not With No
## 56 Real With Loss
## 57 Real W/out Edu
## 58 Not W/out Loss
## 59 Real W/out No
## 60 Real W/out Gain
## 61 Not With Loss
## 62 Not W/out Gain
## 63 Real W/out No
## 64 Not With Loss
## 65 Not With Edu
## 66 Real W/out Gain
## 67 Real W/out Loss
## 68 Not With Gain
## 69 Real With Loss
## 70 Not W/out Gain
## 71 Not W/out Edu
## 72 Not W/out Loss
## 73 Real With No
## 74 Not With Edu
## 75 Not W/out Gain
## 76 Not W/out Edu
## 77 Not With Gain
## 78 Not W/out Loss
## 79 Real With Edu
## 80 Not W/out No
## 81 Real With Gain
## 82 Not W/out Edu
## 83 Not W/out Loss
## 84 Real With No
## 85 Real W/out Edu
## 86 Not With No
## 87 Not With No
## 88 Real With Loss
## 89 Real W/out Edu
## 90 Not W/out Loss
## 91 Real W/out No
## 92 Real W/out Gain
## 93 Not With Loss
## 94 Not W/out Gain
## 95 Real W/out No
## 96 Not With Loss
## 97 Not With Edu
## 98 Real W/out Gain
## 99 Real W/out Loss
## 100 Not With Gain
## 101 Real With Loss
## 102 Not W/out Gain
## 103 Not W/out Edu
## 104 Not W/out Loss
## 105 Real With No
## 106 Not With Edu
## 107 Not W/out Gain
## 108 Not W/out Edu
## 109 Not With Gain
## 110 Not W/out Loss
## 111 Real With Edu
## 112 Not W/out No
## 113 Real With Gain
## 114 Not W/out Edu
## 115 Not W/out Loss
## 116 Real With No
## 117 Real W/out Edu
## 118 Not With No
## 119 Not With No
## 120 Real With Loss
## 121 Real W/out Edu
## 122 Not W/out Loss
## 123 Real W/out No
## 124 Real W/out Gain
## 125 Not With Loss
## 126 Not W/out Gain
## 127 Real W/out No
## 128 Not With Loss
## 129 Not With Edu
## 130 Real W/out Gain
## 131 Real W/out Loss
## 132 Not With Gain
## 133 Real With Loss
## 134 Not W/out Gain
## 135 Not W/out Edu
## 136 Not W/out Loss
## 137 Real With No
## 138 Not With Edu
## 139 Not W/out Gain
## 140 Not W/out Edu
## 141 Not With Gain
## 142 Not W/out Loss
## 143 Real With Edu
## 144 Not W/out No
## 145 Real With Gain
## 146 Not W/out Edu
## 147 Not W/out Loss
## 148 Real With No
## 149 Real W/out Edu
## 150 Not With No
## 151 Not With No
## 152 Real With Loss
## 153 Real W/out Edu
## 154 Not W/out Loss
## 155 Real W/out No
## 156 Real W/out Gain
## 157 Not With Loss
## 158 Not W/out Gain
## 159 Real W/out No
## 160 Not With Loss
## 161 Not With Edu
## 162 Real W/out Gain
## 163 Real W/out Loss
## 164 Not With Gain
## 165 Real With Loss
## 166 Not W/out Gain
## 167 Not W/out Edu
## 168 Not W/out Loss
## 169 Real With No
## 170 Not With Edu
## 171 Not W/out Gain
## 172 Not W/out Edu
## 173 Not With Gain
## 174 Not W/out Loss
## 175 Real With Edu
## 176 Not W/out No
## 177 Real With Gain
## 178 Not W/out Edu
## 179 Not W/out Loss
## 180 Real With No
## 181 Real W/out Edu
## 182 Not With No
## 183 Not With No
## 184 Real With Loss
## 185 Real W/out Edu
## 186 Not W/out Loss
## 187 Real W/out No
## 188 Real W/out Gain
## 189 Not With Loss
## 190 Not W/out Gain
## 191 Real W/out No
## 192 Not With Loss
## 193 Not With Edu
## 194 Real W/out Gain
## 195 Real W/out Loss
## 196 Not With Gain
## 197 Real With Loss
## 198 Not W/out Gain
## 199 Not W/out Edu
## 200 Not W/out Loss
## 201 Real With No
## 202 Not With Edu
## 203 Not W/out Gain
## 204 Not W/out Edu
## 205 Not With Gain
## 206 Not W/out Loss
## 207 Real With Edu
## 208 Not W/out No
## 209 Real With Gain
## 210 Not W/out Edu
## 211 Not W/out Loss
## 212 Real With No
## 213 Real W/out Edu
## 214 Not With No
## 215 Not With No
## 216 Real With Loss
## 217 Real W/out Edu
## 218 Not W/out Loss
## 219 Real W/out No
## 220 Real W/out Gain
## 221 Not With Loss
## 222 Not W/out Gain
## 223 Real W/out No
## 224 Not With Loss
## 225 Not With Edu
## 226 Real W/out Gain
## 227 Real W/out Loss
## 228 Not With Gain
## 229 Real With Loss
## 230 Not W/out Gain
## 231 Not W/out Edu
## 232 Not W/out Loss
## 233 Real With No
## 234 Not With Edu
## 235 Not W/out Gain
## 236 Not W/out Edu
## 237 Not With Gain
## 238 Not W/out Loss
## 239 Real With Edu
## 240 Not W/out No
## 241 Real With Gain
## 242 Not W/out Edu
## 243 Not W/out Loss
## 244 Real With No
## 245 Real W/out Edu
## 246 Not With No
## 247 Not With No
## 248 Real With Loss
## 249 Real W/out Edu
## 250 Not W/out Loss
## 251 Real W/out No
## 252 Real W/out Gain
## 253 Not With Loss
## 254 Not W/out Gain
## 255 Real W/out No
## 256 Not With Loss
## 257 Not With Edu
## 258 Real W/out Gain
## 259 Real W/out Loss
## 260 Not With Gain
## 261 Real With Loss
## 262 Not W/out Gain
## 263 Not W/out Edu
## 264 Not W/out Loss
## 265 Real With No
## 266 Not With Edu
## 267 Not W/out Gain
## 268 Not W/out Edu
## 269 Not With Gain
## 270 Not W/out Loss
## 271 Real With Edu
## 272 Not W/out No
## 273 Real With Gain
## 274 Not W/out Edu
## 275 Not W/out Loss
## 276 Real With No
## 277 Real W/out Edu
## 278 Not With No
## 279 Not With No
## 280 Real With Loss
## 281 Real W/out Edu
## 282 Not W/out Loss
## 283 Real W/out No
## 284 Real W/out Gain
## 285 Not With Loss
## 286 Not W/out Gain
## 287 Real W/out No
## 288 Not With Loss
## 289 Not With Edu
## 290 Real W/out Gain
## 291 Real W/out Loss
## 292 Not With Gain
## 293 Real With Loss
## 294 Not W/out Gain
## 295 Not W/out Edu
## 296 Not W/out Loss
## 297 Real With No
## 298 Not With Edu
## 299 Not W/out Gain
## 300 Not W/out Edu
## 301 Not With Gain
## 302 Not W/out Loss
## 303 Real With Edu
## 304 Not W/out No
## 305 Real With Gain
## 306 Not W/out Edu
## 307 Not W/out Loss
## 308 Real With No
## 309 Real W/out Edu
## 310 Not With No
## 311 Not With No
## 312 Real With Loss
## 313 Real W/out Edu
## 314 Not W/out Loss
## 315 Real W/out No
## 316 Real W/out Gain
## 317 Not With Loss
## 318 Not W/out Gain
## 319 Real W/out No
## 320 Not With Loss
## 321 Not With Edu
## 322 Real W/out Gain
## 323 Real W/out Loss
## 324 Not With Gain
## 325 Real With Loss
## 326 Not W/out Gain
## 327 Not W/out Edu
## 328 Not W/out Loss
## 329 Real With No
## 330 Not With Edu
## 331 Not W/out Gain
## 332 Not W/out Edu
## 333 Not With Gain
## 334 Not W/out Loss
## 335 Real With Edu
## 336 Not W/out No
## 337 Real With Gain
## 338 Not W/out Edu
## 339 Not W/out Loss
## 340 Real With No
## 341 Real W/out Edu
## 342 Not With No
## 343 Not With No
## 344 Real With Loss
## 345 Real W/out Edu
## 346 Not W/out Loss
## 347 Real W/out No
## 348 Real W/out Gain
## 349 Not With Loss
## 350 Not W/out Gain
## 351 Real W/out No
## 352 Not With Loss
## 353 Not With Edu
## 354 Real W/out Gain
## 355 Real W/out Loss
## 356 Not With Gain
## 357 Real With Loss
## 358 Not W/out Gain
## 359 Not W/out Edu
## 360 Not W/out Loss
## 361 Real With No
## 362 Not With Edu
## 363 Not W/out Gain
## 364 Not W/out Edu
## 365 Not With Gain
## 366 Not W/out Loss
## 367 Real With Edu
## 368 Not W/out No
## 369 Real With Gain
## 370 Not W/out Edu
## 371 Not W/out Loss
## 372 Real With No
## 373 Real W/out Edu
## 374 Not With No
## 375 Not With No
## 376 Real With Loss
## 377 Real W/out Edu
## 378 Not W/out Loss
## 379 Real W/out No
## 380 Real W/out Gain
## 381 Not With Loss
## 382 Not W/out Gain
## 383 Real W/out No
## 384 Not With Loss
## 385 Not With Edu
## 386 Real W/out Gain
## 387 Real W/out Loss
## 388 Not With Gain
## 389 Real With Loss
## 390 Not W/out Gain
## 391 Not W/out Edu
## 392 Not W/out Loss
## 393 Real With No
## 394 Not With Edu
## 395 Not W/out Gain
## 396 Not W/out Edu
## 397 Not With Gain
## 398 Not W/out Loss
## 399 Real With Edu
## 400 Not W/out No
## 401 Real With Gain
## 402 Not W/out Edu
## 403 Not W/out Loss
## 404 Real With No
## 405 Real W/out Edu
## 406 Not With No
## 407 Not With No
## 408 Real With Loss
## 409 Real W/out Edu
## 410 Not W/out Loss
## 411 Real W/out No
## 412 Real W/out Gain
## 413 Not With Loss
## 414 Not W/out Gain
## 415 Real W/out No
## 416 Not With Loss
## 417 Not With Edu
## 418 Real W/out Gain
## 419 Real W/out Loss
## 420 Not With Gain
## 421 Real With Loss
## 422 Not W/out Gain
## 423 Not W/out Edu
## 424 Not W/out Loss
## 425 Real With No
## 426 Not With Edu
## 427 Not W/out Gain
## 428 Not W/out Edu
## 429 Not With Gain
## 430 Not W/out Loss
## 431 Real With Edu
## 432 Not W/out No
## 433 Real With Gain
## 434 Not W/out Edu
## 435 Not W/out Loss
## 436 Real With No
## 437 Real W/out Edu
## 438 Not With No
## 439 Not With No
## 440 Real With Loss
## 441 Real W/out Edu
## 442 Not W/out Loss
## 443 Real W/out No
## 444 Real W/out Gain
## 445 Not With Loss
## 446 Not W/out Gain
## 447 Real W/out No
## 448 Not With Loss
## 449 Not With Edu
## 450 Real W/out Gain
## 451 Real W/out Loss
## 452 Not With Gain
## 453 Real With Loss
## 454 Not W/out Gain
## 455 Not W/out Edu
## 456 Not W/out Loss
## 457 Real With No
## 458 Not With Edu
## 459 Not W/out Gain
## 460 Not W/out Edu
## 461 Not With Gain
## 462 Not W/out Loss
## 463 Real With Edu
## 464 Not W/out No
## 465 Real With Gain
## 466 Not W/out Edu
## 467 Not W/out Loss
## 468 Real With No
## 469 Real W/out Edu
## 470 Not With No
## 471 Not With No
## 472 Real With Loss
## 473 Real W/out Edu
## 474 Not W/out Loss
## 475 Real W/out No
## 476 Real W/out Gain
## 477 Not With Loss
## 478 Not W/out Gain
## 479 Real W/out No
## 480 Not With Loss
## 481 Not With Edu
## 482 Real W/out Gain
## 483 Real W/out Loss
## 484 Not With Gain
## 485 Real With Loss
## 486 Not W/out Gain
## 487 Not W/out Edu
## 488 Not W/out Loss
## 489 Real With No
## 490 Not With Edu
## 491 Not W/out Gain
## 492 Not W/out Edu
## 493 Not With Gain
## 494 Not W/out Loss
## 495 Real With Edu
## 496 Not W/out No
## 497 Real With Gain
## 498 Not W/out Edu
## 499 Not W/out Loss
## 500 Real With No
## 501 Real W/out Edu
## 502 Not With No
## 503 Not With No
## 504 Real With Loss
## 505 Real W/out Edu
## 506 Not W/out Loss
## 507 Real W/out No
## 508 Real W/out Gain
## 509 Not With Loss
## 510 Not W/out Gain
## 511 Real W/out No
## 512 Not With Loss
## 513 Not With Edu
## 514 Real W/out Gain
## 515 Real W/out Loss
## 516 Not With Gain
## 517 Real With Loss
## 518 Not W/out Gain
## 519 Not W/out Edu
## 520 Not W/out Loss
## 521 Real With No
## 522 Not With Edu
## 523 Not W/out Gain
## 524 Not W/out Edu
## 525 Not With Gain
## 526 Not W/out Loss
## 527 Real With Edu
## 528 Not W/out No
## 529 Real With Gain
## 530 Not W/out Edu
## 531 Not W/out Loss
## 532 Real With No
## 533 Real W/out Edu
## 534 Not With No
## 535 Not With No
## 536 Real With Loss
## 537 Real W/out Edu
## 538 Not W/out Loss
## 539 Real W/out No
## 540 Real W/out Gain
## 541 Not With Loss
## 542 Not W/out Gain
## 543 Real W/out No
## 544 Not With Loss
## 545 Not With Edu
## 546 Real W/out Gain
## 547 Real W/out Loss
## 548 Not With Gain
## 549 Real With Loss
## 550 Not W/out Gain
## 551 Not W/out Edu
## 552 Not W/out Loss
## 553 Real With No
## 554 Not With Edu
## 555 Not W/out Gain
## 556 Not W/out Edu
## 557 Not With Gain
## 558 Not W/out Loss
## 559 Real With Edu
## 560 Not W/out No
## 561 Real With Gain
## 562 Not W/out Edu
## 563 Not W/out Loss
## 564 Real With No
## 565 Real W/out Edu
## 566 Not With No
## 567 Not With No
## 568 Real With Loss
## 569 Real W/out Edu
## 570 Not W/out Loss
## 571 Real W/out No
## 572 Real W/out Gain
## 573 Not With Loss
## 574 Not W/out Gain
## 575 Real W/out No
## 576 Not With Loss
## 577 Not With Edu
## 578 Real W/out Gain
## 579 Real W/out Loss
## 580 Not With Gain
## 581 Real With Loss
## 582 Not W/out Gain
## 583 Not W/out Edu
## 584 Not W/out Loss
## 585 Real With No
## 586 Not With Edu
## 587 Not W/out Gain
## 588 Not W/out Edu
## 589 Not With Gain
## 590 Not W/out Loss
## 591 Real With Edu
## 592 Not W/out No
## 593 Real With Gain
## 594 Not W/out Edu
## 595 Not W/out Loss
## 596 Real With No
## 597 Real W/out Edu
## 598 Not With No
## 599 Not With No
## 600 Real With Loss
## 601 Real W/out Edu
## 602 Not W/out Loss
## 603 Real W/out No
## 604 Real W/out Gain
## 605 Not With Loss
## 606 Not W/out Gain
## 607 Real W/out No
## 608 Not With Loss
## 609 Not With Edu
## 610 Real W/out Gain
## 611 Real W/out Loss
## 612 Not With Gain
## 613 Real With Loss
## 614 Not W/out Gain
## 615 Not W/out Edu
## 616 Not W/out Loss
## 617 Real With No
## 618 Not With Edu
## 619 Not W/out Gain
## 620 Not W/out Edu
## 621 Not With Gain
## 622 Not W/out Loss
## 623 Real With Edu
## 624 Not W/out No
## 625 Real With Gain
## 626 Not W/out Edu
## 627 Not W/out Loss
## 628 Real With No
## 629 Real W/out Edu
## 630 Not With No
## 631 Not With No
## 632 Real With Loss
## 633 Real W/out Edu
## 634 Not W/out Loss
## 635 Real W/out No
## 636 Real W/out Gain
## 637 Not With Loss
## 638 Not W/out Gain
## 639 Real W/out No
## 640 Not With Loss
## 641 Not With Edu
## 642 Real W/out Gain
## 643 Real W/out Loss
## 644 Not With Gain
## 645 Real With Loss
## 646 Not W/out Gain
## 647 Not W/out Edu
## 648 Not W/out Loss
## 649 Real With No
## 650 Not With Edu
## 651 Not W/out Gain
## 652 Not W/out Edu
## 653 Not With Gain
## 654 Not W/out Loss
## 655 Real With Edu
## 656 Not W/out No
## 657 Real With Gain
## 658 Not W/out Edu
## 659 Not W/out Loss
## 660 Real With No
## 661 Real W/out Edu
## 662 Not With No
## 663 Not With No
## 664 Real With Loss
## 665 Real W/out Edu
## 666 Not W/out Loss
## 667 Real W/out No
## 668 Real W/out Gain
## 669 Not With Loss
## 670 Not W/out Gain
## 671 Real W/out No
## 672 Not With Loss
## 673 Not With Edu
## 674 Real W/out Gain
## 675 Real W/out Loss
## 676 Not With Gain
## 677 Real With Loss
## 678 Not W/out Gain
## 679 Not W/out Edu
## 680 Not W/out Loss
## 681 Real With No
## 682 Not With Edu
## 683 Not W/out Gain
## 684 Not W/out Edu
## 685 Not With Gain
## 686 Not W/out Loss
## 687 Real With Edu
## 688 Not W/out No
## 689 Real With Gain
## 690 Not W/out Edu
## 691 Not W/out Loss
## 692 Real With No
## 693 Real W/out Edu
## 694 Not With No
## 695 Not With No
## 696 Real With Loss
## 697 Real W/out Edu
## 698 Not W/out Loss
## 699 Real W/out No
## 700 Real W/out Gain
## 701 Not With Loss
## 702 Not W/out Gain
## 703 Real W/out No
## 704 Not With Loss
## 705 Not With Edu
## 706 Real W/out Gain
## 707 Real W/out Loss
## 708 Not With Gain
## 709 Real With Loss
## 710 Not W/out Gain
## 711 Not W/out Edu
## 712 Not W/out Loss
## 713 Real With No
## 714 Not With Edu
## 715 Not W/out Gain
## 716 Not W/out Edu
## 717 Not With Gain
## 718 Not W/out Loss
## 719 Real With Edu
## 720 Not W/out No
## 721 Real With Gain
## 722 Not W/out Edu
## 723 Not W/out Loss
## 724 Real With No
## 725 Real W/out Edu
## 726 Not With No
## 727 Not With No
## 728 Real With Loss
## 729 Real W/out Edu
## 730 Not W/out Loss
## 731 Real W/out No
## 732 Real W/out Gain
## 733 Not With Loss
## 734 Not W/out Gain
## 735 Real W/out No
## 736 Not With Loss
## 737 Not With Edu
## 738 Real W/out Gain
## 739 Real W/out Loss
## 740 Not With Gain
## 741 Real With Loss
## 742 Not W/out Gain
## 743 Not W/out Edu
## 744 Not W/out Loss
## 745 Real With No
## 746 Not With Edu
## 747 Not W/out Gain
## 748 Not W/out Edu
## 749 Not With Gain
## 750 Not W/out Loss
## 751 Real With Edu
## 752 Not W/out No
## 753 Real With Gain
## 754 Not W/out Edu
## 755 Not W/out Loss
## 756 Real With No
## 757 Real W/out Edu
## 758 Not With No
## 759 Not With No
## 760 Real With Loss
## 761 Real W/out Edu
## 762 Not W/out Loss
## 763 Real W/out No
## 764 Real W/out Gain
## 765 Not With Loss
## 766 Not W/out Gain
## 767 Real W/out No
## 768 Not With Loss
## 769 Not With Edu
## 770 Real W/out Gain
## 771 Real W/out Loss
## 772 Not With Gain
## 773 Real With Loss
## 774 Not W/out Gain
## 775 Not W/out Edu
## 776 Not W/out Loss
## 777 Real With No
## 778 Not With Edu
## 779 Not W/out Gain
## 780 Not W/out Edu
## 781 Not With Gain
## 782 Not W/out Loss
## 783 Real With Edu
## 784 Not W/out No
## 785 Real With Gain
## 786 Not W/out Edu
## 787 Not W/out Loss
## 788 Real With No
## 789 Real W/out Edu
## 790 Not With No
## 791 Not With No
## 792 Real With Loss
## 793 Real W/out Edu
## 794 Not W/out Loss
## 795 Real W/out No
## 796 Real W/out Gain
## 797 Not With Loss
## 798 Not W/out Gain
## 799 Real W/out No
## 800 Not With Loss
## 801 Not With Edu
## 802 Real W/out Gain
## 803 Real W/out Loss
## 804 Not With Gain
## 805 Real With Loss
## 806 Not W/out Gain
## 807 Not W/out Edu
## 808 Not W/out Loss
## 809 Real With No
## 810 Not With Edu
## 811 Not W/out Gain
## 812 Not W/out Edu
## 813 Not With Gain
## 814 Not W/out Loss
## 815 Real With Edu
## 816 Not W/out No
## 817 Real With Gain
## 818 Not W/out Edu
## 819 Not W/out Loss
## 820 Real With No
## 821 Real W/out Edu
## 822 Not With No
## 823 Not With No
## 824 Real With Loss
## 825 Real W/out Edu
## 826 Not W/out Loss
## 827 Real W/out No
## 828 Real W/out Gain
## 829 Not With Loss
## 830 Not W/out Gain
## 831 Real W/out No
## 832 Not With Loss
## 833 Not With Edu
## 834 Real W/out Gain
## 835 Real W/out Loss
## 836 Not With Gain
## 837 Real With Loss
## 838 Not W/out Gain
## 839 Not W/out Edu
## 840 Not W/out Loss
## 841 Real With No
## 842 Not With Edu
## 843 Not W/out Gain
## 844 Not W/out Edu
## 845 Not With Gain
## 846 Not W/out Loss
## 847 Real With Edu
## 848 Not W/out No
## 849 Real With Gain
## 850 Not W/out Edu
## 851 Not W/out Loss
## 852 Real With No
## 853 Real W/out Edu
## 854 Not With No
## 855 Not With No
## 856 Real With Loss
## 857 Real W/out Edu
## 858 Not W/out Loss
## 859 Real W/out No
## 860 Real W/out Gain
## 861 Not With Loss
## 862 Not W/out Gain
## 863 Real W/out No
## 864 Not With Loss
## 865 Not With Edu
## 866 Real W/out Gain
## 867 Real W/out Loss
## 868 Not With Gain
## 869 Real With Loss
## 870 Not W/out Gain
## 871 Not W/out Edu
## 872 Not W/out Loss
## 873 Real With No
## 874 Not With Edu
## 875 Not W/out Gain
## 876 Not W/out Edu
## 877 Not With Gain
## 878 Not W/out Loss
## 879 Real With Edu
## 880 Not W/out No
## 881 Real With Gain
## 882 Not W/out Edu
## 883 Not W/out Loss
## 884 Real With No
## 885 Real W/out Edu
## 886 Not With No
## 887 Not With No
## 888 Real With Loss
## 889 Real W/out Edu
## 890 Not W/out Loss
## 891 Real W/out No
## 892 Real W/out Gain
## 893 Not With Loss
## 894 Not W/out Gain
## 895 Real W/out No
## 896 Not With Loss
## 897 Not With Edu
## 898 Real W/out Gain
## 899 Real W/out Loss
## 900 Not With Gain
## 901 Real With Loss
## 902 Not W/out Gain
## 903 Not W/out Edu
## 904 Not W/out Loss
## 905 Real With No
## 906 Not With Edu
## 907 Not W/out Gain
## 908 Not W/out Edu
## 909 Not With Gain
## 910 Not W/out Loss
## 911 Real With Edu
## 912 Not W/out No
## 913 Real With Gain
## 914 Not W/out Edu
## 915 Not W/out Loss
## 916 Real With No
## 917 Real W/out Edu
## 918 Not With No
## 919 Not With No
## 920 Real With Loss
## 921 Real W/out Edu
## 922 Not W/out Loss
## 923 Real W/out No
## 924 Real W/out Gain
## 925 Not With Loss
## 926 Not W/out Gain
## 927 Real W/out No
## 928 Not With Loss
## 929 Not With Edu
## 930 Real W/out Gain
## 931 Real W/out Loss
## 932 Not With Gain
## 933 Real With Loss
## 934 Not W/out Gain
## 935 Not W/out Edu
## 936 Not W/out Loss
## 937 Real With No
## 938 Not With Edu
## 939 Not W/out Gain
## 940 Not W/out Edu
## 941 Not With Gain
## 942 Not W/out Loss
## 943 Real With Edu
## 944 Not W/out No
## 945 Real With Gain
## 946 Not W/out Edu
## 947 Not W/out Loss
## 948 Real With No
## 949 Real W/out Edu
## 950 Not With No
## 951 Not With No
## 952 Real With Loss
## 953 Real W/out Edu
## 954 Not W/out Loss
## 955 Real W/out No
## 956 Real W/out Gain
## 957 Not With Loss
## 958 Not W/out Gain
## 959 Real W/out No
## 960 Not With Loss
## 961 Not With Edu
## 962 Real W/out Gain
## 963 Real W/out Loss
## 964 Not With Gain
## 965 Real With Loss
## 966 Not W/out Gain
## 967 Not W/out Edu
## 968 Not W/out Loss
## 969 Real With No
## 970 Not With Edu
## 971 Not W/out Gain
## 972 Not W/out Edu
## 973 Not With Gain
## 974 Not W/out Loss
## 975 Real With Edu
## 976 Not W/out No
## 977 Real With Gain
## 978 Not W/out Edu
## 979 Not W/out Loss
## 980 Real With No
## 981 Real W/out Edu
## 982 Not With No
## 983 Not With No
## 984 Real With Loss
## 985 Real W/out Edu
## 986 Not W/out Loss
## 987 Real W/out No
## 988 Real W/out Gain
## 989 Not With Loss
## 990 Not W/out Gain
## 991 Real W/out No
## 992 Not With Loss
## 993 Not With Edu
## 994 Real W/out Gain
## 995 Real W/out Loss
## 996 Not With Gain
## 997 Real With Loss
## 998 Not W/out Gain
## 999 Not W/out Edu
## 1000 Not W/out Loss
## 1001 Real With No
## 1002 Not With Edu
## 1003 Not W/out Gain
## 1004 Not W/out Edu
## 1005 Not With Gain
## 1006 Not W/out Loss
## 1007 Real With Edu
## 1008 Not W/out No
## 1009 Real With Gain
## 1010 Not W/out Edu
## 1011 Not W/out Loss
## 1012 Real With No
## 1013 Real W/out Edu
## 1014 Not With No
## 1015 Not With No
## 1016 Real With Loss
## 1017 Real W/out Edu
## 1018 Not W/out Loss
## 1019 Real W/out No
## 1020 Real W/out Gain
## 1021 Not With Loss
## 1022 Not W/out Gain
## 1023 Real W/out No
## 1024 Not With Loss
## 1025 Not With Edu
## 1026 Real W/out Gain
## 1027 Real W/out Loss
## 1028 Not With Gain
## 1029 Real With Loss
## 1030 Not W/out Gain
## 1031 Not W/out Edu
## 1032 Not W/out Loss
## 1033 Real With No
## 1034 Not With Edu
## 1035 Not W/out Gain
## 1036 Not W/out Edu
## 1037 Not With Gain
## 1038 Not W/out Loss
## 1039 Real With Edu
## 1040 Not W/out No
## 1041 Real With Gain
## 1042 Not W/out Edu
## 1043 Not W/out Loss
## 1044 Real With No
## 1045 Real W/out Edu
## 1046 Not With No
## 1047 Not With No
## 1048 Real With Loss
## 1049 Real W/out Edu
## 1050 Not W/out Loss
## 1051 Real W/out No
## 1052 Real W/out Gain
## 1053 Not With Loss
## 1054 Not W/out Gain
## 1055 Real W/out No
## 1056 Not With Loss
## 1057 Not With Edu
## 1058 Real W/out Gain
## 1059 Real W/out Loss
## 1060 Not With Gain
## 1061 Real With Loss
## 1062 Not W/out Gain
## 1063 Not W/out Edu
## 1064 Not W/out Loss
## 1065 Real With No
## 1066 Not With Edu
## 1067 Not W/out Gain
## 1068 Not W/out Edu
## 1069 Not With Gain
## 1070 Not W/out Loss
## 1071 Real With Edu
## 1072 Not W/out No
## 1073 Real With Gain
## 1074 Not W/out Edu
## 1075 Not W/out Loss
## 1076 Real With No
## 1077 Real W/out Edu
## 1078 Not With No
## 1079 Not With No
## 1080 Real With Loss
## 1081 Real W/out Edu
## 1082 Not W/out Loss
## 1083 Real W/out No
## 1084 Real W/out Gain
## 1085 Not With Loss
## 1086 Not W/out Gain
## 1087 Real W/out No
## 1088 Not With Loss
## 1089 Not With Edu
## 1090 Real W/out Gain
## 1091 Real W/out Loss
## 1092 Not With Gain
## 1093 Real With Loss
## 1094 Not W/out Gain
## 1095 Not W/out Edu
## 1096 Not W/out Loss
## 1097 Real With No
## 1098 Not With Edu
## 1099 Not W/out Gain
## 1100 Not W/out Edu
## 1101 Not With Gain
## 1102 Not W/out Loss
## 1103 Real With Edu
## 1104 Not W/out No
## 1105 Real With Gain
## 1106 Not W/out Edu
## 1107 Not W/out Loss
## 1108 Real With No
## 1109 Real W/out Edu
## 1110 Not With No
## 1111 Not With No
## 1112 Real With Loss
## 1113 Real W/out Edu
## 1114 Not W/out Loss
## 1115 Real W/out No
## 1116 Real W/out Gain
## 1117 Not With Loss
## 1118 Not W/out Gain
## 1119 Real W/out No
## 1120 Not With Loss
## 1121 Not With Edu
## 1122 Real W/out Gain
## 1123 Real W/out Loss
## 1124 Not With Gain
## 1125 Real With Loss
## 1126 Not W/out Gain
## 1127 Not W/out Edu
## 1128 Not W/out Loss
## 1129 Real With No
## 1130 Not With Edu
## 1131 Not W/out Gain
## 1132 Not W/out Edu
## 1133 Not With Gain
## 1134 Not W/out Loss
## 1135 Real With Edu
## 1136 Not W/out No
## 1137 Real With Gain
## 1138 Not W/out Edu
## 1139 Not W/out Loss
## 1140 Real With No
## 1141 Real W/out Edu
## 1142 Not With No
## 1143 Not With No
## 1144 Real With Loss
## 1145 Real W/out Edu
## 1146 Not W/out Loss
## 1147 Real W/out No
## 1148 Real W/out Gain
## 1149 Not With Loss
## 1150 Not W/out Gain
## 1151 Real W/out No
## 1152 Not With Loss
## 1153 Not With Edu
## 1154 Real W/out Gain
## 1155 Real W/out Loss
## 1156 Not With Gain
## 1157 Real With Loss
## 1158 Not W/out Gain
## 1159 Not W/out Edu
## 1160 Not W/out Loss
## 1161 Real With No
## 1162 Not With Edu
## 1163 Not W/out Gain
## 1164 Not W/out Edu
## 1165 Not With Gain
## 1166 Not W/out Loss
## 1167 Real With Edu
## 1168 Not W/out No
## 1169 Real With Gain
## 1170 Not W/out Edu
## 1171 Not W/out Loss
## 1172 Real With No
## 1173 Real W/out Edu
## 1174 Not With No
## 1175 Not With No
## 1176 Real With Loss
## 1177 Real W/out Edu
## 1178 Not W/out Loss
## 1179 Real W/out No
## 1180 Real W/out Gain
## 1181 Not With Loss
## 1182 Not W/out Gain
## 1183 Real W/out No
## 1184 Not With Loss
## 1185 Not With Edu
## 1186 Real W/out Gain
## 1187 Real W/out Loss
## 1188 Not With Gain
## 1189 Real With Loss
## 1190 Not W/out Gain
## 1191 Not W/out Edu
## 1192 Not W/out Loss
## 1193 Real With No
## 1194 Not With Edu
## 1195 Not W/out Gain
## 1196 Not W/out Edu
## 1197 Not With Gain
## 1198 Not W/out Loss
## 1199 Real With Edu
## 1200 Not W/out No
## 1201 Real With Gain
## 1202 Not W/out Edu
## 1203 Not W/out Loss
## 1204 Real With No
## 1205 Real W/out Edu
## 1206 Not With No
## 1207 Not With No
## 1208 Real With Loss
## 1209 Real W/out Edu
## 1210 Not W/out Loss
## 1211 Real W/out No
## 1212 Real W/out Gain
## 1213 Not With Loss
## 1214 Not W/out Gain
## 1215 Real W/out No
## 1216 Not With Loss
## 1217 Not With Edu
## 1218 Real W/out Gain
## 1219 Real W/out Loss
## 1220 Not With Gain
## 1221 Real With Loss
## 1222 Not W/out Gain
## 1223 Not W/out Edu
## 1224 Not W/out Loss
## 1225 Real With No
## 1226 Not With Edu
## 1227 Not W/out Gain
## 1228 Not W/out Edu
## 1229 Not With Gain
## 1230 Not W/out Loss
## 1231 Real With Edu
## 1232 Not W/out No
## 1233 Real With Gain
## 1234 Not W/out Edu
## 1235 Not W/out Loss
## 1236 Real With No
## 1237 Real W/out Edu
## 1238 Not With No
## 1239 Not With No
## 1240 Real With Loss
## 1241 Real W/out Edu
## 1242 Not W/out Loss
## 1243 Real W/out No
## 1244 Real W/out Gain
## 1245 Not With Loss
## 1246 Not W/out Gain
## 1247 Real W/out No
## 1248 Not With Loss
## 1249 Not With Edu
## 1250 Real W/out Gain
## 1251 Real W/out Loss
## 1252 Not With Gain
## 1253 Real With Loss
## 1254 Not W/out Gain
## 1255 Not W/out Edu
## 1256 Not W/out Loss
## 1257 Real With No
## 1258 Not With Edu
## 1259 Not W/out Gain
## 1260 Not W/out Edu
## 1261 Not With Gain
## 1262 Not W/out Loss
## 1263 Real With Edu
## 1264 Not W/out No
## 1265 Real With Gain
## 1266 Not W/out Edu
## 1267 Not W/out Loss
## 1268 Real With No
## 1269 Real W/out Edu
## 1270 Not With No
## 1271 Not With No
## 1272 Real With Loss
## 1273 Real W/out Edu
## 1274 Not W/out Loss
## 1275 Real W/out No
## 1276 Real W/out Gain
## 1277 Not With Loss
## 1278 Not W/out Gain
## 1279 Real W/out No
## 1280 Not With Loss
## 1281 Not With Edu
## 1282 Real W/out Gain
## 1283 Real W/out Loss
## 1284 Not With Gain
## 1285 Real With Loss
## 1286 Not W/out Gain
## 1287 Not W/out Edu
## 1288 Not W/out Loss
## 1289 Real With No
## 1290 Not With Edu
## 1291 Not W/out Gain
## 1292 Not W/out Edu
## 1293 Not With Gain
## 1294 Not W/out Loss
## 1295 Real With Edu
## 1296 Not W/out No
## 1297 Real With Gain
## 1298 Not W/out Edu
## 1299 Not W/out Loss
## 1300 Real With No
## 1301 Real W/out Edu
## 1302 Not With No
## 1303 Not With No
## 1304 Real With Loss
## 1305 Real W/out Edu
## 1306 Not W/out Loss
## 1307 Real W/out No
## 1308 Real W/out Gain
## 1309 Not With Loss
## 1310 Not W/out Gain
## 1311 Real W/out No
## 1312 Not With LossNow that we have the data frame that we want to use to calculate the predicted probabilities, we can tell R to create the predicted probabilities. predict1$V1P tells R that we want to create a new column in the data frame ‘predict1’ called ‘V1P’. Then we use the predict() function to make predictions. It has three arguments: 1. ‘logit1’ - This tells R to make the predictions based on the previous logistic regression 2. ‘newdata’ - This tells R to make the predictions using the values of the predictor variables 3. ‘type’ - This tells R that the type of prediction to be conducted is a predicted probability.
The output is shown by ‘head(predict1)’. we see that the predicted probability of someone choosing an image that is educational, contains a URL link and is not a real image is 0.54.
predict1$V1P <- predict(logit1, newdata = predict1, type = "response")
head(predict1)## V2 V3 V1 V1P
## 1 Not With Edu 0.5409345
## 2 Real W/out Gain 0.7199191
## 3 Real W/out Loss 0.3006551
## 4 Not With Gain 0.7163199
## 5 Real With Loss 0.3644291
## 6 Not W/out Gain 0.6543670predict3 <- cbind(predict1, predict(logit1, newdata = predict1, type="link", se=TRUE))
predict3 <- within(predict3, {
PredictedProb <- plogis(fit)
LL <- plogis(fit - (1.96 * se.fit))
UL <- plogis(fit + (1.96 * se.fit))
})
head(predict3)## V2 V3 V1 V1P fit se.fit residual.scale UL
## 1 Not With Edu 0.5409345 0.1641052 0.1412085 1 0.6084672
## 2 Real W/out Gain 0.7199191 0.9440604 0.1501711 1 0.7752847
## 3 Real W/out Loss 0.3006551 -0.8441804 0.1498270 1 0.3657439
## 4 Not With Gain 0.7163199 0.9262797 0.1497228 1 0.7720172
## 5 Real With Loss 0.3644291 -0.5561914 0.1513809 1 0.4354903
## 6 Not W/out Gain 0.6543670 0.6382908 0.1347073 1 0.7114254
## LL PredictedProb
## 1 0.4718639 0.5409345
## 2 0.6569485 0.7199191
## 3 0.2427169 0.3006551
## 4 0.6531293 0.7163199
## 5 0.2988253 0.3644291
## 6 0.5924890 0.6543670jj
ggplot(predict3, aes(x = V2, y = PredictedProb)) +
geom_ribbon(aes(ymin=LL, ymax=UL, fill=V1), alpha = 0.2) +
geom_line(aes(colour=V1), size=1)
ss
ggplot(predict3, aes(x = V3, y = PredictedProb)) +
geom_ribbon(aes(ymin=LL, ymax=UL, fill=V1), alpha = 0.2) +
geom_line(aes(colour=V1), size=1)
## 2.9 How well does this model fit s
with(logit1, null.deviance - deviance)## [1] 137.1173ss
with(logit1, df.null - df.residual)## [1] 5ss
with(logit1, pchisq(null.deviance - deviance, df.null - df.residual, lower.tail = FALSE))## [1] 7.333095e-28ss
logLik(logit1)## 'log Lik.' -840.8505 (df=6)ss