Chapter 1 - Introduction to Data Graded: 1.8, 1.10, 1.28, 1.36, 1.48, 1.50, 1.56, 1.70

library(openintro)
library(ggplot2)

Q1.8 Smoking habits of UK residents. A survey was conducted to study the smoking habits of UK residents. Below is a data matrix displaying a portion of the data collected in this survey. Note that “£” stands for British Pounds Sterling, “cig” stands for cigarettes, and “N/A” refers to a missing component of the data.

No. sex age marital grossIncome smoke amtWeekends amtWeekdays
1 Female 42 Single Under£2,600 Yes 12 cig/day 12 cig/day
2 Male 44 Single £10,400 to £15,600 No N/A N/A
3 Male 53 Married Above£36,400 Yes 6 cig/day 6 cig/day
… … … … … … … …
1691 Male 40 Single £2,600 to £5,200 Yes 8 cig/day 8 cig/day

  1. What does each row of the data matrix represent?

    Each row is a data sample.

  2. How many participants were included in the survey?

    1691 participants

  3. Indicate whether each variable in the study is numerical or categorical. If numerical, identify as continuous or discrete. If categorical, indicate if the variable is ordinal.

    categorical-non ordinal:Sex,marital,smoke
categorical-ordinal:grossIncome,amtWeekends,amtWeekdays
numerical-discrect:age

Q1.10 Cheaters, scope of inference. Exercise 1.5 introduces a study where researchers studying the relationship between honesty, age, and self-control conducted an experiment on 160 children between the ages of 5 and 15. The researchers asked each child to toss a fair coin in private and to record the outcome (white or black) on a paper sheet, and said they would only reward children who report white. Half the students were explicitly told not to cheat and the others were not given any explicit instructions. Di???erences were observed in the cheating rates in the instruction and no instruction groups, as well as some differences across children’s characteristics within each group.

  1. Identify the population of interest and the sample in this study.

    Identify the population from these variables: honesty, age, self-control
sample size: 160 children between the ages 5-15

  2. Comment on whether or not the results of the study can be generalized to the population, and if the findings of the study can be used to establish causal relationships.

    Undefined.
The reasons:
1.There is not enough information to know whether 160 child samples refer to a target all children, for unknown factors like location, culture, age, ect. It is impossible to pick 160 samples from NYC to target the population in CA, or pick 160 Asian kids to target the population for all kids in US. 
2.There is no information about the distribution of boys and girls in two different groups. What if there were majority boys in first study group, and majority girls in second group, then it is hard to trust that no instruction group probability of cheating was found to uniform across groups based on boys and girls characteristics, and girls were less likely to cheat when they told not to cheat.

Q1.28 Reading the paper. Below are excerpts from two articles published in the NY Times: (a) An article titled Risks: Smokers Found More Prone to Dementia states the following:61 “Researchers analyzed data from 23,123 health plan members who participated in a voluntary exam and health behavior survey from 1978 to 1985, when they were 50-60 years old. 23 years later, about 25% of the group had dementia, including 1,136 with Alzheimer’s disease and 416 with vascular dementia. After adjusting for other factors, the researchers concluded that pack-aday smokers were 37% more likely than nonsmokers to develop dementia, and the risks went up with increased smoking; 44% for one to two packs a day; and twice the risk for more than two packs.”

Based on this study, can we conclude that smoking causes dementia later in life? Explain your reasoning.

I believe this is unbiased analysis, smoking causes dementia later in life. 
Reasons: 
1. The sample size of 23,123 50-60 year old members from health plan is large enough to target population. 
2. The given information that there are 25% of people having dementia and among of them there were 1,136 with Alzheimer and 416 with vascular dementia, close to the expected value in other research. Therefore, I can assume these 23,123 can represent the population. 
3. Clusters from 23,123 are well divided in different groups, non-smoke, pack-a-day smoke, 1-2 pack day smoke, and  2 pack day smoke . The analysis outcome is from completed data sets.
  1. Another article titled The School Bully Is Sleepy states the following:62 “The University of Michigan study, collected survey data from parents on each child’s sleep habits and asked both parents and teachers to assess behavioral concerns. About a third of the students studied were identified by parents or teachers as having problems with disruptive behavior or bullying. The researchers found that children who had behavioral issues and those who were identified as bullies were twice as likely to have shown symptoms of sleep disorders.” A friend of yours who read the article says, “The study shows that sleep disorders lead to bullying in school children.” Is this statement justified? If not, how best can you describe the conclusion that can be drawn from this study? ``` The statement is not justified. Reasons:
  1. The finding of the 3rd research is basic on children who have the behavior problem with disruptive behavior and bullying, and finding bullies were twice as likely to have sleeping order.
  2. However, we are lack of infomation about sleep non-disorder cause disruptive behavior and bullying. We can not state sleep disorders lead to bullying in shool children. ```

Q1.36 researcher is interested in the e???ects of exercise on mental health and he proposes the following study: Use stratified random sampling to ensure representative proportions of 18-30, 31-40 and 41- 55 year olds from the population. Next, randomly assign half the subjects from each age group to exercise twice a week, and instruct the rest not to exercise. Conduct a mental health exam at the beginning and at the end of the study, and compare the results. (a) What type of study is this?

Experiments study

  1. What are the treatment and control groups in this study?

    treatment groups: people in each age group to exercise twice a week
control groups:people in each age group not to exercise

  2. Does this study make use of blocking? If so, what is the blocking variable?

    18-30, 31-40 and 41- 55 year olds

  3. Does this study make use of blinding?

    No, patient don't do excise knowing they are in control group.

  4. Comment on whether or not the results of the study can be used to establish a causal relationship between exercise and mental health, and indicate whether or not the conclusions can be generalized to the population at large.

    Undefine, to establish a causal relationship is only when exercise and mental health are correlated.
Yes,sampling ensures representative proportions of age groups from the population.

  5. Suppose you are given the task of determining if this proposed study should get funding. Would you have any reservations about the study proposal?

    The result sometimes would be bais since patient are not blided.

Q. 1.48 Stats scores. Below are the final exam scores of twenty introductory statistics students.

57, 66, 69, 71, 72, 73, 74, 77, 78, 78, 79, 79, 81, 81, 82, 83, 83, 88, 89, 94

Create a box plot of the distribution of these scores. The five number summary provided below may be useful.

Min Q1 Q2 (Median) Q3 Max
57 72.5 78.5 82.5 94 
scores<-c(57, 66, 69, 71, 72, 73, 74, 77, 78, 78, 79, 79, 81, 81, 82, 83, 83, 88, 89, 94)
boxplot(scores)

Q. 50 Mix-and-match. Describe the distribution in the histograms below and match them to the box plots.

(1)<-c
(2)<-a
(3)<-b

Q.56 For each of the following, state whether you expect the distribution to be symmetric, right skewed, or left skewed. Also specify whether the mean or median would best represent a typical observation in the data, and whether the variability of observations would be best represented using the standard deviation or IQR. Explain your reasoning.

  1. Housing prices in a country where 25% of the houses cost below $350,000, 50% of the houses cost below $450,000, 75% of the houses cost below $1,000,000 and there are a meaningful number of houses that cost more than $6,000,000.

    Left skewed:cost rang btw (1st Q,2nd Q) is much lesser btw (2nd Q, 3rd Q), which mean right side has long tail;
median:use when the data distribution is not symmetric;
IQR: when the outlier data affect the result.

  2. Housing prices in a country where 25% of the houses cost below $300,000, 50% of the houses cost below $600,000, 75% of the houses cost below $900,000 and very few houses that cost more than $1,200,000.

    symmetric:rang btw (1st Q,2nd Q) is similar in btw (2nd Q, 3rd Q);
Mean:use when the data distribution is symmetric;
Standar deviation:when the outlier data doesn't affect the result.

  3. Number of alcoholic drinks consumed by college students in a given week. Assume that most of these students don’t drink since they are under 21 years old, and only a few drink excessively.

    Right skewed:rang btw (1st Q,2nd Q) is much large in btw (2nd Q, 3rd Q), which mean left side has very long tail;
median:use when the data distribution is not symmetric;
IQR:when the outlier data affect the result.

  4. Annual salaries of the employees at a Fortune 500 company where only a few high level executives earn much higher salaries than the all other employees.

    Left skewed:rang btw (1st Q,2nd Q) is much less in btw (2nd Q, 3rd Q);
median:use when the data distribution is not symmetric;
IQR:when the outlier data affect the result.

Q.70 (a) Based on the mosaic plot, is survival independent of whether or not the patient got a transplant? Explain your reasoning.(See the next page for additional parts to this question.)

Yes, data sets are disjoin.

  1. What do the box plots below suggest about the efficacy (effectiveness) of the heart transplant treatment.

    Patient who received treatment have more survival time

  2. What proportion of patients in the treatment group and what proportion of patients in the control group died?

    proportion of patients died in the treatment group less than in control gorup.
  3. One approach for investigating whether or not the treatment is effective is to use a randomization technique.

  1. What are the claims being tested?

    whether or not the treatment is effective to use a randomization technique.

  2. The paragraph below describes the set up for such approach, if we were to do it without using statistical software. Fill in the blanks with a number or phrase, whichever is appropriate.

library(DATA606)

Welcome to CUNY DATA606 Statistics and Probability for Data Analytics 
This package is designed to support this course. The text book used 
is OpenIntro Statistics, 3rd Edition. You can read this by typing 
vignette('os3') or visit www.OpenIntro.org. 
 
The getLabs() function will return a list of the labs available. 
 
The demo(package='DATA606') will list the demos that are available. 
data(heartTr)
table(heartTr$survived)

alive  dead 
   28    75 
table(heartTr$transplant)

  control treatment 
       34        69 
table(heartTr$survived, heartTr$transplant)
       
        control treatment
  alive       4        24
  dead       30        45
prop.table(table(heartTr$survived, heartTr$transplant))
       
           control  treatment
  alive 0.03883495 0.23300971
  dead  0.29126214 0.43689320
We write alive on **28** cards representing patients who were alive at the end of the study, and 
dead on **75** cards representing patients who were not. Then, we shuffle these cards and split 
them into two groups: one group of size **69** representing treatment, and another group of size
**34** representing control. We calculate the difference between the proportion of dead cards in 
the treatment and control groups (treatment - control) and record this value. We repeat this 
100 times to build a  distribution centered at **0**. Lastly, we calculate the fraction of 
simulations where the simulated differences in proportions are **.23 or higher**. If this fraction
is low, we conclude that it is unlikely to have observed such an outcome by chance and that the 
null hypothesis should be rejected in favor of the alternative.
n.alive <- 28
n.dead <- 75
n.treat <- 69
n.control <- 34
n.samples <- 100

# Verify these are the same
n.alive + n.dead
[1] 103
n.treat + n.control
[1] 103
cards <- c(rep('alive', n.alive), rep('dead', n.dead))
cards
  [1] "alive" "alive" "alive" "alive" "alive" "alive" "alive" "alive"
  [9] "alive" "alive" "alive" "alive" "alive" "alive" "alive" "alive"
 [17] "alive" "alive" "alive" "alive" "alive" "alive" "alive" "alive"
 [25] "alive" "alive" "alive" "alive" "dead"  "dead"  "dead"  "dead" 
 [33] "dead"  "dead"  "dead"  "dead"  "dead"  "dead"  "dead"  "dead" 
 [41] "dead"  "dead"  "dead"  "dead"  "dead"  "dead"  "dead"  "dead" 
 [49] "dead"  "dead"  "dead"  "dead"  "dead"  "dead"  "dead"  "dead" 
 [57] "dead"  "dead"  "dead"  "dead"  "dead"  "dead"  "dead"  "dead" 
 [65] "dead"  "dead"  "dead"  "dead"  "dead"  "dead"  "dead"  "dead" 
 [73] "dead"  "dead"  "dead"  "dead"  "dead"  "dead"  "dead"  "dead" 
 [81] "dead"  "dead"  "dead"  "dead"  "dead"  "dead"  "dead"  "dead" 
 [89] "dead"  "dead"  "dead"  "dead"  "dead"  "dead"  "dead"  "dead" 
 [97] "dead"  "dead"  "dead"  "dead"  "dead"  "dead"  "dead" 
length(cards)
[1] 103
set.seed(2112) # To reproduce exact results

simulation <- data.frame()
for(i in seq_len(n.samples)) {
    test <- data.frame(survived = cards, transplant = 'control', stringsAsFactors = FALSE)
    test[sample(nrow(test), n.treat),]$transplant <- 'treat'
    # prop.table(table(test$survived, test$transplant))
    simulation <- rbind(simulation, data.frame(
        iter = i,
        TreatAndAlive = sum(test$survived == 'alive' & test$transplant == 'treat') / nrow(test),
        stringsAsFactors = FALSE
    ))
}

ggplot(simulation, aes(x = TreatAndAlive, fill = (TreatAndAlive > 0.23))) + geom_histogram()
`stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

# Proportion of the samples where treatment and alive was greater than 0.23
sum(simulation$TreatAndAlive >= 0.23) / nrow(simulation)
[1] 0.01