STM1001 Topic 4B Readings

class: middle
background-image: url(data:image/png;base64,#LTU_logo.jpg)
background-position: top left
background-size: 30%

# STM1001 [Topic 4B](https://bookdown.org/a_shaker/STM1001_Topic_4B_Sci_S/) Readings
## Designing a Study Part II
### La Trobe University
This presentation complements the [Topic 4B readings](https://bookdown.org/a_shaker/STM1001_Topic_4B_Sci_S/)

---

# Topic 4B: Designing a Study Part II

### In this topic, we will continue our discussion of internal validity. We will then briefly consider the identification of study limitations, and procedures for collecting data.

---
#  Introduction

The conclusions drawn from a study are only as good as the data that the conclusions are based on, and the data are only as good as the study design that the data emerge from.

A good study requires high *internal validity*: When studying the relationship between the response and explanatory variable, other possible issues that might influence the value of the response variable should be eliminated.

Specific design strategies that we consider for maximising internal validity are:

* Managing **confounding**;
* Managing the **carry-over effect** using washout periods;
* Managing the **Hawthorne effect** by blinding individuals;
* Managing the **observer effect** by blinding the researchers; and
* Managing the **placebo effect** using controls.

---

# Managing confounding

Confounding has the potential to compromise the internal validity of the study. Suppose, for example, that the researchers created two groups:

Group A: Women recruited at a female-only gym.

Group B: Men recruited at a local nursing home.

The researchers then gave Himalaya 292 to Group A, and the refined cereal to Group B. If a difference in faecal weight was found between the two groups, the difference may because: *diet*, *sex*, *age*, *health and fitness levels*.

The **groups** being compared **should be as similar as possible**.

Confounding can be managed by:

* **Restricting** the study to a certain group (e.g., only people under 30).

* **Blocking.** Analyse the data separately for different groups (e.g., analyse the data separately for people under 30, and 30 and over).

* **Analysing** using special methods (after measuring ages).

* **Randomly allocating** people to groups: Older and younger people would be spread approximately evenly between groups.

---

# Random allocation vs random sampling

*Random sampling* and *random allocation* are two different concepts, that serve two different purposes, but are often confused:

**Random sampling** allows results to be generalised to a larger population, and impacts *external* validity. It concerns *how the sample is found to study*.

**Random allocation** tries to eliminate confounding issues, by evening-out possible confounders across treatment groups. *Random allocation* of treatments helps establish cause-and-effect, and impacts internal validity. It concerns *how the members of the chosen sample get the treatments*.

---

# Carry-over effect and washout periods
In the *Himalaya* study, what if patients spent two weeks on the *Himalaya* 292 diet, then the next two weeks on the refined cereal diet?

.content-box-blue[
**Definition (Carryover effect)** The carry-over effect is when the influence of past experience(s) of the individuals carry over to influence future experience(s) of the individuals.]

---

# Hawthorne effect and blinding individuals
What if the patients in the Himalaya 292 study were being watched (or waited for) while defecating? People often behave differently (either positively or negatively) if they know (or think) they are in a study or are being watched. This is called the **Hawthorne effect**.

.content-box-blue[
**Definition (Hawthorne effect)** The Hawthorne effect is the tendency of people (or animals, or...) to behave differently if they know (or think) they are being observed.]

The impact of the Hawthorne effect can be minimized by *blinding* the individuals in the experiment so that they do not know:

* that they are in a study;
* the aims of the study, and/or
* which treatment they are receiving.

---

# Observer effect and blinding researchers

What if the *researchers* assessing the outcomes *knew the diet* allocated to each patient? Perhaps surprisingly, this can have an (unconscious) impact on the values of the response variable. This is called the *observer effect*. This could also compromise the internal validity of the study.

.content-box-blue[
**Definition (Observer effect)** The observer effect is when the researchers unintentionally influence the behaviour of subjects.]

The impact of the observer effect can be minimized by blinding the researchers so that they do not know: which treatments the individuals are receiving.

---
# Placebo effect and using controls
What if people *thought* they were on the wholegrain diet, but they weren't? Perhaps surprisingly, individuals in a study may report effects of a treatment (either positive or negative), even if they have not received an active treatment. This could also compromise the internal validity of the study.

.content-box-blue[
**Definition (Placebo effect)** The placebo effect is when individuals report perceived or actual effects without having received the treatment.]

Managing the placebo effect is difficult! However, impact of the placebo effect can be minimized using a *control group*: units of analysis without the treatment applied, but as similar as possible in every other way to those units of analysis receiving the treatment.

A **control** is a unit of analysis without the treatment applied. A **placebo** is a treatment with no intended effect or active ingredient.
---

# Internal validity and observational studies
In experimental studies, many aspects of the study design typically can be controlled by the researcher, so experimental studies are often easier to design to maximise internal validity. In contrast, *observational studies* have fewer design features that can be controlled by the researchers.

For example, treatments are not imposed in observational studies, so random allocation of treatments is impossible, and hence *confounding* is always a potential threat to internal validity in observational studies.

The best advice for observational studies is **to measure, observe, assess or record all the information that is likely to be important for understanding the data.** Observational studies can (and often do) have control groups. Indeed, one specific type of observational study is called a **case-control study.**

---

# Internal validity and observational studies

The carry-over effect is a possible compromise to internal validity in observational studies. However, since treatments are not allocated in observational studies, carry-over effects may be difficult to prevent.

In observational studies, individuals may or may not know they are being observed. As with experimental studies, efforts should be made to ensure that individuals do not know that they are being observed (that is, that the participants are blinded).

The observer effect can be an issue in observational as well as experimental studies. E.g.  if the researchers know whether or not the individual is a smoker when they record the blood pressure, then the observer effect could still come into play. The observer effect could be managed if the researchers first measured, and then asked if the individual was a smoker or not.

The placebo effect is concerned with treatments, so are not directly relevant to observational studies.

---

# Identifying study limitations

The *type* of study and how that study is designed can determine how the results of the study should be interpreted. Ideally, a study would be perfectly externally and internally valid, but in practice this is very difficult to achieve. Practically *every* study has limitations.

Limitations can often be discussed through three components:

* **External validity** (the applicability of the study results outside the sample): The generalisability of the results to the intended population.

* **Internal validity** (the effectiveness of the study in the sample): The steps taken to maximise the internal validity of the study, and the impacts of these on the interpretation of results.

* **Ecological validity** (the practicality of the results to real life): The practicality of the results in the real world; how the study methods, materials and context approximate the real situation being studied.

Almost every study has limitations. *Identifying* them, and *discussing* the impact that they have on the interpretation of the study results, is important and ethical.

---

# Limitations: External validity
External validity refers to the ability to *generalise* the results to other groups in the population apart from the sample studied (see section: External and internal validity).

.content-box-blue[
Importantly, external validity refers to how well the sample is likely to represent the target population as given in the RQ.]

External validity refers to the *applicability* or the *generalisability* of the *results* to the target (or intended) population (Example), which depends on how the sample was obtained: *results from random samples are likely to generalise to the population and be externally valid when appropriately analysed*. Furthermore, results from approximately representative samples may generalise to the population and be externally valid if those in the study are not obviously different than those not in the study.

---

# Limitations: Internal validity

Internal validity refers to how reasonable and logical the results from the study are: the strength of the inferences that can be made from the sample. That is, an internally valid study is effective in demonstrating that the conclusions made from the sample cannot be explained any other way.

Internal validity can be compromised by confounding, the carryover effect, the Hawthorne effect, the observer effect, and/or the placebo effect. Consequently, if any of these issues are likely to compromise internal validity, the implications on the interpretation of the results should be discussed.

The internal validity of observational studies is often compromised because confounding can be less effectively managed than for experimental studies.

---

# Limitations: Ecological validity

The *practicality* of the study results in the real world should also be discussed. This is called *ecological validity*.

.content-box-blue[
**Definition (Ecological validity)** A study is ecologically valid if the study methods, materials and context approximate the real situation being studied.]

Studies don't need to be ecologically valid to be useful; much can be learnt under special conditions, as long as the potential limitations are understood when applying the results to the real world. Although ecological validity is not essential for a good study, ecological validity is useful if it is possible to achieve.

---

# Procedures for collecting data
So far, we have learn to ask a RQ, identify different ways of obtaining data, and design the study.

If the RQ is well-constructed, all terms are clearly defined, and the research design is clear and well explained, then collecting the data should be reasonably easy to implement. However, data collection may be time-consuming, tedious and expensive so getting the data collection correct first time is important.

Before collecting the data, a plan should be established and documented that explains exactly how the data will be obtained. This plan is a draft **protocol**.

.content-box-blue[
**Definition (Protocol)** A protocol is a procedure documenting the details of the design and implementation of studies, and for data collection.]

---

# Collecting data using surveys

Many issues must be kept in mind when framing survey questions; here are some.

* **Avoid leading questions** which may indicate how respondents are expected to answer.
* **Avoid ambiguity**: Avoid terms that may be unfamiliar, and questions that are unclear.
* **Avoid complex and double-barrelled** questions; these are often hard to understand.
* **Avoid problems with confidentiality**, which would be considered unethical. Ethics committees usually look very carefully for questions that are unethical.
* **Ensure** that questions are clearly and precisely worded.
* **Ensure** that options for multiple-choice questions are mutually exclusive (all answers fit into only one category) and exhaustive (the categories cover all possible options).
---

name: menti
class: middle
background-image: url(data:image/png;base64,#menti.jpg)
background-size: 115%

# Have questions?

## Ask your computer lab demonstrator

## or discuss with your peers

---
class: middle

These notes have been prepared by Illia Donhauzer. They are based on material written by Peter K. Dunn. Unless otherwise stated, material within this work is licensed under a Creative Commons Attribution-Non Commercial-Share Alike License <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/">CC BY-NC-SA </a>