The Intention-Behavior Gap
Developing and Validating a New Measure
Daniel J Wilson1, Cendri Hutcherson1
1University of Toronto
Last updated: 09 March, 2021
The Development of a Comprehensive Intention-Behavior Gap Scale
This workflow comes directly from the paper Best Practices for Developing and Validating Scales for Health, Social,and Behavioral Research: A Primer (Boateng et al., 2018).
1. Item Development
1.1 Identification of Domain and Item Generation: Selecting Which Items to Ask
Domain Identification is used to specify the boundaries of the domain and facilitate item generation. Item generation involves appropriate identification of questions that fit the the identified domain (Hinkin, 1995a; Morgado et al., 2017).
1.1.1 Domain Identification
To specify the boundaries of the domain and facilitate item generation
1.1.1.1 Specify the purpose of the domain
“Good resolutions are simply checks that men draw on a bank where they have no account.”
Oscar Wilde may have had a particularly gloomy assessment of the ability of humans to realize their better intentions, but he is clearly speaking to a well-known, and commonly experienced, phenomenon. Perhaps the most maligned example of Wilde’s critique is our tradition of New Year’s resolutions (see Figure 1). The takeaway being that even if people desire to be healthy, eat well, and live a life that aligns with their goals and ideals, just wanting it does not make it so.
Figure 1: A plot of US Google search frequencies of the terms resolution, diet and fitness. Note the peaks in January of each year.
This is not to say that intention is not important. Sheeran (2002) showed in a large meta-analysis of meta-analyses that the average sample-weighted correlation between intention and behavior across experiments was a robust 0.53 (\(95\% CI = [0.5,0.53], n=82,107\)). It is, without question, the best predictor we have. And yet, it still leaves 72% of the variance as to whether a given intention is acted upon unexplained.
While there has been a great deal of work attempting to better understand human motivation, goals and self-regulation in recent years (Aarts & Elliot, 2012; Moskowitz & Grant, 2009; Shah & Gardner, 2008; Vohs & Baumeister, 2016), much of this work does not provide an explicit test of the relationship between intention and behavior (Sheeran & Webb, 2016). Even when such a test is performed, it is almost always focused on a narrow domain, for example consumer and leisure decisions (e.g., Warshaw & Davis, 1984), exercise (e.g., Norman & Smith, 1995), smoking (e.g. Norman et al., 1999), voting (e.g., Bassili, 1995), or prosocial behaviors such as blood donation (e.g., Warshaw et al., 1986), just to pick a few representative papers.
We are proposing a new instrument that seeks to provide a comprehensive intention-behavior gap magnitude measure. That is to say that given the intention-behavior correlation of 0.53 is the mean of a global gap distribution, we want a method of locating where any given individual might lie on that curve. The measure is purposely uninformative about what the mechanisms of the gap might be. It aims to do one thing and, hopefully, do it well. We believe that this will promote progress in two specific areas. First, it will serve as a critical dependent variable (or endpoint of a directed acyclic graph) in any model trying to establish the causal mechanisms contributing to the gap. Second, we believe it could provide actionable information on its own, both at the population and individual levels. At the population level it would serve to shed light on the distribution of the intention-behavior gap, information that could then also be stratified by sub-populations to reveal what might be important group differences based on demographic, geographic or cultural factors. At the individual level, knowing someone’s gap could provide insight into effective intervention strategies. For example, for someone with a very large intention-behavior gap strategies that focus on changing intentions may not be effective, as in this case, the individual’s intentions are poorly correlated with their future behavior. Whereas for someone with a small gap this might be a reasonable approach.
We know, both from personal experience and the literature, that when we are successful in actualizing our goals about who we would ideally like to be, or believe we ought to be, we feel better for it (e.g., Cooley, 2017; James, 1910; Rogers, 1995). Our hope is that this measure can be a tool that allows us to try and move a bit closer toward this goal.
Possible addition:
- Changing behavior important: many behaviors linked to leading causes of death in developing and developed countries (Davis et al., 2014)
- Current theory is wanting when it comes to affecting behavioral change (intervention) (Davis et al., 2014)
1.1.1.2 Confirm that there are no existing instruments
Numerous theories have been developed that deal with intentions, behavior and the disconnect that may arise between the two (e.g. Theory of Reasoned Action (Fishbein et al., 1980) , Theory of Planned Behavior (Ajzen, 1985), Attitude Behavior Theory (Triandis, 1980), Protection Motivation Theory (Maddux & Rogers, 1983)). We will not deal with the details of these theories. As pointed out previously, there have been many studies that operationalize these theories, and also critiques of these studies (e.g. Sutton, 1998), but in general they are focused on the theoretical model behind the gap and implemented in narrow behavioral contexts.
That said, our examination of the literature has uncovered a number of instruments that overlap with the measure we seek to create. We will briefly describe these existing measures and then explain how our proposed comprehensive intention-behavior gap measure differs. Most related measures tend to be informed by Higgins’ Self-Discrepancy Theory (1985), which proposes that individuals tend to compare their actual selves with an internalized standard of an “ideal” and “ought” self (Higgins et al., 1985). His measure of Regulatory Focus Strength attempts to quantify an individuals self-discrepancy based on this definition (Higgins et al., 1997). He used an idiographic strategy for the measure, where subjects came up with three to five ought and ideal attributes for themselves off the top of their head. They then rated the extent that they ought, or ideally, should possess these attributes, and also the extent to which they believed they actually did. This input informed both a self-discrepancy measure - based on subtracting the desired ideal/ought scores they gave each selected attribute with the actual scores they had indicated - as well as a self-guide strength measure, which was based off of subjects’ response times in coming up with the attributes and indicating their desired and actual scores for each (Higgins et al., 1997).
Numerous iterations on this measure and it’s closely related predecessor, the Selves Questionnaire (Higgins et al., 1985), have been made. For example, Tangney et al. (1998) ran a nomothetic version of the Selves Questionnaire, providing adjectives for participants to score instead of having them self-generate attributes. In another version, The Self Lines measure required participants to manually draw the distance between their actual self and their self-generated ought and ideal self attributes (Francis et al., 2006).
Hardin and Lakin (2009) surveyed the various approaches and attempted to draw what they thought was best from each to create their hybrid Integrated Self-Discrepancy Index. They use a novel combined ideographic and nomothetic approach. This serves to limits the risk of introducing verbal fluency as a confound with an ideographic-only approach, or of not successfully capturing participants’ most accessible self discrepancies with a constrained nomothetic approach. They also address another critique of the Selves Questionnaire leveled by Tangney et al. (1998), in which he reported that "one of the most frequently asked questions during our data collection sessions . . .was, ‘What’s the difference between ideal and ought?"’ (p. 265). Hardin and Lakin address this critique by including the concept of morality in the description of the ought self, as a means of clarifying the distinction. They justify this choice based on Higgins’ own explanation of the ought self (1985, p. 54).
In the Comprehensive Intention-Behavior Gap measure we draw inspiration and guidance from these implementations, but make some modifications and additions we feel necessary. Modifications were made either to ensure the questionnaire items and flow were as clear and simple as possible for participants in order to maximize the quality of our data, or to add dimensions that we felt were necessary for the measure to be a truly comprehensive accounting of the intention-behavior gap.
1.1.1.3 Describe the domain and provide preliminary conceptual definition
We would like to pause at this point to clearly define what we mean when we refer to a “Comprehensive Intention-Behavior Gap.” We broadly agree with the definition of behavior provided by Levitis et al. in their review paper on defining behavior, which drew on survey responses from 174 members of three behavior-focused scientific societies. They landed on: “Behavior is the internally coordinated responses (actions or inactions) of whole living organisms (individuals or groups) to internal and/or external stimuli, excluding responses more easily understood as developmental changes (Levitis et al., 2009).” We would tweak this language slightly for our measure, to be: the action or inaction of individuals in response to their internal drives and environment. A key aspect of behavior, and one that many of the respondents in Levitis’ study point out, is that it is observable (Levitis et al., 2009). This is not the case for intentions.
The unseen nature of intentions makes both their measurement and their definition difficult. Even so, a clear description is required in order to anchor our measurements. Sheeran and Webb (2016) define a behavioral intention as self-instructions to perform particular actions directed toward achieving a desired outcome. Their conception of intention involves both a level of the set goal (they give the example of a set amount of time a person decides to work on a paper), and a level of commitment, which they construe as how determined a person is to achieve the decided upon level of their set goal. This formulation is useful for many of the intention-behavior gap studies that focus on one narrow measurable metric (e.g. exercise). However, you can think of intentions that do not have “levels” per se, either because they are binary (e.g. taking the trash to the curb in the morning), or because they are higher order goals whose gap can only be estimated by an array of actions. Take the attribute “caring,” for example. In Hardin & Lakin’s study this was one of the 10 most common ideal attributes people aspired to as an “ultimate goal” (2009). Yet, it is difficult to conceive of achieving this goal based on levels. For example, if you decide to volunteer for a cause you care about for one hour per day and then perform that action, but are grumpy and rude while doing so, have you achieved your goal? We would also contend that while determination might be a useful measure that is predictive of how likely an intention is to be realized, it is not an inherent component of an intention. We would therefore modify the Sheeran and Webb definition: A behavioral intention is, on a high level, self-instructions to act in congruence with your values, and, on a lower level, self-instructions to perform particular actions directed toward achieving a desired outcome.
NEW
👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇👇
1.1.1.4 Specify the dimensions of the domain if they exist a priori
In this formulation the high level self-instructions (conceived of as attributes in our scale) inform the low-level self-instructions (see Figure 2. For example, a high level self-instruction to be generous could be implemented as the low-level self-instruction to spend one hour researching local volunteer opportunities over the weekend. For a comprehensive measure we need to take both of these levels of intentions into account. We refer to these the “Value” level and the “Action” level, as in Figure 2 going forward. We also introduce the idea of a top “meta” level which captures the idea of whether we think that we are, overall, fulfilling our potential.
Figure 2: Multi-level Model of Intention-Behavior Gap. Note that attributes can lead both to actions via goals, or directly to actions.
Between level gap discrepancy
We anticipate a correlation between the meta, value, and action level intention-behavior gap magnitudes, but it is not required by the model. For example, one could reasonably anticipate differing gaps between levels under certain conditions. That is to say that you may be fulfilling your action goals but, those action goals are not necessarily well aligned with your values. We are familiar with the idea of working hard and being highly “productive” in a job we don’t like. Alternatively, it can be the case that we are in fact fulfilling our value goals but experience a larger gap at the level of our action goals. This could be the case, for example, if you were to set yourself naive goals that are far beyond your abilities, or if you had your goals thwarted by an outside variable (a global pandemic, for instance).
Within level goal conflict
The action level goal conflict is a well known problem of limited resources. We cannot be in multiple places at once nor add hours to the day. This means that we attempt to negotiate a resource allocation when deciding on our action level goals that we feel align with our value level goals. This can even happen in the context of a single value; take for example the attribute of being responsible. Acting responsibly in the context of being a parent can suggest very different actions than the goal of being responsible in the context of your professional life. This is partly why work-life balance can be so tricky to negotiate.
There can also be conflict at the value level, as it can be the case that different attributes we would like to fulfill suggest goals a the action level that are misaligned. One could imagine that If we consider creativity as an ideal attribute goal we may wish to take the action of applying for an MFA program, while at the same time our ought attribute of responsibility could be pushing us toward what seems like a more stable MBA path. Different levels of ought/ideal attribute overlap could be expected to correspond to different magnitudes of value level goal conflict (see 3 for a simple illustration of this concept). There could also be conflict within your ideal attributes themselves. If you have goal attributes of being ambitious and being kind there might be instances when these are unaligned.
Figure 3: Ought Value vs. Ideal Value congruence. The overlapping area represents those values that are common to both our set of ideal attribtues and ought attributes..
## (polygon[GRID.polygon.68], polygon[GRID.polygon.69], polygon[GRID.polygon.70], polygon[GRID.polygon.71], text[GRID.text.72], text[GRID.text.73], text[GRID.text.74], text[GRID.text.75], text[GRID.text.76])The Comprehensive Intention-Behavior Gap measure includes questions that aim to quantify the gap at all three levels (meta, values and action), as well as to access additional dimensions within each leave. These additional dimensions at the meta level are the proportion of the the gap that participants ascribe to outside causes vs internal causes. At the values level we will differentiate between the ought attribute and the ideal attribute gaps. And finally at the action level we will ask questions that allow us to estimate the gap across different timescales.
1.1.2 Item Generation
To identify appropriate questions that fit the identified domain
1.1.2.1 Deductive methods: literature review and assessment of existing scales
To develop the items for our measure we used a combination of deductive and inductive methods (Hinkin, 1995b), with an initial literature review and assessment of existing scales providing important initial deductive guidance in drafting questions, followed by testing on focus groups/colleagues whose feedback we used to adapt the measure.
Meta Level | We simply directly ask subjects to what extent they feel like they are fulfilling their potential, as well as to specify to what degree any gap that may exist is due to outside versus internal causes.
Value level | We felt that Higgins’ self-discrepancy measure (1997) with its “ought” and “ideal” categories did a good job of capturing the dimensions of the value level gap we are interested in. We also believed that Hardin & Lakin’s (2009) updated version of the measure made some valuable updates to the original measure, specifically in how it combined both an idiographic and nomothetic approach to its question delivery.
Explain how it works and why valuable
Explain the distinction between ought and ideal and why that was an issue and remained an issue with Hardin and how we modify that. (note that we say they CAN be the same - based on focus group feedback)
Addition of question about overall for both cateogries.
Addition of question about overalap and conflict.
Action Level | As mentioned, most existing measures generally tend to focus on a single specific goal, or intention, at the action level. A study by Wegwarth et al. (2014), which looked at stated intention to get the HPV vaccine and actual uptake at a second time point, is a good example of the narrow domain research into action gaps tends to focus on.
- General questions about accomplishing days tasks at different time scales
👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆👆
END
Attributes (define)
Own/Ideal
Other/Ought
Implementation across timeframes
Potential
Expectation
Begins like Higgins with warm up questions
Decided to replicate Hardin with 3 idiographic, then 2 nomothetic (ref that 5 is a good number)
Explain where nomothetic came from here?
Idea of attributes…RICH is not an attribute…nor is MARRIED
Attributes do not depend on your context…you can’t be rich if you have no assets but you can be friendly or determined.
Oxford: “a quality or feature regarded as a characteristic or inherent part of someone or something.”
Differed in terms of own/other viewpoints compared to Hardin - just picked one of each
- Explain how other/ought are kind of hard to disambiguate…already know that people have a hard time disambiguating…
Added explicit Intention-Behavior gap over 3 timescales
example of a “to do” list…more concrete than idea of embodying an attribute
also capture difference between intention-expectation…idea being that perhaps expectation differs both from intention and behavior. Give example (should be in the middle normally)
Added potential/expectation questions
Expect these to correlate with the other measures of the gap
Expectation is a non-time bound version of the previous questions’ ‘expect’ question
On expectation:
Warshaw and Davis (1985) drew a distinction between behavioral intentions and behavioral expectations. Whereas intentions are typically measured by responses to statements such as, “I intend to do X,” expectations are measured by estimates of the likelihood of performance, such as “How likely are you to do X?” Expectations are assumed to capture “various factors that could cause [participants] to be unsuccessful in their attempt to carry out [their] intentions” (Sheppard et al., 1988, p. 328) and should, therefore, possess greater predictive validity than intentions. Findings from Sheppard et al.’s (1988) meta-analysis supported this view, where the average correlations were 0.57 and 0.49 for expectations and intentions, respectively. However, more recent meta-analyses by Armitage and Conner (in press) and Sheeran and Orbell (1998) both failed to obtain reliable differences in the average correlations obtained for intentions versus expectations.
Also: https://pubmed.ncbi.nlm.nih.gov/25623893/
- Added intention-behavior gap “feeling” question
Expected Self
We believe (supporting evidence?) that expectations and goals do not always match. We may have an ideal goal of straight As and then actually end up with a B average. This would be captured by the ideal/actual self-discrepancy measure. However, our ideal goal and our true expectation may often differ. As an example, think of the sports fan who supports a perennially basement-dwelling team. Their ideal goal for the team would be that they win the league championship, but their (perhaps optimistic) true expectation is that the team will just make it onto the playoffs before being eliminated. This in turn could differ from the actual reality of the team finishing last in the league.
We therefore have two gap dimensions, goal-actual and expected-actual, which we combine into our comprehensive measure.
- another option could be to just give a list of activities (e.g. grades, health, relationships, honesty, study habits, punctuality, etc…) and have them indicate their expectation and actual achievement levels (where 100% is fulfilling their goals)
Time Horizon
We also think it is meaningful to think about goals in terms of time frame. It could be the case that someone is poor at planning their days, ending up with many items unchecked on their to-do list, but actually good at planning (and executing those plans) over the course of a year.
- How to operationalize this? Just ask them about how good they are at achieving their goals on different timescales?
Higgins:
Scoring
1-4 likert
increase granularity 0-100%?
Questions
- reduce questions…don’t ask to what degree you hope/ought to possess it but rather what proportion/percentage of the ideal amount you have achieved.
Timing
- don’t care about timing…but perhaps capture anyway?
Hardin
- Description
The type of person you ideally want to be refers to your beliefs concerning the attributes you would ideally like to possess; your ultimate goals for yourself.
The type of person you think you ought to be refers to your beliefs concerning the attributes you believe you should possess or that you are obligated to possess; your moral rules for yourself.
There are no wrong answers or wrong words—you may use any words you want to describe these different types of self. Sometimes it is hard to tell the difference between these different types of self.
Here is an example: As an example of how the ideal and ought are different, I may ideally want or hope to be rich someday, but I do not think I have a duty or am morally obligated to be rich. So, rich would be a word that describes the type of person I ideally want to be, but it is not a word that describes the type of person I think I ought to be.
Sequence
- idiographic (list 5 qualities) -> nomothetic (finish up or change; 100 adjectives)
assessment of self-discrepancies from standpoint of significant other?
- participants would be asked to, “Please list 5 qualities that you believe [your significant other] would ideally like for you to be or to possess.”
Scoring
- 5 point scale: 1=completely applies to me, 2=applies to me very much, 3=applies to me somewhat, 4=applies to me a little, and 5=doesn’t apply to me at all.
Add empathy
Sheeran and others here?
For nomothetic portion we selected the 50 most common attributes from Hardin
library(tidytext)
library(SnowballC)
# Import Hardin data
# ought
ought_atts = readxl::read_xls('../../3_1_1_raw_data/erin_hardin_attributes.xls', sheet = "Ought words", col_names = 'word')
# lowercase
ought_atts$word = tolower(ought_atts$word)
# ideal
ideal_atts = readxl::read_xls('../../3_1_1_raw_data/erin_hardin_attributes.xls', sheet = "Ideal")
# drop unused columns
ideal_atts = subset(ideal_atts, select = -c(SUBJ, Race))
# convert to long
ideal_atts <- pivot_longer(ideal_atts, cols=1:10, names_to = "label", values_to = "word")
# drop unused column
ideal_atts = subset(ideal_atts, select = -c(label))
# lowercase
ideal_atts$word = tolower(ideal_atts$word)
# unstemmed version
all_counts_raw = count(rbind(ideal_atts, ought_atts), word, sort = TRUE)
# Clean Text
# Remove stop words
ought_atts = ought_atts %>%
unnest_tokens(word, word) %>%
anti_join(stop_words)## Joining, by = "word"# Stem
ought_atts = ought_atts %>%
dplyr::mutate(word = wordStem(word))
# Remove stop words
ideal_atts = ideal_atts %>%
unnest_tokens(word, word) %>%
anti_join(stop_words)## Joining, by = "word"# Stem
ideal_atts = ideal_atts %>%
mutate(word = wordStem(word))
# combined
all_atts = rbind(ideal_atts, ought_atts)
# Counts
ideal_counts = count(ideal_atts, word, sort = TRUE)
ought_counts = count(ought_atts, word, sort = TRUE)
all_counts = count(all_atts, word, sort = TRUE)
# Plot
# ideal
ggplot(data = ideal_counts[0:10,], mapping = aes(x = n, y = reorder(word, n))) +
geom_bar(stat = "identity", aes(fill = n)) +
labs(x = "Count", y = "Attribute", title = "IDEAL ATTRIBUTES") +
theme_light()
# ought
ggplot(data = ought_counts[0:10,], mapping = aes(x = n, y = reorder(word, n))) +
geom_bar(stat = "identity", aes(fill = n)) +
labs(x = "Count", y = "Attribute", title = "OUGHT ATTRIBUTES") +
theme_light()
# top words stemmed
top = all_counts[1:50,]$word
top.alpha = sort(top)
print(top.alpha)## [1] "ambiti" "beauti" "care" "confid" "cultur"
## [6] "educ" "ethic" "faith" "famili" "famou"
## [11] "friend" "friendli" "fun" "funni" "gener"
## [16] "give" "happi" "hard" "head" "healthi"
## [21] "help" "honest" "honesti" "independ" "intellig"
## [26] "job" "kind" "life" "love" "marri"
## [31] "mind" "moral" "nice" "outgo" "patient"
## [36] "peopl" "person" "polit" "religi" "respect"
## [41] "respons" "rich" "smart" "strong" "success"
## [46] "trustworthi" "truth" "understand" "unselfish" "wealthi"# removed
removed = c('friendli', 'funni', 'marri', 'honesti', 'job', 'life', '')
# job, life, peopl, person don't make sense - probably word in phrase
# stemming loses some stuff, like cool-headed and hardworking and openminded (just has head and hard and mind)
# married just doesn't fit
# remove giving (same as generous?)
# top words unstemmed
top = all_counts_raw[1:68,]$word
top.alpha = sort(top)
print(top.alpha)## [1] "ambitious" "appreciative" "beautiful" "brilliant"
## [5] "caring" "confident" "cool-headed" "cultured"
## [9] "educated" "ethical" "faithful" "famous"
## [13] "fashionable" "forgiving" "friendly" "fun"
## [17] "funny" "generous" "gentle" "giving"
## [21] "good" "happiness" "happy" "hard working"
## [25] "hardworking" "healthy" "helpful" "honest"
## [29] "honesty" "hopeful" "independent" "intelligence"
## [33] "intelligent" "kind" "kindness" "love"
## [37] "loved" "loving" "loyal" "married"
## [41] "moral" "nice" "obedient" "open minded"
## [45] "open-minded" "outgoing" "patient" "pleasant"
## [49] "polite" "positive" "reasonable" "religious"
## [53] "respect" "respectful" "responsibility" "responsible"
## [57] "rich" "smart" "strong" "success"
## [61] "successful" "thoughtful" "trustworthy" "truthful"
## [65] "understanding" "unselfish" "wealth" "wealthy"Estimate Gap
To what degree do you think that you are maximizing your abilities?
How on track do you estimate you are in terms of achieving your goals?
Estimate first order preference gap: What percentage of what you want to do do you actually do?
Estimate second order gap: What percentage what you want to want to do do you actually do?
Questions
Intention behavior gap measure to capture overall magnitude via three dimensions:
Goal - Actual Gap
Ideal-Actual
Ought - Actual
Expectation - Actual Gap
Behavior - Outcome Gap
When I do things they generally work out (much worse than I expect - much better than I expect)
My grades tend to be (much better/worse) than I expect
Re-estimate gap
To what degree do you think that you are maximizing your abilities?
1.1.2.2 Inductive methods: exploratory research methodologies including focus group discussions and interviews
Completed a series of discussions with colleagues in the field…
1.2 Content Validity: Assessing if the Items Adequately Measure the Domain of Interest
1.2.1 Evaluation by experts
To evaluate each of the items constituting the domain for content relevance, representativeness, and technical quality
1.2.1.1 Quantify assessments of 5-7 expert judges using formalized scaling and statistical procedures including content validity ratio, content validity index, or Cohen’s coefficient alpha
Expert Judges?
Higgins
Hardin
Sheeran
1.2.1.2 Conduct Delphi method with expert judges
1.2.2 Evaluation by target population
To evaluate each item constituting the domain for representativeness of actual experience from target population
1.2.2.1 Conduct cognitive interviews with end users of scale items to evaluate face validity
2. Scale Development
2.1 Pre-testing Questions: Ensuring the Questions and Answers Are Meaningful
2.1.1 Cognitive interviews
To assess the extent to which questions reflect the domain of interest and that answers produce valid measurements
2.1.1.1 Administer draft questions to 5–15 interviewees in 2–3 rounds while allowing respondents to verbalize the mental process entailed in providing answers
*Maybe do this on SONA ASAP…
2.2 Survey Administration and Sample Size: Gathering Enough Data from the Right People
2.2.1 Survey Administration
To collect data with minimum measurement errors
2.2.1.1 Administer potential scale items on a sample that reflects range of target population using paper or device
2.2.2 Establishing the sample size
To ensure the availability of sufficient data for scale development
2.2.2.1 Recommended sample size is 10 respondents per survey item and/or 200-300 observations
2.2.3 Determining the type of data to use
To ensure the availability of data for scale development and validation
2.2.3.1 Use cross-sectional data for exploratory factor analysis
2.2.3.2 Use data from a second time point, at least 3 months later in a longitudinal dataset, or an independent sample for test of dimensionality (3.1)
2.3 Item Reduction: Ensuring Your Scale Is Parsimonious
2.3.1 Item difficulty index
To determine the proportion of correct answers given per item (CTT)
To determine the probability of a particular examinee correctly answering a given item (IRT)
2.3.1.1 Proportion can be calculated for CTT and item difficulty parameter estimated for IRT using statistical packages
2.3.2 Item discrimination test
To determine the degree to which an item or set of test questions are measuring a unitary attribute (CTT)
To determine how steeply the probability of correct response changes as ability increases (IRT)
2.3.2.1 Estimate biserial correlations or item discrimination parameter using statistical packages
2.3.3 Inter-item and item-total correlations
To determine the correlations between scale items, as well as the correlations between each item and sum score of scale items
2.3.3.1 Estimate inter-item/item communalities, item-total, and adjusted item-total correlations using statistical packages
2.3.4 Distractor efficiency analysis
To determine the distribution of incorrect options and how they contribute to the quality of items
2.3.4.1 Estimate distractor analysis using statistical packages
2.3.5 Deleting or imputing missing cases
To ensure the availability of complete cases for scale development
2.3.5.1 Delete items with many cases that are permanently missing, or use multiple imputation or full information maximum likelihood for imputation of data
2.4 Extraction of Factors: Exploring the Number of Latent Constructs that Fit Your Observed Data
2.4.1 Factor Analysis
To determine the optimal number of factors or domains that fit a set of items
2.4.1.1 Use scree plots, exploratory factor analysis, parallel analysis,minimum average partial procedure, and/or the Hull method
3. Scale Evaluation
3.1 Tests of Dimensionality: Testing if Latent Constructs Are as Hypothesized
3.2 Tests of Reliability: Establishing if Responses Are Consistent When Repeated
3.2.1 Calculate reliability statistics
To assess the internal consistency of the scale. i.e., the degree to which the set of items in the scale co-vary, relative to their sum score
3.2.1.1 Estimate using Cronbach’s alpha
3.2.1.2 Other tests such as Raykov’s rho, ordinal alpha, and Revelle’s beta can be used to assess scale reliability
3.2.2 Test–retest reliability
To assess the degree to which the participant’s performance is repeatable; i.e., how consistent their scores are across time
3.2.2.1 Estimate the strength of the relationship between scale items over two or three time points; variety of measures possible
3.3 Tests of Validity: Ensuring You Measure the Latent Dimension You Intended
3.3.1 Criterion validity: Predictive validity
To determine if scores predict future outcomes
3.3.1.1 Use bivariate and multivariable regression; stronger and significant associations or causal effects suggest greater predictive validity
3.3.2 Criterion validity: Concurrent validity
To determine the extent to which scale scores have a stronger relationship with criterion measurements made near the time of administration
3.3.2.1 Estimate the association between scale scores and “gold standard” of scale measurement; stronger significant association in Pearson product-moment correlation suggests support for concurrent validity
3.3.3 Construct validity: Convergent validity
To examine if the same concept measured in different ways yields similar results
3.3.3.1 Estimate the relationship between scale scores and similar constructs using multitrait multi-method matrix, latent variable modeling, or Pearson product-moment coefficient;higher/stronger correlation coefficients suggest support for convergent validity
3.3.4 Construct validity: Discriminant validity
To examine if the concept measured is different from some other concept
3.3.4.1 Estimate the relationship between scale scores and distinct constructs using multi-trait multi-method matrix, latent variable modeling, or Pearson product-moment coefficient;lower/weaker correlation coefficients suggest support for discriminant validity
3.3.5 Construct validity: Differentiation by “known groups”
To examine if the concept measured behaves as expected in relation to “known groups”
3.3.5.1 Select known binary variables based on theoretical and empirical knowledge and determine the distribution of the scale scores over the known groups; use t-tests if binary, ANOVA if multiple groups
3.3.6 Construct validity: Correlation analysis
To determine the relationship between existing measures or variables and newly developed scale scores
3.3.6.1 Correlate scale scores and existing measures or, preferably, use linear regression, intraclass correlation coefficient, and analysis of standard deviations of the differences between scores
Discussion
attributes are a good way of getting at this as general level
one action can relate to multiple attributes…they may achieve some goals but not others. If we are talking to a friend who has just gone through a bad breakup but have a meeting we need to leave for right away if we are to be on time it will be hard to avoid having a gap…trying to be responsible and operationalize that as always being punctual that may mean that we are in conflict with our desire to be caring.
idea that you can have a lower level intention that you successfully enact but then realize that it conflicts with your actual goal if it backfires…
specific goals like getting an A in a class or losing 10 pounds we expect to be connected to the attributes that are reported
If there are goals that are not represented then we have the other measures…
needs to rely on participants self-reports since we do not have access to their intentions and, even if we did, we cannot feasibly observe all their behaviors.
Idea of time
less practical…
concrete…can have an intention to be honest…
What makes this comprehensive? In one sense we use the term comprehensive simply to distinguish the goal of our measure in comparison to the narrowly focused measurements of intention-behavior gaps (e.g. exercise, condom-use, etc.) that dominate the literature.
R Markdown Tips
Citations
Citations go inside square brackets and are separated by semicolons. Each citation must have a key, composed of ‘@’ + the citation identifier from the database, and may optionally have a prefix, a locator, and a suffix. Here are some examples:
Blah blah [see @doe99, pp. 33-35; also @smith04, ch. 1].
Blah blah [@doe99, pp. 33-35, 38-39 and *passim*].
Blah blah [@smith04; @doe99].A minus sign (-) before the @ will suppress mention of the author in the citation. This can be useful when the author is already mentioned in the text:
Smith says blah [-@smith04].You can also write an in-text citation, as follows:
@smith04 says blah.
@smith04 [p. 33] says blah.Directed Acyclic Graphs in R
https://cran.r-project.org/web/packages/ggdag/vignettes/intro-to-dags.html