Construct Validity of the VI-SPDAT

Jiyoon Park
March 26, 2019

VI-SPDAT

The Vulnerability Index-Service Prioritization Decision Assistance Tool (VI-SPDAT; Community Solutions & OrgCode Consulting, Inc., 2014) is the primary assessment tool the District's continuum uses to support matching decisions during the Coordinated Entry process. It was designed for rapid, interview-style administration and self-report. The VI-SPDAT is a 26-item summated rating scale based on 4 content domains (History of Housing and Homelessness; Risks; Socialization & Daily Functioning; Wellness). Due to its ease of use and other potential strengths (https://bitfocus.com/homeless-management-information-system-hmis/vi-spdat-standardized-assessment/), the VI-SPDAT has been widely adopted for coordinated assessment, now on its second version. Such an approach in housing prioritization assumes the scores derived from the VI-SPDAT represent the degree of a person's vulnerability (or self-sufficiency). However, there have been concerns in scores from VI-SPDAT used to prioritize housing resources; that the scores do not represent vulnerability of people experiencing homelessness, therefore, creating sequential issues in housing prioritization process. Our study is rooted from this anecdotal concern about VI-SPDAT that is intended to assess vulnerability.

Scores from the VI-SPDAT are used to make an infererence about housing need. In order to make a valid inference, the assessment tool should be supported by evidence from scientific methods. U.S. Department of Housing and Urban Development (2015) provided recommendations to adopt evidence-based assessments. However, based on our research, evidence of the validity of the VI-SPDAT and other coordinated assessment tools is limited. Although the developers asserted that the VI-SPDAT is "evidence-informed, criteria-driven," and "the strongest tool" available based upon its evidence and testing (https://bitfocus.com/homeless-management-information-system-hmis/vi-spdat-standardized-assessment/), to our knowledge, validity of the VI-SPDAT has not been scientifically measured. Moreoever, despite the wide uses of the VI-SPDAT across the states in U.S., little evidence exists supporting the validity of the VI-SPDAT when used within homeless population. Therefore, whether or not the VI-SPDAT is a valid tool to represent vulnerability is unknown. Study on validity of the VI-SPDAT is critical due to its implications for housing outcomes of individuals who experience homelessness. In order to address these limitations the current research will employ a validation study of the VI-SPDAT.

What is Validity and How Is It Supported?

Instrument validation is a critical step that both researchers and practitioners should employ in order to ensure the generation of scientifically valid knowledge. Without it, the basis of research findings and the generalization of such are threatened. This is especially true in measuring psychological aspects (e.g., vulnerability) where majority of practitioners utilize subjective instruments to assess that aspects or to collect data. Consequently, validation process for any uses of instuments has increasingly become common practice. Without validation process based on methodologies scientifically approved, the quality of data and the quality of the descisions and inferences made based on the scores from instruments are not meaningful.

Validity refers to the quality of the inferences, claims, or descisions drawn from the scores of an instrument. Validation is the process in which we gather and evaluate the evidence to support the appropriateness, meaningfulness, and usefulness of the descisions and inferences that can be made from instrument scores. The VI-SPDAT for singles is intended to measure vulnerability of individuals who are experiencing homelessness. One validation question for the VI-SPDAT can be about content; are the items, format and wording of the items, response options, and the administration and scoring procesures appropriately written (Content validity)? Evidence for content validity can be obtained from examining the relationship between the content of the VI-SPDAT and the construct (vulnerabiltiy) it intends to measure. Among different types of validity, construct validity is the focus of validity.

Construct Validity

A construct, or psychological construct, is an attribute, proficiency, ability, or skill that happens in the human brain and is defiend by established theories. In the case of measuring vulnerability, "overall level of vulnerability" is a construct. It exists in theory and has been observed in practice. Construct validity is involved whenever a test is to be interpreted as a measure of some attribute or quality which is not "operationally defined". (Cronbach & Meehl, 1955). Construct validity would be involved in answering such questions as: To what extent in this assessment culture-free? Does this assessment of "vulnerability" measure degree of vulnerability?

Although it is clear to defined the construct, there is not best way to study of how construct validity is demonstrated. The more strategies used to demonstrate the validity of a test, the more confidence test users have in the construct validity of the test, but only if the evidence provided by those strategies is convincing (Cronbach & Meehl, 1955; Messick, 1977).A wide variety of research designs and statistical approaches can be used to gather construct validity evidence, including dimensionality from factor analysis, item-test correlations (including reliability), measurement invariance, differential item functioning (DIF), etc (Cronbach & Meehl, 1955). As sequential validation study, in the current study, we examine construct validity of the VI-SPDAT, espcially on reliability and dimensionality.

Method

Data and Scoring of the VI-SPDAT Questions

Data were limited to the records obtained from invidiual homeless clients who have taken VI-SPDAT between August 2016 and March 2019 due to administration timeframe of the new VI-SPDAT form (VI-SPDAT v2.0) was utilized in D.C. We obtained 18,717 individual records of people from HMIS. Among these records, only those with the most recent scores were used for the current study, which includes 12,722 records. This sample was utilized for examination of the VI-SPDAT reliability and factor structure. Table 1 presents numbers of score records obtained within each year's timeframe.

Table 1. Number of VI-SPDAT Records Each Year

Categorical and frequency items were dichotomized based on the scoring system as it was originally designed. Due to unclear summation scoring method of the original VI-SPDAT, we dichotomized each item (scored 0 or 1) instead of using its own scoring methods. For example, in case a question has several sub-questions with one score (1 or 0), each sub-question was dichotomized based on endorsement of the question asked. This way, we were be able to examine which item does (or does not) contribute the total reliability or factor structure as evidence of construct validity. One question was not included in the analysis (the Tri-Morbidity), as it is a prescreen score item dependent on other items in the Wellness domain (DePaul's study).

Analysis

Construct validity of the VI-SPDAT was examined in two approaches, 1) factor structure of the VI-SPDAT, and 2) internal consistency of the instrument and item discrimination of each items. Confirmatory factor analysis (CFA) was used for factor structure and reliability anlaysis was utilized for internal consistency test. In order to examine whether or not the results are similar over time, anlayses were conducted for the entire data across the years as well as for the data in each year.

Factor Structure of the VI-SPDAT

In practice, the VI-SPDAT is utilized as a unidimensional scale measuring overall vulnerability. If this is true, our data will show a satisfactory model fit to a single factor model, with the global factor, "vulnerability" as a latent factor measured by 34 question variables. Two differnet factor models were fitted to our sample; 1) a single factor model as one global factor measured by 34 question variables (See Figure 1), and 2) a bifactor model as one primary factor (vulnerability) with four secondory factors (four content domains: Homeless history, Risks, Wellness, and Socialization/Daily Functioning) (See Figure 2). These two factor models were selected from the original design and structure of the VI-SPDAT. Model fits for these two factor models were assessed and compared from goodness-of-fit tests. CFI (Comparative Fit Index), TLI (Tucker-Lewis Index), RMSEA (Root Mean Square Rrror Approximation), and SRMR (Standardized root mean square residual) were taken into consideration. For these fit statistics, the RMSEA value must be below 0.05 for the fitness of the model (Browne and Cudeck, 1993), SRMR values must be below 0.10 (Kline, 2005), the CFI values greater than 0.90 (Brown, 2006), and the TLI values greater than 0.95 to meet the fit criteria. Software R was utilized to run CFA analysis.

Figure 1. Single Factor Model

Figure 2. Bifactor Model

Reliability of the VI-SPDAT

There are mainly three ways to study about reliability of an instrument; 1) whether or not the scores measured from the instrument would be consistent over time (test-retest reliability), 2) whether or not the scores assessed for the same person by different administrators would be the same (inter-rater reliability), and 3) the extent to which individual items produce results consistent with the overall questionnaire (internal consistency). We explored internal consistency only in the current study. For test-retest reliability, in the cases where people take the VI-SPDAT multiple times over time, it is unclear whether or not changes in their responses are due to inconsistency of the instrument or due to the participant's condition changes. Since there is no definitive way to control participants' condition changes (e.g., changes in health condition), measures of test-retest reliability would not be meaningful. Inter-rater reliability was not explored due to quality of data obtained from HMIS; there are several missing information of rater identifier. To examine internal consistency of the VI-SPDAT, we try to answer the question, "to what extent participants' responses to each item in the VI-SPDAT are consistent across the items on the VI-SPDAT?". The Cronbach's alpha was used as a measure of consistency for the entire instrument. In order to check item-level reliability (i.e. the extent to which each item is measuring the construct consistently), we examined item-total correlation as a way to examine item discrimination for each item. Item-total correlations are the correlations between each item and the total score from the VI-SPDAT. If the VI-SPDAT is reliable and consistent across the items, all items should correlate well with the total score. The items that do not correlate well with the total score would indicate low consistency to the total score measured for the latent trait, vulnerability.

Results

Findings from Confirmatory Factor Analysis

Factor structure was explored to examine whether the VI-SPDAT shows unidimensional factor indicating items of the instrument are measuring the latent trait, vulnerability. Single factor model with one global factor (vulnerability) and bifactor model with one global factor and 4 domain factors were used in order to test the original structure of the VI-SPDAT. To examine changes of factor structures over time, the full sample as well as yearly samples between 2016 and 2019 (Total N = 12,722) were used. Table 2 presents model fit statistics from two CFA models for two sets of sample (full sample versus sample by year). Our result showed that niether one single factor model nor bi-factor model with four sub-factors, did not have satisfactory fit (Table 2 and Figure 1). Only RMSEA values and SRMR values in bifactor model barely meet the fit criteria. Table 3 presents standardized factor loadings of the two CFA models.

Table 2. Goodness-of-Fit Statistics from CFA Models

Note: () indicates that this model satisfies fit criteria.

Figure 1.Goodness-of-Fit Statistics from CFA Models

Findings from Reliability Analysis

The reliability anlaysis was carried out from the perspective of internal consistency. The internal consistency of the 34 individually dichotomized items was estimated by Cronbach's Alpha method. The reliability coefficient obtained from the full sample is 0.832 highly significant with p-value <0.001, indicating that the degree of homogeneity among 34 items is quite acceptable. Similar results were obtained from the yearly sample (2016 to 2019) with reliability coefficients ranged between 0.82 and 0.84 (Table 4).

Table 4. Cronbach-Alpha of the VI-SPDAT (Reliability)

According to the Classical Test Theory (CTT), the disrimination value (item-total correlation) calculated in the form of the correlation coefficient varies between -1 and 1. It is necessary to investigate questions with a low item-total correlation, and remove them from the test if necessary (Crocker and Algona, 1986). The item-total correlation values are presented in Table 5 and Figure 2.

Table 5. Item Total Correlations

As is seen from Table 5, the correlation coefficients of the 34 items range between 0 and 0.61. The items with the correlation coefficients higher than 0.30 indicate that the item can serve to measure the relevant factor significantly (Pall ant, 2007). In our analysis, ten items do not meet this criteria indicating these items do not assess the latent variable, vulnerability, in a consistent way, therefore, need to be revisited, revised, or removed.

Figure 2.Item-Total Correlation of Each Item

*Note: The items having alpha < .3 were flagged due to low item-total correlations. This result shows that ten items did not meet criteria of item-total correlation (Three items from homeless domain, 4 items from Wellness domain, 2 items from Wellness domain, and one item from Risk Domain).

Discussion and Future Study

The main goal of the study was to explore the construct validity of the VI-SPDAT. Confirmatory factor analysis was utilized to determine the factor structure of the VI-SPDAT with two factor models--single factor model with the global factor of vulnarability and bifactor model with the global factor and four domain factors as secondary factors. In this analysis, both of these factor models fit poorly our sample. Several items on the VI-SPDAT were not associated well or were negatively associated with the global factor of vulnarability and/or with the four domain factors. Results from factor analysis suggest that the construct validity of the VI-SPDAT may be threatened due to those items that show unexpected association to the global factor. This result is similar to the ifndings of Brown et al. (2018).
Reliability of the VI-SPDAT was conducted from the cronbach-alpha method and the item-total correlations in order to determine the extent to which the instrument is able to measure the vulnerability consistently. Cronbach-alpha value for the entire sample and yearly sample range between 0.81 and 0.84 promising that the total scores from the VI-SPDAT may be consistent across the items. However, the correlation values between each item of the VI-SPDAT and total scores vary between 0 and 0.61. There are ten items that have low item-total correlations indicating construct validity of the VI-SPDAT may be improved by revising or removing these items. Consequently, it can be said that the VI-SPDAT has lack of evidence to support of it's construct validity. This may be caused by design of the instrument, implementation of the tool, or administration of the assessment (e.g., lack of standardized training interviewers).

Reference