Colombia’s Institute for educational evaluation (ICFES) evaluates the quality of technical and technological education through the TyT tests that are applied to students who have completed at least 75% of the credits of the program in which they are enrolled. This test evaluates five generic skills, among which is the critical reading competency.

The reflection on the development of critical reading in higher education in the country is permanent, explainable interest if the decisive role of this competition for productivity and competitiveness is taken into account. There is a large number of studies in this regard. However, these focus on the professional academic level, leaving a knowledge gap about the development of generic critical reading skills in Professional Technical and Technological Education. Critical reading must be explicitly addressed in the training processes because it is a generic competence necessary for the workplace.

Databases provided by the ICFES, which account for the levels of performance in critical reading achieved by students at the technical and technological high education levels, is an opportunity to conduct a rigorous analysis based on institutional practices that exist for the development of reading skills and its relationship with the performances achieved by students at these levels of training.

In this sense, the research question arises: Is there a relationship between the presence of specific subjects or actions for the development of reading skills in the technological programs taught at high education programs in Bogotá and the results of the TyT tests in critical reading?

As this is an example of a portfolio, not all variables and methodological considerations are explained. It is just a small extract. The complete document of the final version can be requested from ICFES: “Critical reading in technological education in Bogotá”

…..

Forty complete data sets were prepared, each with different assignments for each missing value, representing uncertainty. Through the processes explained in the methodological chapter, models with various interactions between the variables of interest were adjusted for each group of complete data. Then the standard errors were calculated using the weighting process based on Rubin’s rules, also explained in the methodological chapter. This procedure only applies to Model I. Model II was adjusted in the final version according to the formulation of the first to make them comparable, except that there was no imputation, and those students that do not cross with the data from the secondary education level test (SABER 11) were suppressed.

For the final selection of variables that were part of the model I, both at the student level and in the programs, a step-by-step variable addition process was applied following what was explained in the methodological chapter.

It should be noted that programs can apply a combination of strategies. They can include in the curriculum the teaching of reading and to teach to read by means of disciplinary subject texts, have articulation strategies between these subjects and the rest, choose structured or unstructured strategies, and accompany them with preparation for SABER TyT tests. All these combinations were modeled.

Some figures are shown in Table 1.

Table 1. Statistics on the number of students.
Model I
Model II
Statistics High Education Institute Program High Education Institute Program
Minimum 36 36 36 36
Average 646 185 617 176
Maximum 7932 2982 7443 2831

Note that Model I obtains figures similar to Model II in terms of the number of records. The model I gains information through imputation. Model II increases knowledge by incorporating all students, discarding those that do not intersect with SABER 11 test data.

Due to the various indicator variables incorporated, there were some selected as reference categories and, therefore, they are not explicit in the model. Father having primary studies, the mother having secondary studies, surfing the Internet between 61 and 240 minutes a day, not having books at home, or access to the Internet, or PCs, not reading for entertainment, and studying in a public-based technology institution of the Administration and Tourism group, being a woman, having an average socioeconomic index, and having the average age configures the reference student.

Table 2 presents the results. Variables not present were eliminated in the stepwise step selection process. It is presented for each variable: its coefficient, its corresponding standard deviation and its level of significance.

Table 2. Results of the adjusted model
Variable Model I Model II
Reference student score 97,644 95,314
(4,15061) (3,2645)
*** ***
isSubject 4,009 5,632
(2,20676) (2,7718)
.
Disciplinary -1,241 -6,034
(4,8472) (5,6619)
Structured_strategy -2,654 0,943
(2,98784) (3,6823)
No_ structured_strategy 7,788 3,356
(5,43627) (5,568)
TyT_prepare 0,019 -0,478
(1,99951) (2,537)
Sex 0,46 1,213
(0,29838) (0,3258)
***
Scaled_age 2,736 -0,885
(0,24011) (0,2229)
*** ***
Scaled_age2 -0,383 -0,271
(0,08568) (0,0931)
*** **
Less_60_min_internet 0,389 0,656
(0,30691) (0,3378)
More_240_min_internet -1,689 -2,04
(0,36906) (0,4019)
*** ***
11_to_25_books 1,865 2,804
(0,32289) (0,3543)
*** ***
26_to_100_books 3,453 5,277
(0,36587) (0,3947)
*** ***
more_than_100_books 3,738 5,68
(0,58567) (0,6345)
*** ***
less_30_min_reading 0,495 0,424
(0,40573) (0,4414)
between_30_and_60_min_reading 1,873 2,048
(0,4373) (0,4718)
*** ***
between_61_and_120_min_reading 1,81 2,235
(0,57566) (0,6314)
** ***
More_than_120_min_reading 1,148 1,357
(0,94443) (1,0401)
mother_no_studies -3,999 NA
(1,13623)
***
mother_primary -1,217 NA
(0,35922)
***
mother_graduated -1,352 NA
(0,51956)
**
father_no_studies -0,368 NA
(0,66926)
father_secondary 0,097 NA
(0,33857)
father_graduated 0,508 NA
(0,58313)
Internet_access 0,863 1,237
(0,45571) (0,4893)
*
PC_access 1,501 1,977
(0,50485) (0,5485)
** ***
centered_lang_score 0,757 NA
(0,02189)
***
centered_math_score 0,435 NA
(0,01887)
***
SES_ind_centered 0,115 0,174
(0,02255) (0,021)
*** ***
mean_SES_ind_centered_prog 0,813 0,258
(0,3942) (0,2076)
AccreditedProgram 1,317 0,934
(1,61655) (1,9163)
mean_centered_math_score_prog 0,21 NA
(0,30513)
mean_father_no_studies_prog -9,135 NA
(18,64915)
mean_father_secundary_prog 12,59 NA
(7,70206)
mean_father_graduated_prog -27,395 NA
(14,10608)
HEI_accredited 5,446 7,171
(2,35122) (2,8238)
* *
mean_SES_ind_centered_prog 0,813 0,258
(0,3942) (0,2076)
SENA -0,874 -2,199
(2,55468) (2,754)
university_institution 0,007 0,141
(1,31046) (1,5479)
university 0,696 3,686
(1,52809) (1,7088)
. .
Private -1,62 -3,082
(1,69) (2,2919)

Confidence level: *** 99.9% or more; ** 99% or more; * 95% or more; . 90% or more.

Table 3. Variance components of the fitted model.
Estimated Variance Standard Dev. Variance Standard Dev.
Program level 6.84 2.61 12.13 3.48
Students level 310.23 17.61 372.42 19.30

There is no evidence that the presence of strategies, alone or in the company of others, is associated with changes in the average score of the reference student.

The lack of evidence should be understood to be “on average.” Figure 1 shows, based on the random component, the differences between programs. There are some whose confidence interval is outside of zero; that is, they differ from the average. And keep in mind that the calculations make extreme cases tend to average (shrinkage).

Figure 1. Academic programs, according to the estimate of the intercept average. Model I.

Figure 1. Academic programs, according to the estimate of the intercept average. Model I.

Source: Authorship.

With the adjusted model data, the model I diagnosis and identification of influential programs were carried out, as described in the methodology. The HLMdiag v0.2.2 R package allows, in general, to perform the diagnosis of two-level models, and lets, in particular, to automate the adjustment of the multilevel regression model by eliminating one program at a time to identify influential programs.

Twelve programs were shown as influential. They are characterized by having, on average, students with higher SES than the general average, but not all. However, their respective students have SABER 11 scores, on average, higher than the general average, so even if they do not receive students with a high SES, they are allowed to select them, probably because the demand for quotas exceeds what they can offer.

The results of applying the same regression model to the data, excluding these twelve programs are, for practical purposes, the same.

Another sensitivity analysis was to eliminate outlier programs in terms of the age variable. Two programs presented students with an average age of over 40, seemingly not typical programs, but structured for a specific situation. The results also did not change significantly.

Regarding the remaining variables incorporated into the model, from the reading in Table 2, the following are significant:

Having access to books at home, compared to not owning them, is positive. It increases 1.9 points if you have 11 to 25 books, 3.5 points if you have between 25 and 100, and 3.7 points if you have more than 100.

Having access to a PC increases the score by 1.5 points on average.

Against not reading for entertainment, who reads between half an hour and 2 hours, increases 1.8 points. And reading more than two hours increases it a little less: 1.1 points.

In front of surfing the Internet between 30 and 240 minutes a day, overcoming that time reduces the score by 1.7 points.

With respect to a student whose mother has a high school education, that the mother has primary education would decrease the score of the reference student by 1.2 points. And if she doesn’t have studies, she would drop it by 4 points. But having tertiary studies would also decrease the reference student’s score by 1.3 points. This last data is not in line with what was expected, but the low number of students with parents with tertiary education forces not to consider the data robust.

The relative location of the student concerning the average socioeconomic status is directly related to the score. For each change in 10 points in the SES, I would change the TyT score by 1.2 points. The centered SES varies between -35 and 35. Therefore, a student can vary his critical reading score by a maximum of 3.86 based on his socioeconomic conditions.

The SABER 11 test score in language and mathematics is directly related to the SABER TyT critical reading test result. For each variation of 10 points in language, I would change the SABER TyT score by 7.6 points. The scores vary between -37 and 35. In this case, a student could change the maximum critical reading score between -28.12 and +26.6 points, significant figures.

And regarding mathematical knowledge, every 10 points of variation would change the SABER TyT score by 4.4 points. Mathematics scores vary between -48 and 41, and therefore the maximum critical reading score could range between 21 points less and 18 additional.

Indeed, in Model II, where the SABER 11 scores in language and mathematics are absent, the size of the coefficients grows in the other variables: Having access to the Internet, reading for entertainment, number of books at home.

It is striking that sex has significance in Model II, in favor of men, not in Model I.

Age influences. There is a growing effect from the age of 17 that reaches its maximum at 49 in Model I, age from which it begins to decrease. It is what is called a quadratic effect. In Model II, this quadratic form is maintained, but the age at which it reaches its maximum varies slightly.

Institutional accreditation is an indicator that has a direct relationship with critical reading scores. A reference student, located in an accredited institution, gets 5.5 points above.

Concerning model II, it is also striking that the growth in the SES is not as substantial as in the variables mentioned above. The effect seems to shift to the variable being an accredited institution. It means that they capture a good number of students with better SABER 11 scores.