These are a few commands to include the libraries and data to be used. I find it useful to see the errors and allow RMarkdown to continue knitting when there are errors. Thus, I included “error = TRUE”.
library(tidyverse)
## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr 1.2.1 ✔ readr 2.2.0
## ✔ forcats 1.0.1 ✔ stringr 1.6.0
## ✔ ggplot2 4.0.3 ✔ tibble 3.3.1
## ✔ lubridate 1.9.5 ✔ tidyr 1.3.2
## ✔ purrr 1.2.2
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(openintro)
## Loading required package: airports
## Loading required package: cherryblossom
## Loading required package: usdata
library(infer)
library(ggplot2)
data('hfi', package='openintro')
The data for the exercise
hfi
## # A tibble: 1,458 × 123
## year ISO_code countries region pf_rol_procedural pf_rol_civil
## <dbl> <chr> <chr> <chr> <dbl> <dbl>
## 1 2016 ALB Albania Eastern Europe 6.66 4.55
## 2 2016 DZA Algeria Middle East & North… NA NA
## 3 2016 AGO Angola Sub-Saharan Africa NA NA
## 4 2016 ARG Argentina Latin America & the… 7.10 5.79
## 5 2016 ARM Armenia Caucasus & Central … NA NA
## 6 2016 AUS Australia Oceania 8.44 7.53
## 7 2016 AUT Austria Western Europe 8.97 7.87
## 8 2016 AZE Azerbaijan Caucasus & Central … NA NA
## 9 2016 BHS Bahamas Latin America & the… 6.93 6.01
## 10 2016 BHR Bahrain Middle East & North… NA NA
## # ℹ 1,448 more rows
## # ℹ 117 more variables: pf_rol_criminal <dbl>, pf_rol <dbl>,
## # pf_ss_homicide <dbl>, pf_ss_disappearances_disap <dbl>,
## # pf_ss_disappearances_violent <dbl>, pf_ss_disappearances_organized <dbl>,
## # pf_ss_disappearances_fatalities <dbl>, pf_ss_disappearances_injuries <dbl>,
## # pf_ss_disappearances <dbl>, pf_ss_women_fgm <dbl>,
## # pf_ss_women_missing <dbl>, pf_ss_women_inheritance_widows <dbl>, …
First, I try to explore the elements in the data set
?hfi
## starting httpd help server ... done
In particular, I want to know the period covered and the number of countries.
summary(hfi$year)
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 2008 2010 2012 2012 2014 2016
The data set has 1,458 observations. However, this is due to the data frame having several observations for the same country as it covers nine years (i.e. the period 2008-2016).
summary(hfi$countries)
## Length N.unique N.blank Min.nchar Max.nchar
## 1458 162 0 4 22
The data frame includes 162 countries (162 x 9 = 1,458). They are grouped in ten regions.
Also, it has 123 variables, These variables can be divided in two groups. The ones about the observations (country name, country code, year, etc), and the substantive ones (about justice, freedom, etc). The latter can also be divided in groups. This can be seen in their names (e.g. about human freedom, economic freedom, etc).
I was particularly interested in the formula of the index. I have searched various websites (e.g. https://www.researchgate.net/profile/Tanja-Porcnik/publication/324864992_The_Human_Freedom_Index_A_Global_Measurement_of_Personal_Civil_and_Economic_Freedom/links/5ae85e65a6fdcc03cd8e2eef/The-Human-Freedom-Index-A-Global-Measurement-of-Personal-Civil-and-Economic-Freedom.pdf and https://www.fraserinstitute.org/sites/default/files/2025-12/human-freedom-index-2025.pdf).
Although I could not find the formula, one point seems to be clear: the contribution of each component, all else being equal, is linear. The index seems to be constructed as a simple average of indicators.
First, I explore the data.
summary(hfi$pf_score)
## Min. 1st Qu. Median Mean 3rd Qu. Max. NAs
## 2.167 6.197 7.189 7.201 8.449 9.568 80
summary(hfi$pf_expression_control)
## Min. 1st Qu. Median Mean 3rd Qu. Max. NAs
## 0.000 3.750 5.250 5.259 7.250 9.250 80
By construction, both variables are bounded between 0 and 10. Empirically, pf_expression_control starts at 0 and reaches 9.25, but pf_score only starts at 2.167 and reaches 9.568
Following the exercise instructions, the plot can be constructed with:
ggplot(data = hfi, aes(x = pf_expression_control, y = pf_score)) +
geom_point()
## Warning: Removed 80 rows containing missing values or values outside the scale range
## (`geom_point()`).
I have also produced a scatter plot with both axes ranging between 0 and 10. This graph helps me better visualize the relationship.
ggplot(data = hfi, aes(x = pf_expression_control, y = pf_score)) +
geom_point()+
xlim(0,10) + ylim(0,10)
## Warning: Removed 80 rows containing missing values or values outside the scale range
## (`geom_point()`).
While the relationship looks somewhat linear, there are two elements to consider. One is that the response variable is bounded (at 10). This raises the possibility that a high value of pf_expression_control could result in a value of pf_score above 10. However, as per construction (the formula of the index) pf_score is an average of several variables, including pf_expression_control, this may not be an issue.
Secondly, looking at the highest pf_score values there may be a flattening effect, in particular after pf_expression_control surpasses 5. The formula to calculate pf_score should be further analyzed.
Clearly, higher values of pf_expression_control lead to higher values of pf_score.
The scatter plot shows a “dense” cloud. Following the instructions, assuming linearity, the strength of the relationship can be explored with:
hfi |>
summarise(cor(pf_expression_control, pf_score, use = "complete.obs"))
## # A tibble: 1 × 1
## `cor(pf_expression_control, pf_score, use = "complete.obs")`
## <dbl>
## 1 0.796
If the relationship were linear, it would be considered rather strong.
Even without considering values which are a bit “outside of the dense” cloud, for any given value of pf_expression_control the range of possible values for pf_score vary widely. E.g for a pf_expression_control of 2.5 pf_score varies between a bit over 4 to 7 and for a pf_expression_control of 5 pf_score varies between 5 and 9.
However, this wide variability diminishes at higher values of pf_expression_control. Thus, the country with a pf_score of 5 for a pf_expression_control of almost 9, looks unusual.
remotes::install_github("jbryer/DATA606")
## Skipping install of 'DATA606' from a github remote, the SHA1 (919d6624) has not changed since last install.
## Use `force = TRUE` to force installation
library('DATA606')
## Loading required package: shiny
## Warning: package 'shiny' was built under R version 4.6.2
##
## Attaching package: 'shiny'
## The following object is masked from 'package:infer':
##
## observe
## Loading required package: markdown
##
## Welcome to CUNY DATA606 Statistics and Probability for Data Analytics
## This package is designed to support this course. The text book used
## is OpenIntro Statistics, 4th Edition. You can read this by typing
## vignette('os4') or visit www.OpenIntro.org.
##
## The getLabs() function will return a list of the labs available.
##
## The demo(package='DATA606') will list the demos that are available.
##
## Attaching package: 'DATA606'
## The following objects are masked from 'package:openintro':
##
## calc_streak, famuss, nhanes.samp.adult.500, present, qqnormsim
## The following object is masked from 'package:utils':
##
## demo
hfi <- hfi |> filter(complete.cases(pf_expression_control, pf_score))
DATA606::plot_ss(x = hfi$pf_expression_control, y = hfi$pf_score)
## Click two points to make a line.
## Call:
## lm(formula = y ~ x, data = pts)
##
## Coefficients:
## (Intercept) x
## 4.6171 0.4914
##
## Sum of Squares: 952.153
Apologies, the line:
“## Click two points to make a line.”
which appears in the knitted version, does not appear in my console. Thus, I was not able to explore different lines to choose the one which seems to minimize the sum of squares. The line that has come up automatically is the best one, as it has the same intercept and slope as the one estimated in exercise 6.
DATA606::plot_ss(x = hfi$pf_expression_control, y = hfi$pf_score, showSquares = TRUE)
## Click two points to make a line.
## Call:
## lm(formula = y ~ x, data = pts)
##
## Coefficients:
## (Intercept) x
## 4.6171 0.4914
##
## Sum of Squares: 952.153
The linear model regression estimates can be obtained with:
hfregr <- lm(hf_score ~ pf_expression_control, data = hfi)
The results are:
summary(hfregr)
##
## Call:
## lm(formula = hf_score ~ pf_expression_control, data = hfi)
##
## Residuals:
## Min 1Q Median 3Q Max
## -2.6198 -0.4908 0.1031 0.4703 2.2933
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 5.153687 0.046070 111.87 <2e-16 ***
## pf_expression_control 0.349862 0.008067 43.37 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.667 on 1376 degrees of freedom
## Multiple R-squared: 0.5775, Adjusted R-squared: 0.5772
## F-statistic: 1881 on 1 and 1376 DF, p-value: < 2.2e-16
Thus the estimated equations is:
yhat = 5.15 + 0.35 * x
or
Predicted hf_score = 5.15 + 0.35 pf_expression_control
Basically, this regression means that at a value of 0 for pf_expression_control, the hf_score would be 5.15, This is not due to a lack of political pressure on media content “lifting” human freedom to a score of 5 but to the presence of the effect of other elements which this model does not capture. The model explains a little more than half of the variability of hf_score (the R-squared is 0.5775).
In terms of the slope, it means that, starting at a value of 0 for pf_expression_control, each unit increase would lift the hf_score by 0.35
Thus, at a value of 1 for pf_expression_control, the value for hf_score would be predicted to be 5.5 (5.15 + 0.35)
At a value of 2 for pf_expression_control, the value for hf_score would be predicted to be 5.85 (5.5 + 0.35)
And at a value of 10 for pf_expression_control, the value for hf_score would be predicted to be 8.65 (5.15 + 0.35 * 10).
Following the example in the instructions, first m1 is estimated.
m1 <- lm(pf_score ~ pf_expression_control, data = hfi)
The results are:
summary(m1)
##
## Call:
## lm(formula = pf_score ~ pf_expression_control, data = hfi)
##
## Residuals:
## Min 1Q Median 3Q Max
## -3.8467 -0.5704 0.1452 0.6066 3.2060
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 4.61707 0.05745 80.36 <2e-16 ***
## pf_expression_control 0.49143 0.01006 48.85 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.8318 on 1376 degrees of freedom
## Multiple R-squared: 0.6342, Adjusted R-squared: 0.634
## F-statistic: 2386 on 1 and 1376 DF, p-value: < 2.2e-16
ggplot(data = hfi, aes(x = pf_expression_control, y = pf_score)) +
geom_point()+
xlim(0,10) + ylim(0,10) +
stat_smooth(method = "lm", se = FALSE)
## `geom_smooth()` using formula = 'y ~ x'
Using only the prediction line, at a value of 6.7 for pf_expression_control, the value for pf_score wold be: 7.9 (4.61707 + 0.49143 * 6.7)
Looking at the scatter plot and the location of the line, it seems the pf_score predicted value of 7.9 for a value of 6.7 for pf_expression control is higher than most of the observed cases. In other words, there seem to be three cases for which the actual pf_score is higher than 7.9 for a value of 6.7 for pf_expression control while there seem to be about ten or twelve cases on which the pf_score is below 7.9. It seems the line overestimates the impact of pf_expression control on pf_score.
Following the exercise instructions, a scatter plot of residuals is generated.
ggplot(data = m1, aes(x = .fitted, y = .resid)) +
geom_point() +
geom_hline(yintercept = 0, linetype = "dashed") +
xlab("Fitted values") +
ylab("Residuals")
## Warning: `fortify(<lm>)` was deprecated in ggplot2 4.0.0.
## ℹ Please use `broom::augment(<lm>)` instead.
## ℹ The deprecated feature was likely used in the ggplot2 package.
## Please report the issue at <https://github.com/tidyverse/ggplot2/issues>.
## This warning is displayed once per session.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
## generated.
For fitted values between 6 and 8 the residuals seem to be evenly and densely distributed around 0. They range between +/+ 2.
However, this is not the case for fitted values below 6. These residuals are more dispersed.
Moreover, after around 8.5 the residuals seem to follow a different pattern. First of all, most of them are above 0. Also, their variability is considerably lower. Other than the country with a high pf_expression_control and relatively low pf_score (mentioned in Exercise 2), the residuals are less than 1 in absolute value and they are even less than 0.5 at the highest values for pf_expression_control.
Given these elements, the linearity assumption is suspect.
Following the instructions, a histogram of residuals and a normal probability plot of residuals are constructed.
ggplot(data = m1, aes(x = .resid)) +
geom_histogram(binwidth = 25) +
xlab("Residuals")
ggplot(data = m1, aes(sample = .resid)) +
stat_qq()
I have also constructed a histogram with a much small bin (I have also tried with a width of 5, which looks similar to the one below but with only one rectangle in the middle, as well as with 10 and 15 which look like the one in the exercise instructions).
ggplot(data = m1, aes(x = .resid)) +
geom_histogram(binwidth = 2) +
xlab("Residuals")
The normal probability plot seems alright in the middle. However, at both ends, it seems to deviate from the theoretical distribution. This would be consistent with the results in the previous exercise.
The histogram with a bin width of 25 (or even 10) does not look like a normal distribution. It looks like a uniform distribution. Using much smaller bin widths the distribution does not look uniform, but it does not look normal either (e.g. it is not symmetrical).
As discussed in exercise 7, as the fitted values increase, the residuals decline (after a fitted value of around 6) and then increase (after a fitted value of about 8.5). The fitted values do not have a constant variability.
For this exercise, first, I have explored pf_expression_newspapers. This variable measures freedom to access to foreign newspapers, according to the information in the help console (called with “?hfi”, which was in Exercise 1).
summary(hfi$pf_expression_newspapers)
## Min. 1st Qu. Median Mean 3rd Qu. Max. NAs
## 0.000 7.500 10.000 8.588 10.000 10.000 255
Interestingly, this variable is quite lopsided, with a median at the highest possible value (10). This shape can easily be seen in a simple histogram.
ggplot(data = hfi, aes(x = pf_expression_newspapers)) +
geom_histogram(binwidth = 2)
## Warning: Removed 255 rows containing non-finite outside the scale range
## (`stat_bin()`).
Then, I explored pf_expression_internet, which measures state control over internet access. These two variables could potentially be associated as the internet is an ideal medium to access foreign newspapers.
summary(hfi$pf_expression_internet)
## Min. 1st Qu. Median Mean 3rd Qu. Max. NAs
## 0.000 7.500 10.000 8.556 10.000 10.000 249
Interestingly, this variable is also lopsided. The median at the highest possible value (10) too. This shape can easily be seen in a simple histogram. It also has more NAs than pf_expression_newspapers (329 against 255).
Then, I put both variables together in a scatter plot.
ggplot(data = hfi, aes(x = pf_expression_internet)) +
geom_histogram(binwidth = 2)
## Warning: Removed 249 rows containing non-finite outside the scale range
## (`stat_bin()`).
ggplot(data = hfi, aes(x = pf_expression_internet, y = pf_expression_newspapers)) +
geom_point() +
stat_smooth(method = "lm", se = FALSE)
## `geom_smooth()` using formula = 'y ~ x'
## Warning: Removed 255 rows containing non-finite outside the scale range
## (`stat_smooth()`).
## Warning: Removed 255 rows containing missing values or values outside the scale range
## (`geom_point()`).
This scatter plot does not seem to portray a linear relationship.
Nevertheless, I did try to estimate the regression.
newregr <- lm(pf_expression_newspapers ~ pf_expression_internet, data = hfi)
The results are:
summary(newregr)
##
## Call:
## lm(formula = pf_expression_newspapers ~ pf_expression_internet,
## data = hfi)
##
## Residuals:
## Min 1Q Median 3Q Max
## -7.9366 -0.4366 0.5301 0.5301 3.5968
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 3.33648 0.17525 19.04 <2e-16 ***
## pf_expression_internet 0.61334 0.01973 31.09 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 1.564 on 1121 degrees of freedom
## (255 observations deleted due to missingness)
## Multiple R-squared: 0.463, Adjusted R-squared: 0.4625
## F-statistic: 966.5 on 1 and 1121 DF, p-value: < 2.2e-16
This is not a very good fit, as expected. The R-squared is 0.46, meaning more than half of the variability of the response variable is NOT explained by the model.
So I have tried with a completely different freedom: ECONOMIC. I have used the economic freedom score (ef_score).
summary(hfi$ef_score)
## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 2.880 6.250 6.900 6.786 7.410 9.190
These data do not seem to be as lopsided as the previous variables I was exploring. Also, there are less NAs (only 80).
ggplot(data = hfi, aes(x = ef_score)) +
geom_histogram(binwidth = 2)
Then, I tried the scatter plot.
ggplot(data = hfi, aes(x = pf_expression_internet, y = ef_score)) +
geom_point() +
stat_smooth(method = "lm", se = FALSE)
## `geom_smooth()` using formula = 'y ~ x'
## Warning: Removed 249 rows containing non-finite outside the scale range
## (`stat_smooth()`).
## Warning: Removed 249 rows containing missing values or values outside the scale range
## (`geom_point()`).
Also, I run the regression.
econregr <- lm(ef_score ~ pf_expression_internet, data = hfi)
The results are:
summary(econregr)
##
## Call:
## lm(formula = ef_score ~ pf_expression_internet, data = hfi)
##
## Residuals:
## Min 1Q Median 3Q Max
## -3.6203 -0.5498 0.2006 0.6102 2.5727
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 6.04128 0.09985 60.501 < 2e-16 ***
## pf_expression_internet 0.09181 0.01125 8.161 8.82e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.8925 on 1127 degrees of freedom
## (249 observations deleted due to missingness)
## Multiple R-squared: 0.0558, Adjusted R-squared: 0.05496
## F-statistic: 66.6 on 1 and 1127 DF, p-value: 8.82e-16
The linear regression is very weak. It only explains less than 6 percent of the variability of the economic freedom score.
Then, I explored the relationship between economic and personal freedom. First, with a scatter plot.
ggplot(data = hfi, aes(x = ef_score, y = pf_score)) +
geom_point() +
xlim(0,10) + ylim(0, 10)+
stat_smooth(method = "lm", se = FALSE)
## `geom_smooth()` using formula = 'y ~ x'
Then, with a regression. It should be mentioned that a case could be made to try to explain economic freedom with personal. As the linear regression does not imply causality both would work and result in the same R-squared (I actually did try it, the results are not shown for simplicity and flow of exposition).
scoresregr2 <- lm(pf_score ~ ef_score, data = hfi)
The results are:
summary(scoresregr2)
##
## Call:
## lm(formula = pf_score ~ ef_score, data = hfi)
##
## Residuals:
## Min 1Q Median 3Q Max
## -4.6373 -0.7334 0.1090 0.8353 2.9840
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 0.51812 0.22223 2.331 0.0199 *
## ef_score 0.98490 0.03248 30.326 <2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 1.065 on 1376 degrees of freedom
## Multiple R-squared: 0.4006, Adjusted R-squared: 0.4002
## F-statistic: 919.7 on 1 and 1376 DF, p-value: < 2.2e-16
This regression is stronger than the previous ones in the exercise. The R-squared is at 0.4.
A scatter plot of the residuals is constructed.
ggplot(data = scoresregr2, aes(x = .fitted, y = .resid)) +
geom_point() +
geom_hline(yintercept = 0, linetype = "dashed") +
xlab("Fitted values") +
ylab("Residuals")
There does seem to be a pattern of positive residuals at low fitted values and negative residuals at high fitted values.
Then, a histogram is constructed.
ggplot(data = scoresregr2, aes(x = .resid)) +
geom_histogram(binwidth = 5) +
xlab("Residuals")
The histogram does not look like a normal distribution, in spite of a small bin (see the effect of small bins on histograms in exercise 8).
I have also estimated the normal probability plot.
ggplot(data = scoresregr2, aes(sample = .resid)) +
stat_qq()
The normal probability plot seems roughly alright in the middle. However, at both ends, it seems to deviate from the expected pattern.
In this case, I will only concentrate on the comparison between the relationship between pf_expression_control and pf_score and the relationship between ef_score and pf_score.
While the R-squared in the first regression is 0.63, it is only 0.4 in the second one.
Thus, it seems pf_expression_control is a better predictor of the pf_score than the ef_score.
Although I would have expected the score of economic freedom to be a good predictor of personal freedom, it does not seem to be strong. On the other hand, as pf_expression_control is a component of the personal freedom score, they seem to be more strongly associated than ef_score and pf_score are.
As mentioned in the previous exercise (where the model diagnostics are shown), I was expecting a strong association between economic and personal freedom, I was also expecting a strong relationship between control of the internet and control of access to foreign newspapers. The weak regression results in these two cases surprised me.