Linespots Runtime Empirical Analysis

1 Setup

1.1 Packages and Functions

The code below simply cleans your environment to avoid loading unnecessary functions or variables and loads the libraries used in our script. We begin by installing and loading the required packages. For BDA, we use mainly Bürkner (2020), whereas Gabry and Mahr (2020) provides support with various plots and functions to calculate credible intervals.

rm( list = ls() )  # Cleans the environment.
# You can install packages in case they are not installed already.
# In case you don't have rstan installed, see:
# https://github.com/stan-dev/rstan/wiki/RStan-Getting-Started
# install.packages( c(("skimr", "ggdag", "dagitty", "loo", "tidyverse", "patchwork",
#                      "brms", "bayesplot", "ggthemes")
library(tidyverse) # For transforming and visualizing data.
library(skimr)     # For getting data summaries
library(ggthemes)  # Themes for ggplot
ggplot2::theme_set(theme_tufte())
library(patchwork) # Combining many plots into the same figure.

library(ggdag)     # For DAG analysis
library(dagitty)   # For DAG analysis

library(brms)      # BDA packages. Alternatively, one can use rethinking & rstanarm.
library(loo)       # For comparing different models' performance
library(bayesplot) # Plotting BDA output by Gabry et al.
bayesplot::color_scheme_set("viridis") #Uses the viridis palette on bayesplots
library(projpred)

For reproducibility and efficiency we set an arbitrary seed, sampling parameters and the number of cores to speed up the MCMC sampling.

SAMPLES = 5000
WARMUP = 1000
CHAINS = 4
SEED = 2020
DELTA = 0.999
TREE = 13
set.seed(SEED)
options(mc.cores = parallel::detectCores())

---

2 Overview of the Dataset

First we load the data and take a look at it. As this report is analyzing the performance differences between Bugspots and Linespots, we focus only on those variables relevant to runtime.

d = read_delim(
  '../data/full_evaluation.csv',
  delim = ",",
  locale = locale(decimal_mark = "."),
  col_names = TRUE,
  col_types = cols(
    AUCEC1 = col_double(),
    AUCEC100 = col_double(),
    AUCEC20 = col_double(),
    AUCEC5 = col_double(),
    AUROC = col_double(),
    Algorithm = col_factor(),
    Depth = col_double(),
    EInspect10 = col_double(),
    EInspect100 = col_double(),
    EInspect200 = col_double(),
    EInspect50 = col_double(),
    EInspectF = col_double(),
    EXAM = col_double(),
    FixCount = col_double(),
    Future = col_double(),
    LOC = col_double(),
    Origin = col_double(),
    Project = col_factor(),
    Runtime = col_double(),
    Time = col_factor(),
    Weight = col_factor(),
    comit_version = col_factor(),
    commits = col_double(),
    language = col_factor(),
    url = col_factor()
  )
)
d = data.frame("Algorithm" =  d$Algorithm, "LOC" = d$LOC, "FixCount" = d$FixCount,
               "Project" = d$Project, "language" = d$language, "AUCEC1" = d$AUCEC1,
               "AUCEC5" = d$AUCEC5, "AUROC" = d$AUROC, "EInspect10" = d$EInspect10,
               "EInspect100" = d$EInspect100, "EInspectF" = d$EInspectF, "EXAM" = d$EXAM)

Descriptive Statistics

Table 2.1: Data summary

Name	d
Number of rows	486
Number of columns	12
_______________________
Column type frequency:
factor	3
numeric	9
________________________
Group variables	None

Variable type: factor

skim_variable	n_missing	complete_rate	ordered	n_unique	top_counts
Algorithm	0	1	FALSE	2	Lin: 243, Bug: 243
Project	0	1	FALSE	32	mys: 18, woo: 18, pre: 18, ser: 18
language	0	1	FALSE	8	Jav: 126, Jav: 72, Typ: 66, C: 54

Variable type: numeric

skim_variable	n_missing	complete_rate	mean	sd	p0	p25	p50	p75	p100	hist
LOC	0	1	1658395.80	3014179.45	11741	245703.00	619122.00	1554904.00	20705587.00	▇▁▁▁▁
FixCount	0	1	90.66	72.62	0	39.25	68.00	113.75	418.00	▇▃▁▁▁
AUCEC1	0	1	0.08	0.06	0	0.04	0.07	0.11	0.35	▇▆▂▁▁
AUCEC5	0	1	0.20	0.11	0	0.12	0.19	0.26	0.56	▅▇▅▂▁
AUROC	0	1	0.02	0.03	0	0.00	0.00	0.02	0.16	▇▁▁▁▁
EInspect10	0	1	0.10	0.34	0	0.00	0.00	0.00	2.00	▇▁▁▁▁
EInspect100	0	1	0.66	1.29	0	0.00	0.00	1.00	8.00	▇▁▁▁▁
EInspectF	0	1	0.00	0.01	0	0.00	0.00	0.00	0.08	▇▁▁▁▁
EXAM	0	1	0.22	0.06	0	0.18	0.22	0.26	0.38	▁▃▇▇▂

There seem to be some cases where no faults were found in the pseudo future. As those cases can’t tell us anything, we will remove them.

d = subset(d, d$FixCount != 0)

Distributions (Histograms)

Scaled Distributions (Histograms)

With both LOC and FixCount spanning multiple orders of magnitude, we standardize them to improve sampling performance.

d$LOC = scale(d$LOC)
d$FixCount = scale(d$FixCount)

---

3 DAG Analysis

Based on the the data we gathered, we built a DAG, representing the causal relationships as we assume them. In this case, the analysis is rather simple. We designed the experiment in such a way, that there is no incoming causal relationship to algorithm so we could use all parameters without confounding problems.

DAG

Runtime_dag <- dagify(
  Project ~ Language,
  LOC ~ Project,
  FixCount ~ Project,
  EvaluationMetrics ~ Algorithm + Project + LOC + FixCount + Language,
  exposure = "Algorithm",
  outcome = "EvaluationMetrics",
  labels = c(
    "Project" = "Project",
    "Language" = "Language",
    "LOC" = "LOC",
    "FixCount" = "Fix\nCount",
    "Algorithm" = "Algorithm",
    "EvaluationMetrics" = "Evaluation\nMetrics"
  )
)

ggdag(Runtime_dag, text = FALSE, use_labels = "label", layout="circle") +
  theme_dag()

***

Causal Paths

The graph shows that there is only a single possible causal path from ‘Algorithm’ to ‘Evaluation Metrics’, so regardless of which other predictor we add to a model, they will not add bias or confounding.

ggdag_paths(Runtime_dag,
            text = FALSE,
            use_labels = "label",
            shadow = TRUE,adjust_for = c("LOC", "Project", "Language", "FixCount"),
            layout="circle") +
  theme_dag()

---

Adjustment Sets

Finally, we can test the three sets of predictors we plan on using for being adjustment sets.

isAdjustmentSet(Runtime_dag, c("LOC"))
isAdjustmentSet(Runtime_dag, c("LOC", "FixCount"))
isAdjustmentSet(Runtime_dag, c("LOC", "FixCount", "Project"))
isAdjustmentSet(Runtime_dag, c("LOC", "FixCount", "Project", "Language"))

## [1] TRUE
## [1] TRUE
## [1] TRUE
## [1] TRUE

---

4 BDA workflow

We are interested in the evaluation metics as our outcome, Algorithm as our exposure and control for LOC, Project and language. We build generalized linear models (GLM) for different predictor combinations and compare different likelihoods.

When applicable, we use projpred for feature selection. The current implementation does not support all likelihoods and varying intercepts so we only use it for the direct effects and as a rough guideline here.

4.1 AUROC Models

For the area under the ROC curve we use a beta likelihood, as it represents the ratio of the union square that is under the ROC curve.

M4

mauroc4p = brm(
  formula = AUROC ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language),
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mauroc4p) + scale_y_continuous(trans='sqrt')

mauroc4= brm(
  formula = AUROC ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language),
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
summary(mauroc4)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: beta 
##   Links: mu = logit; phi = identity 
## Formula: AUROC ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language) 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~language (Number of levels: 8) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     0.35      0.16     0.11     0.75 1.00     4731     5671
## 
## ~Project (Number of levels: 32) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     0.22      0.08     0.06     0.40 1.00     3494     4616
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -3.10      0.16    -3.42    -2.76 1.00     7337     7873
## AlgorithmBugspots    -0.24      0.09    -0.42    -0.07 1.00    24002    11668
## LOC                  -0.09      0.08    -0.24     0.06 1.00    13917    12278
## FixCount              0.12      0.05     0.02     0.22 1.00    18648    12057
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi     7.09      0.48     6.17     8.03 1.00    19364    12099
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

The comparison shows a slightly better loo performance for M4 and as it samples good enough, we use it as our candidate model.

Results: The posterior predictive check shows that the model struggles to fit the early peak and has a longer tail than the data. It does still follow the overall structure of the data reasonably well.

pp_check(mauroc4) + scale_y_continuous(trans='sqrt')

On the logit scale, the 95% interval for the effect of Bugspots is well negative, although there is still an overlap with 0 further in the tail. The same behavior shows on the outcome scale, where the upper bound for Bugspots’ AUROC is well within Linespots’ 95% interval.

We conclude, that in most cases, Linespots’ produces higher AUROC values than Bugspots.

mcmc_areas(mauroc4, pars = c("b_AlgorithmBugspots"), prob = 0.95)

eff = conditional_effects(mauroc4, effects = c("Algorithm"))
eff$Algorithm

eff

And Diagnostics: All diagnostics look well.

min(neff_ratio(mauroc4))

## [1] 0.2085932

max(rhat(mauroc4))

## [1] 1.001587

plot(mauroc4)

M1

mauroc1p = brm(
  formula = AUROC ~ 1 + Algorithm + LOC,
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.7), class=b),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mauroc1p) + scale_y_continuous(trans='sqrt')

mauroc1= brm(
  formula = AUROC ~ 1 + Algorithm + LOC,
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.7), class=b),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(mauroc1) + scale_y_continuous(trans='sqrt')

summary(mauroc1)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: beta 
##   Links: mu = logit; phi = identity 
## Formula: AUROC ~ 1 + Algorithm + LOC 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -3.05      0.08    -3.20    -2.89 1.00     8134     9480
## AlgorithmBugspots    -0.26      0.09    -0.44    -0.09 1.00     9181     8343
## LOC                  -0.15      0.05    -0.26    -0.05 1.00     9306     8636
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi     6.68      0.47     5.79     7.62 1.00     6957     8770
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

M2

mauroc2p = brm(
  formula = AUROC ~ 1 + Algorithm + LOC + FixCount,
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mauroc2p) + scale_y_continuous(trans='sqrt')

mauroc2= brm(
  formula = AUROC ~ 1 + Algorithm + LOC + FixCount,
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(mauroc2) + scale_y_continuous(trans='sqrt')

summary(mauroc2)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: beta 
##   Links: mu = logit; phi = identity 
## Formula: AUROC ~ 1 + Algorithm + LOC + FixCount 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -3.06      0.08    -3.22    -2.90 1.00    10114    10454
## AlgorithmBugspots    -0.23      0.09    -0.42    -0.05 1.00    11734     9565
## LOC                  -0.15      0.05    -0.26    -0.05 1.00    13308     9694
## FixCount              0.11      0.05     0.02     0.20 1.00    12436     8738
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi     6.71      0.47     5.82     7.66 1.00     8995     9750
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

M3

mauroc3p = brm(
  formula = AUROC ~ 1 + Algorithm + LOC + FixCount + (1 | Project),
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mauroc3p) + scale_y_continuous(trans='sqrt')

mauroc3= brm(
  formula = AUROC ~ 1 + Algorithm + LOC + FixCount + (1 | Project),
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
summary(mauroc3)
pp_check(mauroc3) + scale_y_continuous(trans='sqrt')

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: beta 
##   Links: mu = logit; phi = identity 
## Formula: AUROC ~ 1 + Algorithm + LOC + FixCount + (1 | Project) 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~Project (Number of levels: 32) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     0.31      0.08     0.17     0.48 1.00     4971     6794
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -3.06      0.10    -3.25    -2.86 1.00     9015    10216
## AlgorithmBugspots    -0.24      0.09    -0.42    -0.06 1.00    20280    11707
## LOC                  -0.13      0.07    -0.28     0.01 1.00     9957    10594
## FixCount              0.11      0.05     0.01     0.21 1.00    13513    12171
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi     7.04      0.48     6.10     8.01 1.00    14128    11965
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Comparison

Based on using a 95% z score, M4 performs better than M3 (2*1.7 < 3.7), which would make it the candidate model. However, the sampling behavior of M3 is better and the estimated effects and errors for the Algorithm are within 0.01, which lead us to go with the simpler model M3 as our candidate model.

loo_compare(loo(mauroc1), loo(mauroc2), loo(mauroc3), loo(mauroc4))

##         elpd_diff se_diff
## mauroc4   0.0       0.0  
## mauroc3  -1.6       1.3  
## mauroc2 -19.6       3.1  
## mauroc1 -22.2       3.3

4.2 EXAM Models

For the EXAM score we again use a beta likelihood, as it represents the ratio of LOC one has to inspect to find a fault, averaged across all faults.

M4

mexam4p = brm(
  formula = EXAM ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language),
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mexam4p) + scale_y_continuous(trans='sqrt')

mexam4= brm(
  formula = EXAM ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language),
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
summary(mexam4)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: beta 
##   Links: mu = logit; phi = identity 
## Formula: EXAM ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language) 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~language (Number of levels: 8) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     0.20      0.10     0.07     0.47 1.00     4708     7030
## 
## ~Project (Number of levels: 32) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     0.08      0.04     0.02     0.18 1.00     7612     7961
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -1.16      0.09    -1.35    -0.98 1.00     7043     8648
## AlgorithmBugspots     0.08      0.06    -0.04     0.20 1.00    32957    12169
## LOC                  -0.04      0.04    -0.12     0.04 1.00    18859    12839
## FixCount             -0.01      0.03    -0.08     0.05 1.00    25954    12274
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi    10.24      0.48     9.31    11.19 1.00    28338    11613
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Results:

The posterior predictive check shows, that the model struggles to follow the shape of the data and has bigger tails.

pp_check(mexam4) + scale_y_continuous(trans='sqrt')

The effect of Bugspots on the logit scale, while mostly positive, does overlap with 0 in the 95% interval. This also shows on the outcome scale, where the mean EXAM values for Linespots are lower but the intervals of both overlap widely.

We conclude that the difference in EXAM score between Linespots and Bugspots is small and not consistent, with Linespots having an advantage on average.

mcmc_areas(mexam4, pars = c("b_AlgorithmBugspots"), prob = 0.95)

eff = conditional_effects(mexam4, effects = c("Algorithm"))
eff$Algorithm

eff

And Diagnostics: All diagnostics look good.

min(neff_ratio(mexam4))

## [1] 0.2899017

max(rhat(mexam4))

## [1] 1.000476

plot(mexam4)

M1

mexam1p = brm(
  formula = EXAM ~ 1 + Algorithm + LOC,
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.7), class=b),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mexam1p) + scale_y_continuous(trans='sqrt')

mexam1 = brm(
  formula = EXAM ~ 1 + Algorithm + LOC,
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.7), class=b),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(mexam1) + scale_y_continuous(trans='sqrt')

summary(mexam1)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: beta 
##   Links: mu = logit; phi = identity 
## Formula: EXAM ~ 1 + Algorithm + LOC 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -1.17      0.04    -1.26    -1.09 1.00    13110     9884
## AlgorithmBugspots     0.08      0.06    -0.04     0.20 1.00    13999    10639
## LOC                  -0.03      0.03    -0.09     0.03 1.00    12585     9584
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi    10.01      0.47     9.10    10.94 1.00    13554    11150
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

M2

mexam2p = brm(
  formula = EXAM ~ 1 + Algorithm + LOC + FixCount,
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mexam2p) + scale_y_continuous(trans='sqrt')

mexam2= brm(
  formula = EXAM ~ 1 + Algorithm + LOC + FixCount,
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(mexam2) + scale_y_continuous(trans='sqrt')

summary(mexam2)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: beta 
##   Links: mu = logit; phi = identity 
## Formula: EXAM ~ 1 + Algorithm + LOC + FixCount 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -1.17      0.04    -1.26    -1.09 1.00    14693    11305
## AlgorithmBugspots     0.08      0.06    -0.04     0.20 1.00    15966    11986
## LOC                  -0.03      0.03    -0.09     0.03 1.00    17749    11800
## FixCount             -0.01      0.03    -0.07     0.05 1.00    15367    11198
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi    10.01      0.47     9.11    10.93 1.00    15155    10903
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

M3

mexam3p = brm(
  formula = EXAM ~ 1 + Algorithm + LOC + FixCount + (1 | Project),
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mexam3p) + scale_y_continuous(trans='sqrt')

mexam3= brm(
  formula = EXAM ~ 1 + Algorithm + LOC + FixCount + (1 | Project),
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(mexam3) + scale_y_continuous(trans='sqrt')

summary(mexam3)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: beta 
##   Links: mu = logit; phi = identity 
## Formula: EXAM ~ 1 + Algorithm + LOC + FixCount + (1 | Project) 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~Project (Number of levels: 32) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     0.13      0.05     0.04     0.23 1.00     5487     6225
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -1.18      0.05    -1.28    -1.08 1.00    13507    12392
## AlgorithmBugspots     0.08      0.06    -0.04     0.20 1.00    20636    12018
## LOC                  -0.03      0.04    -0.11     0.04 1.00    12940    11347
## FixCount             -0.00      0.03    -0.07     0.06 1.00    18120    13291
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi    10.16      0.48     9.24    11.10 1.00    18410    11693
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Comparison

M4 has the best loo performance and shows good sampling behavior, so we choose it as our candidate model. M3, the next best model has worse performance, based on a 95% z score (2* 0.7 < 3.7).

loo_compare(loo(mexam1), loo(mexam2), loo(mexam3),
 loo(mexam4))

##        elpd_diff se_diff
## mexam4   0.0       0.0  
## mexam3  -4.1       0.7  
## mexam1 -12.3       1.6  
## mexam2 -13.0       1.6

4.3 EInspectF Models

For the EInspectF we use a beta likelihood, as it represents the ratio of LOC one has to inspect to find the first fault.

M2

mEInspectF2p = brm(
  formula = EInspectF ~ 1 + Algorithm + LOC + FixCount,
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mEInspectF2p) + scale_y_continuous(trans='sqrt')

mEInspectF2 = brm(
  formula = EInspectF ~ 1 + Algorithm + LOC +  FixCount,
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
summary(mEInspectF2)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: beta 
##   Links: mu = logit; phi = identity 
## Formula: EInspectF ~ 1 + Algorithm + LOC + FixCount 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -3.68      0.09    -3.86    -3.50 1.00     8653     9882
## AlgorithmBugspots     0.22      0.09     0.05     0.40 1.00    11751    10179
## LOC                  -0.09      0.05    -0.18    -0.00 1.00    12123    10171
## FixCount             -0.15      0.05    -0.25    -0.06 1.00    11632     9365
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi     6.59      0.50     5.61     7.59 1.00     7787     8674
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

The comparison showed equal performance for M2, M3 and M4, so we choose M2 as the candidate model due to it being the smallest one.

Results: The posterior predictive check shows that the model has a bigger tail than the data but manages to reasonably fit the general shape.

pp_check(mEInspectF2) + scale_y_continuous(trans='sqrt')

The 95% interval of the effect of Bugspots is completely positive, however there is an overlap with 0 further in the tail. On the outcome scale, the overlap for EXAMF scores is big between the two algorithms with a small advantage for Linespots.

We conclude, that the difference in EXAMF score between Bugspots and Linespots is not very consistent but on average Linespots performs better.

mcmc_areas(mEInspectF2, pars = c("b_AlgorithmBugspots"), prob = 0.95)

eff = conditional_effects(mEInspectF2, effects = c("Algorithm"))
eff$Algorithm

eff

And Diagnostics: All diagnostics look good.

min(neff_ratio(mEInspectF2))

## [1] 0.4259381

max(rhat(mEInspectF2))

## [1] 1.000325

plot(mEInspectF2)

M1

mEInspectF1p = brm(
  formula = EInspectF ~ 1 + Algorithm + LOC,
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.7), class=b),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mEInspectF1p) + scale_y_continuous(trans='sqrt')

mEInspectF1 = brm(
  formula = EInspectF ~ 1 + Algorithm + LOC,
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.7), class=b),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(mEInspectF1) + scale_y_continuous(trans='sqrt')

summary(mEInspectF1)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: beta 
##   Links: mu = logit; phi = identity 
## Formula: EInspectF ~ 1 + Algorithm + LOC 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -3.68      0.09    -3.86    -3.50 1.00     7627     8525
## AlgorithmBugspots     0.23      0.09     0.05     0.40 1.00    10084     8904
## LOC                  -0.10      0.05    -0.20    -0.01 1.00    10060     7862
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi     6.52      0.50     5.56     7.52 1.00     7127     8854
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

M3

mEInspectF3p = brm(
  formula = EInspectF ~ 1 + Algorithm + LOC + FixCount + (1 | Project),
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mEInspectF3p) + scale_y_continuous(trans='sqrt')

mEInspectF3= brm(
  formula = EInspectF ~ 1 + Algorithm + LOC + FixCount + (1 | Project),
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(mEInspectF3) + scale_y_continuous(trans='sqrt')

summary(mEInspectF3)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: beta 
##   Links: mu = logit; phi = identity 
## Formula: EInspectF ~ 1 + Algorithm + LOC + FixCount + (1 | Project) 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~Project (Number of levels: 32) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     0.08      0.04     0.01     0.18 1.00    11750     9038
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -3.68      0.09    -3.86    -3.49 1.00    14667    10671
## AlgorithmBugspots     0.22      0.09     0.05     0.40 1.00    23584    11623
## LOC                  -0.09      0.05    -0.19     0.00 1.00    22466    11760
## FixCount             -0.15      0.05    -0.25    -0.05 1.00    23827    11821
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi     6.60      0.51     5.62     7.60 1.00    13844    11192
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

M4

mEInspectF4p = brm(
  formula = EInspectF ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language),
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mEInspectF4p) + scale_y_continuous(trans='sqrt')

mEInspectF4= brm(
  formula = EInspectF ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language),
  data = d,
  family = Beta(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(mEInspectF4) + scale_y_continuous(trans='sqrt')

summary(mEInspectF4)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: beta 
##   Links: mu = logit; phi = identity 
## Formula: EInspectF ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language) 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~language (Number of levels: 8) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     0.11      0.08     0.02     0.31 1.00     7591    10129
## 
## ~Project (Number of levels: 32) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     0.09      0.05     0.01     0.19 1.00    10738     8839
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -3.68      0.11    -3.88    -3.47 1.00    11414    11435
## AlgorithmBugspots     0.22      0.09     0.05     0.40 1.00    25055    11927
## LOC                  -0.10      0.06    -0.22     0.01 1.00    16973    12074
## FixCount             -0.15      0.05    -0.25    -0.05 1.00    23598    12250
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi     6.63      0.51     5.67     7.66 1.00    14923    11376
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Comparison

The loo performance for M2, M3 and M4 is the same within the given errors, which is why we choose M2 as our candidate model. It is the smallest model of the three and shows good sampling behavior.

loo_compare(loo(mEInspectF1), loo(mEInspectF2), loo(mEInspectF3), loo(mEInspectF4))

##             elpd_diff se_diff
## mEInspectF2  0.0       0.0   
## mEInspectF3 -0.5       0.1   
## mEInspectF4 -0.8       0.2   
## mEInspectF1 -6.0       1.0

4.4 EInspect10 Models

For the EInspect10 we use a negative binomial likelihood, as it is a count outcome. We also considered a poisson, but the difference in mean and variance makes the negative binomial the better candidate.

M3

mEInspect103p = brm(
  formula = EInspect10 ~ 1 + Algorithm + LOC + FixCount + (1 | Project),
  data = d,
  family = negbinomial(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.7), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=shape)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mEInspect103p) + scale_x_continuous(trans='log10')

mEInspect103 = brm(
  formula = EInspect10 ~ 1 + Algorithm + LOC + FixCount + (1 | Project),
  data = d,
  family = negbinomial(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.7), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=shape)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
summary(mEInspect103)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: negbinomial 
##   Links: mu = log; shape = identity 
## Formula: EInspect10 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~Project (Number of levels: 32) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     1.04      0.30     0.51     1.69 1.00     5353     7288
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -2.42      0.31    -3.08    -1.87 1.00     9763     9859
## AlgorithmBugspots    -0.85      0.30    -1.45    -0.27 1.00    21709    11662
## LOC                  -0.40      0.33    -1.08     0.19 1.00    14044    11581
## FixCount              0.18      0.15    -0.12     0.47 1.00    16113    12570
## 
## Family Specific Parameters: 
##       Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## shape     1.31      0.43     0.60     2.26 1.00    20809    11129
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

The comparison shows very similar LOO performance for M3 and M4. We chose M3 as our candidate model, as it is the smaller one.

Results:

pp_check(mEInspect103) + scale_x_continuous(trans='log10')

mcmc_areas(mEInspect103, pars = c("b_AlgorithmBugspots"), prob = 0.95)

eff = conditional_effects(mEInspect103, effects = c("Algorithm"))
eff$Algorithm

eff

And Diagnostics: All diagnostics look good.

min(neff_ratio(mEInspect103))

## [1] 0.2394607

max(rhat(mEInspect103))

## [1] 1.001278

plot(mEInspect103)

M1

mEInspect101p = brm(
  formula = EInspect10 ~ 1 + Algorithm + LOC,
  data = d,
  family = negbinomial(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.7), class=b),
    prior(weibull(2, 1), class=shape)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mEInspect101p) + scale_x_continuous(trans='log10')

mEInspect101 = brm(
  formula = EInspect10 ~ 1 + Algorithm + LOC,
  data = d,
  family = negbinomial(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.7), class=b),
    prior(weibull(2, 1), class=shape)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(mEInspect101) + scale_x_continuous(trans='log10')

summary(mEInspect101)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: negbinomial 
##   Links: mu = log; shape = identity 
## Formula: EInspect10 ~ 1 + Algorithm + LOC 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -2.00      0.19    -2.38    -1.65 1.00    13962    10160
## AlgorithmBugspots    -0.87      0.29    -1.47    -0.30 1.00    11365    10155
## LOC                  -0.47      0.26    -1.04    -0.02 1.00    11899     8734
## 
## Family Specific Parameters: 
##       Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## shape     1.07      0.41     0.44     1.99 1.00    13798    10514
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

M2

mEInspect102p = brm(
  formula = EInspect10 ~ 1 + Algorithm + LOC + FixCount,
  data = d,
  family = negbinomial(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=shape)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mEInspect102p) + scale_x_continuous(trans='log10')

mEInspect102 = brm(
  formula = EInspect10 ~ 1 + Algorithm + LOC + FixCount,
  data = d,
  family = negbinomial(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=shape)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(mEInspect102) + scale_x_continuous(trans='log10')

summary(mEInspect102)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: negbinomial 
##   Links: mu = log; shape = identity 
## Formula: EInspect10 ~ 1 + Algorithm + LOC + FixCount 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -2.08      0.18    -2.46    -1.74 1.00    14977    11029
## AlgorithmBugspots    -0.72      0.27    -1.26    -0.20 1.00    12091    11343
## LOC                  -0.44      0.24    -0.96    -0.02 1.00    12914    10049
## FixCount              0.24      0.12    -0.01     0.47 1.00    13810    11800
## 
## Family Specific Parameters: 
##       Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## shape     1.11      0.42     0.45     2.05 1.00    12895    10016
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

M4

mEInspect104p = brm(
  formula = EInspect10 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language),
  data = d,
  family = negbinomial(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=shape)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mEInspect104p) + scale_x_continuous(trans='log10')

mEInspect104= brm(
  formula = EInspect10 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language),
  data = d,
  family = negbinomial(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=shape)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(mEInspect104) + scale_x_continuous(trans='log10')

summary(mEInspect104)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: negbinomial 
##   Links: mu = log; shape = identity 
## Formula: EInspect10 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language) 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~language (Number of levels: 8) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     0.94      0.37     0.31     1.74 1.00     7092     6320
## 
## ~Project (Number of levels: 32) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     0.76      0.32     0.19     1.44 1.00     5788     7690
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -2.50      0.43    -3.38    -1.69 1.00    10759    11978
## AlgorithmBugspots    -0.72      0.27    -1.27    -0.19 1.00    22814    11770
## LOC                  -0.20      0.29    -0.81     0.33 1.00    15503    11258
## FixCount              0.14      0.15    -0.14     0.42 1.00    21490    12659
## 
## Family Specific Parameters: 
##       Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## shape     1.31      0.42     0.61     2.25 1.00    24525    11446
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Comparison

loo_compare(loo(mEInspect101), loo(mEInspect102), loo(mEInspect103), loo(mEInspect104))

##              elpd_diff se_diff
## mEInspect104   0.0       0.0  
## mEInspect103  -1.2       1.4  
## mEInspect102 -12.2       3.2  
## mEInspect101 -13.2       3.8

4.5 EInspect100 Models

For the EInspect100 we use a negative binomial likelihood, as it is a count outcome. We also considered a poisson, but the difference in mean and variance makes the negative binomial the better candidate.

M3

mEInspect1003p = brm(
  formula = EInspect100 ~ 1 + Algorithm + LOC + FixCount + (1 | Project),
  data = d,
  family = negbinomial(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=shape)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mEInspect1003p) + scale_x_continuous(trans='log10')

mEInspect1003 = brm(
  formula = EInspect100 ~ 1 + Algorithm + LOC + FixCount + (1 | Project),
  data = d,
  family = negbinomial(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=shape)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
summary(mEInspect1003)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: negbinomial 
##   Links: mu = log; shape = identity 
## Formula: EInspect100 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~Project (Number of levels: 32) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     1.02      0.20     0.67     1.45 1.00     3887     7257
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -0.71      0.22    -1.17    -0.30 1.00     3869     5792
## AlgorithmBugspots    -0.58      0.15    -0.88    -0.29 1.00    16945    12350
## LOC                  -0.33      0.19    -0.72     0.04 1.00     7902     9904
## FixCount              0.26      0.08     0.10     0.42 1.00    13429    13020
## 
## Family Specific Parameters: 
##       Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## shape     1.35      0.27     0.91     1.96 1.00    16607    12225
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

The comparison shows very similar LOO performance for M2, M3, M5 and M6. We chose M2 as our final model as it shows very good sampling behavior and because it is the smaller model of the two.

Results:

pp_check(mEInspect1003) + scale_x_continuous(trans='log10')

mcmc_areas(mEInspect1003, pars = c("b_AlgorithmBugspots"), prob = 0.95)

eff = conditional_effects(mEInspect1003, effects = c("Algorithm"))
eff$Algorithm

eff

And Diagnostics: All diagnostics look good.

min(neff_ratio(mEInspect1003))

## [1] 0.2042747

max(rhat(mEInspect1003))

## [1] 1.001435

plot(mEInspect1003)

M1

mEInspect1001p = brm(
  formula = EInspect100 ~ 1 + Algorithm + LOC,
  data = d,
  family = negbinomial(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.7), class=b),
    prior(weibull(2, 1), class=shape)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mEInspect1001p) + scale_x_continuous(trans='log10')

mEInspect1001 = brm(
  formula = EInspect100 ~ 1 + Algorithm + LOC,
  data = d,
  family = negbinomial(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.7), class=b),
    prior(weibull(2, 1), class=shape)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(mEInspect1001) + scale_x_continuous(trans='log10')

summary(mEInspect1001)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: negbinomial 
##   Links: mu = log; shape = identity 
## Formula: EInspect100 ~ 1 + Algorithm + LOC 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -0.29      0.11    -0.51    -0.07 1.00    12492    10385
## AlgorithmBugspots    -0.48      0.17    -0.81    -0.15 1.00    11858     9963
## LOC                  -0.53      0.15    -0.83    -0.25 1.00    11578    10265
## 
## Family Specific Parameters: 
##       Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## shape     0.57      0.09     0.41     0.77 1.00    11826    10022
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

M2

mEInspect1002p = brm(
  formula = EInspect100 ~ 1 + Algorithm + LOC + FixCount,
  data = d,
  family = negbinomial(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=shape)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(mEInspect1002p) + scale_x_continuous(trans='log10')

mEInspect1002 = brm(
  formula = EInspect100 ~ 1 + Algorithm + LOC + FixCount,
  data = d,
  family = negbinomial(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=shape)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(mEInspect1002) + scale_x_continuous(trans='log10')

summary(mEInspect1002)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: negbinomial 
##   Links: mu = log; shape = identity 
## Formula: EInspect100 ~ 1 + Algorithm + LOC + FixCount 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -0.42      0.11    -0.65    -0.20 1.00    13133    10863
## AlgorithmBugspots    -0.44      0.16    -0.74    -0.14 1.00    13123    10633
## LOC                  -0.58      0.15    -0.90    -0.30 1.00    11963    10080
## FixCount              0.45      0.08     0.30     0.61 1.00    13361    10934
## 
## Family Specific Parameters: 
##       Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## shape     0.73      0.13     0.51     1.02 1.00    12957    10337
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

M4

mEInspect1004p = brm(
  formula = EInspect100 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language),
  data = d,
  family = negbinomial(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=shape)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(mEInspect1004p) + scale_x_continuous(trans='log10')

mEInspect1004= brm(
  formula = EInspect100 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language),
  data = d,
  family = negbinomial(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=shape)
  ),
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(mEInspect1004) + scale_x_continuous(trans='log10')

summary(mEInspect1004)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: negbinomial 
##   Links: mu = log; shape = identity 
## Formula: EInspect100 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language) 
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~language (Number of levels: 8) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     0.72      0.29     0.21     1.37 1.00     5262     5707
## 
## ~Project (Number of levels: 32) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     0.83      0.21     0.47     1.29 1.00     5151     7803
## 
## Population-Level Effects: 
##                   Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## Intercept            -0.80      0.33    -1.49    -0.19 1.00     8403     9357
## AlgorithmBugspots    -0.59      0.15    -0.89    -0.29 1.00    29786    11368
## LOC                  -0.20      0.20    -0.61     0.17 1.00    13662    11419
## FixCount              0.26      0.08     0.10     0.42 1.00    17425    13039
## 
## Family Specific Parameters: 
##       Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## shape     1.34      0.27     0.90     1.95 1.00    23167    11630
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Comparison

loo_compare(loo(mEInspect1001), loo(mEInspect1002), loo(mEInspect1003), loo(mEInspect1004))

##               elpd_diff se_diff
## mEInspect1003   0.0       0.0  
## mEInspect1004  -0.6       1.0  
## mEInspect1002 -42.4       7.7  
## mEInspect1001 -59.2       9.6

4.6 AUCEC1 Models

For the area under the cost-effectiveness curve at 0.01 LOC we use a zero-inflated beta likelihood. The AUCEC values are all normalized to their respective optimal value which puts them between 0 and 1. However, the low cut-off leads to some 0 results, which we need the zero-inflation for.

M4

maucec14p = brm(
  bf(AUCEC1 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language), zi ~ Algorithm + (1 | Project)),
  data = d,
  family = zero_inflated_beta(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(normal(0, 2), class=b, dpar="zi"),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)

  ),
  init = 0,
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(maucec14p) + scale_y_continuous(trans='sqrt')

maucec14 = brm(
  bf(AUCEC1 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language), zi ~ Algorithm + (1 | Project)),
  data = d,
  family = zero_inflated_beta(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(normal(0, 2), class=b, dpar="zi"),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)

  ),
  init = 0,
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
summary(maucec14)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: zero_inflated_beta 
##   Links: mu = logit; phi = identity; zi = logit 
## Formula: AUCEC1 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language) 
##          zi ~ Algorithm + (1 | Project)
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~language (Number of levels: 8) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     0.24      0.13     0.06     0.56 1.00     4567     6411
## 
## ~Project (Number of levels: 32) 
##                  Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)        0.17      0.07     0.04     0.31 1.00     5072     7063
## sd(zi_Intercept)     2.27      0.61     1.33     3.71 1.00     5690     8728
## 
## Population-Level Effects: 
##                      Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## Intercept               -1.89      0.12    -2.13    -1.66 1.00     8351
## zi_Intercept            -5.39      0.80    -7.11    -3.98 1.00     9038
## AlgorithmBugspots       -0.37      0.08    -0.52    -0.21 1.00    24335
## LOC                      0.15      0.06     0.04     0.27 1.00    11458
## FixCount                -0.03      0.04    -0.12     0.06 1.00    19860
## zi_AlgorithmBugspots     2.06      0.53     1.09     3.15 1.00    21285
##                      Tail_ESS
## Intercept                8606
## zi_Intercept             8774
## AlgorithmBugspots       11536
## LOC                     11572
## FixCount                13106
## zi_AlgorithmBugspots    11112
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi     8.76      0.47     7.85     9.70 1.00    21385    11217
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Results:

pp_check(maucec14) + scale_y_continuous(trans='sqrt')

mcmc_areas(maucec14, pars = c("b_AlgorithmBugspots"), prob = 0.95)

eff = conditional_effects(maucec14, effects = c("Algorithm"))
eff$Algorithm

eff

And Diagnostics: All diagnostics look good.

min(neff_ratio(maucec14))

## [1] 0.2317918

max(rhat(maucec14))

## [1] 1.001268

plot(maucec14)

M1

maucec11p = brm(
  bf(AUCEC1 ~ 1 + Algorithm + LOC, zi ~ Algorithm + (1 | Project)),
  data = d,
  family = zero_inflated_beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.7), class=b),
    prior(normal(0, 2), class=b, dpar="zi"),
    prior(weibull(2, 1), class=phi)
  ),
  init = 0,
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(maucec11p) + scale_y_continuous(trans='sqrt')

maucec11 = brm(
  bf(AUCEC1 ~ 1 + Algorithm + LOC, zi ~ Algorithm + (1 | Project)),
  data = d,
  family = zero_inflated_beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.7), class=b),
    prior(normal(0, 2), class=b, dpar="zi"),
    prior(weibull(2, 1), class=phi)

  ),
  init = 0,
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(maucec11) + scale_y_continuous(trans='sqrt')

summary(maucec11)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: zero_inflated_beta 
##   Links: mu = logit; phi = identity; zi = logit 
## Formula: AUCEC1 ~ 1 + Algorithm + LOC 
##          zi ~ Algorithm + (1 | Project)
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~Project (Number of levels: 32) 
##                  Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(zi_Intercept)     2.30      0.63     1.32     3.74 1.00     4841     7746
## 
## Population-Level Effects: 
##                      Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## Intercept               -1.86      0.06    -1.97    -1.75 1.00    24406
## zi_Intercept            -5.39      0.80    -7.13    -4.00 1.00     6696
## AlgorithmBugspots       -0.36      0.08    -0.52    -0.21 1.00    23912
## LOC                      0.08      0.04     0.00     0.15 1.00    24817
## zi_AlgorithmBugspots     2.06      0.53     1.08     3.15 1.00    20389
##                      Tail_ESS
## Intercept               12470
## zi_Intercept             7627
## AlgorithmBugspots       12729
## LOC                     11330
## zi_AlgorithmBugspots    12378
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi     8.47      0.45     7.61     9.37 1.00    19357    12466
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

M2

maucec12p = brm(
  bf(AUCEC1 ~ 1 + Algorithm + LOC + FixCount, zi ~ Algorithm + (1 | Project)),
  data = d,
  family = zero_inflated_beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(normal(0, 2), class=b, dpar="zi"),
    prior(weibull(2, 1), class=phi)

  ),
  init = 0,
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(maucec12p) + scale_y_continuous(trans='sqrt')

maucec12 = brm(
  bf(AUCEC1 ~ 1 + Algorithm + LOC + FixCount + (1 | Project), zi ~ Algorithm + (1 | Project)),
  data = d,
  family = zero_inflated_beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(normal(0, 2), class=b, dpar="zi"),
    prior(weibull(2, 1), class=phi)

  ),
  init = 0,
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
summary(maucec12)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: zero_inflated_beta 
##   Links: mu = logit; phi = identity; zi = logit 
## Formula: AUCEC1 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) 
##          zi ~ Algorithm + (1 | Project)
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~Project (Number of levels: 32) 
##                  Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)        0.18      0.07     0.03     0.32 1.00     3449     2721
## sd(zi_Intercept)     2.28      0.61     1.33     3.72 1.00     5143     9335
## 
## Population-Level Effects: 
##                      Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## Intercept               -1.87      0.07    -2.00    -1.74 1.00    16714
## zi_Intercept            -5.36      0.78    -7.05    -3.97 1.00     8673
## AlgorithmBugspots       -0.36      0.08    -0.52    -0.21 1.00    22724
## LOC                      0.10      0.05     0.00     0.20 1.00    11979
## FixCount                -0.02      0.04    -0.11     0.06 1.00    17828
## zi_AlgorithmBugspots     2.06      0.53     1.07     3.15 1.00    21520
##                      Tail_ESS
## Intercept               13749
## zi_Intercept             8970
## AlgorithmBugspots       12503
## LOC                     12123
## FixCount                11496
## zi_AlgorithmBugspots    11681
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi     8.67      0.47     7.76     9.62 1.00    17283    12835
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

M3

maucec13p = brm(
  bf(AUCEC1 ~ 1 + Algorithm + LOC + FixCount + (1 | Project), zi ~ Algorithm + (1 | Project)),
  data = d,
  family = zero_inflated_beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(normal(0, 2), class=b, dpar="zi"),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)

  ),
  init = 0,
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(maucec13p) + scale_y_continuous(trans='sqrt')

maucec13 = brm(
  bf(AUCEC1 ~ 1 + Algorithm + LOC + FixCount + (1 | Project), zi ~ Algorithm + (1 | Project)),
  data = d,
  family = zero_inflated_beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(normal(0, 2), class=b, dpar="zi"),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)

  ),
  init = 0,
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(maucec13) + scale_y_continuous(trans='sqrt')

summary(maucec13)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: zero_inflated_beta 
##   Links: mu = logit; phi = identity; zi = logit 
## Formula: AUCEC1 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) 
##          zi ~ Algorithm + (1 | Project)
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~Project (Number of levels: 32) 
##                  Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)        0.21      0.07     0.09     0.35 1.00     5005     5939
## sd(zi_Intercept)     2.28      0.61     1.33     3.71 1.00     5111     9010
## 
## Population-Level Effects: 
##                      Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## Intercept               -1.87      0.07    -2.00    -1.73 1.00    13979
## zi_Intercept            -5.39      0.80    -7.09    -4.01 1.00     8815
## AlgorithmBugspots       -0.36      0.08    -0.52    -0.21 1.00    24596
## LOC                      0.11      0.05     0.00     0.21 1.00    12493
## FixCount                -0.03      0.05    -0.11     0.06 1.00    19509
## zi_AlgorithmBugspots     2.06      0.53     1.08     3.14 1.00    22014
##                      Tail_ESS
## Intercept               12717
## zi_Intercept            10051
## AlgorithmBugspots       12163
## LOC                     11938
## FixCount                11540
## zi_AlgorithmBugspots    11092
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi     8.70      0.47     7.80     9.65 1.00    17952    11931
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Comparison

loo_compare(loo(maucec11), loo(maucec12, reloo=TRUE), loo(maucec13, reloo=TRUE), loo(maucec14))

##          elpd_diff se_diff
## maucec14   0.0       0.0  
## maucec13  -1.9       0.8  
## maucec12  -3.5       0.9  
## maucec11 -13.2       1.8

4.7 AUCEC5 Models

For the area under the cost-effectiveness curve at 0.05 LOC we use a zero-inflated beta likelihood. The AUCEC values are all normalized to their respective optimal value which puts them between 0 and 1. However, the low cut-off leads to some 0 results, which we need the zero-inflation for.

M2

maucec52p = brm(
  bf(AUCEC5 ~ 1 + Algorithm + LOC + FixCount, zi ~ Algorithm + (1 | Project)),
  data = d,
  family = zero_inflated_beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(normal(0, 2), class=b, dpar="zi"),
    prior(weibull(2, 1), class=phi)

  ),
  init = 0,
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(maucec52p) + scale_y_continuous(trans='sqrt')

maucec52 = brm(
  bf(AUCEC5 ~ 1 + Algorithm + LOC + FixCount + (1 | Project), zi ~ Algorithm + (1 | Project)),
  data = d,
  family = zero_inflated_beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(normal(0, 2), class=b, dpar="zi"),
    prior(weibull(2, 1), class=phi)

  ),
  init = 0,
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
summary(maucec52)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: zero_inflated_beta 
##   Links: mu = logit; phi = identity; zi = logit 
## Formula: AUCEC5 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) 
##          zi ~ Algorithm + (1 | Project)
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~Project (Number of levels: 32) 
##                  Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)        0.33      0.06     0.22     0.46 1.00     5619     9437
## sd(zi_Intercept)     1.75      0.87     0.22     3.72 1.00     4469     3144
## 
## Population-Level Effects: 
##                      Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## Intercept               -1.09      0.08    -1.24    -0.95 1.00     6659
## zi_Intercept            -7.32      1.48   -10.65    -4.89 1.00     9267
## AlgorithmBugspots       -0.40      0.07    -0.52    -0.27 1.00    28156
## LOC                      0.12      0.06     0.01     0.23 1.00     9740
## FixCount                 0.01      0.04    -0.07     0.08 1.00    15287
## zi_AlgorithmBugspots     1.82      1.09    -0.10     4.12 1.00    21664
##                      Tail_ESS
## Intercept                9988
## zi_Intercept             9564
## AlgorithmBugspots       12246
## LOC                     11787
## FixCount                12277
## zi_AlgorithmBugspots    11507
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi     8.90      0.45     8.04     9.80 1.00    22098    11443
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Results:

pp_check(maucec52) + scale_y_continuous(trans='sqrt')

mcmc_areas(maucec52, pars = c("b_AlgorithmBugspots"), prob = 0.95)

eff = conditional_effects(maucec52, effects = c("Algorithm"))
eff$Algorithm

eff

And Diagnostics: All diagnostics look good.

min(neff_ratio(maucec52))

## [1] 0.2144166

max(rhat(maucec52))

## [1] 1.000896

plot(maucec52)

M1

maucec51p = brm(
  bf(AUCEC5 ~ 1 + Algorithm + LOC, zi ~ Algorithm + (1 | Project)),
  data = d,
  family = zero_inflated_beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(normal(0, 2), class=b, dpar="zi"),
    prior(weibull(2, 1), class=phi)
  ),
  init = 0,
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(maucec51p) + scale_y_continuous(trans='sqrt')

maucec51 = brm(
  bf(AUCEC5 ~ 1 + Algorithm + LOC, zi ~ Algorithm + (1 | Project)),
  data = d,
  family = zero_inflated_beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(normal(0, 2), class=b, dpar="zi"),
    prior(weibull(2, 1), class=phi)

  ),
  init = 0,
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(maucec51) + scale_y_continuous(trans='sqrt')

summary(maucec51)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: zero_inflated_beta 
##   Links: mu = logit; phi = identity; zi = logit 
## Formula: AUCEC5 ~ 1 + Algorithm + LOC 
##          zi ~ Algorithm + (1 | Project)
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~Project (Number of levels: 32) 
##                  Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(zi_Intercept)     1.73      0.85     0.25     3.69 1.00     5332     4317
## 
## Population-Level Effects: 
##                      Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## Intercept               -1.09      0.05    -1.19    -0.99 1.00    23424
## zi_Intercept            -7.29      1.44   -10.52    -4.91 1.00     9517
## AlgorithmBugspots       -0.39      0.07    -0.52    -0.25 1.00    20576
## LOC                      0.06      0.03    -0.01     0.13 1.00    21633
## zi_AlgorithmBugspots     1.82      1.08    -0.11     4.12 1.00    17725
##                      Tail_ESS
## Intercept               12072
## zi_Intercept             9636
## AlgorithmBugspots       11350
## LOC                     10804
## zi_AlgorithmBugspots    10234
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi     7.96      0.41     7.16     8.79 1.00    19670    11617
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

M3

maucec53p = brm(
  bf(AUCEC5 ~ 1 + Algorithm + LOC + FixCount + (1 | Project), zi ~ Algorithm + (1 | Project)),
  data = d,
  family = zero_inflated_beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(normal(0, 2), class=b, dpar="zi"),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)

  ),
  init = 0,
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(maucec53p) + scale_y_continuous(trans='sqrt')

maucec53 = brm(
  bf(AUCEC5 ~ 1 + Algorithm + LOC + FixCount + (1 | Project), zi ~ Algorithm + (1 | Project)),
  data = d,
  family = zero_inflated_beta(),
  prior = c(
    prior(normal(-1, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(normal(0, 2), class=b, dpar="zi"),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)

  ),
  init = 0,
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(maucec53) + scale_y_continuous(trans='sqrt')

summary(maucec53)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: zero_inflated_beta 
##   Links: mu = logit; phi = identity; zi = logit 
## Formula: AUCEC5 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) 
##          zi ~ Algorithm + (1 | Project)
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~Project (Number of levels: 32) 
##                  Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)        0.34      0.06     0.23     0.47 1.00     5160     8310
## sd(zi_Intercept)     1.75      0.86     0.27     3.70 1.00     5093     3591
## 
## Population-Level Effects: 
##                      Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## Intercept               -1.09      0.08    -1.25    -0.94 1.00     5913
## zi_Intercept            -7.32      1.49   -10.66    -4.88 1.00     9487
## AlgorithmBugspots       -0.40      0.07    -0.53    -0.27 1.00    22086
## LOC                      0.12      0.06     0.01     0.24 1.00     8140
## FixCount                 0.00      0.04    -0.07     0.08 1.00    13607
## zi_AlgorithmBugspots     1.84      1.10    -0.13     4.22 1.00    17401
##                      Tail_ESS
## Intercept                9301
## zi_Intercept             9194
## AlgorithmBugspots       12322
## LOC                     10592
## FixCount                12677
## zi_AlgorithmBugspots    10704
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi     8.90      0.45     8.04     9.79 1.00    19231    11749
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

M4

maucec54p = brm(
  bf(AUCEC5 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language), zi ~ Algorithm + (1 | Project)),
  data = d,
  family = zero_inflated_beta(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(normal(0, 2), class=b, dpar="zi"),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)

  ),
  init = 0,
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = "only",
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)

pp_check(maucec54p) + scale_y_continuous(trans='sqrt')

maucec54 = brm(
  bf(AUCEC5 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language), zi ~ Algorithm + (1 | Project)),
  data = d,
  family = zero_inflated_beta(),
  prior = c(
    prior(normal(-2, 1), class=Intercept),
    prior(normal(0, 0.5), class=b),
    prior(normal(0, 2), class=b, dpar="zi"),
    prior(weibull(2, 1), class=sd),
    prior(weibull(2, 1), class=phi)

  ),
  init = 0,
  iter = SAMPLES,
  warmup = WARMUP,
  chains = CHAINS,
  cores = parallel::detectCores(),
  sample_prior = FALSE,
  control = list(adapt_delta = DELTA, max_treedepth = TREE),
  seed = SEED
)
pp_check(maucec54) + scale_y_continuous(trans='sqrt')

summary(maucec54)

## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
## Running /usr/lib64/R/bin/R CMD SHLIB foo.c
## gcc -I"/usr/include/R/" -DNDEBUG   -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/Rcpp/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/unsupported"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/BH/include" -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/src/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/"  -I"/home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/rstan/include" -DEIGEN_NO_DEBUG  -D_REENTRANT  -DBOOST_DISABLE_ASSERTS -DBOOST_PENDING_INTEGER_LOG2_HPP -include stan/math/prim/mat/fun/Eigen.hpp   -D_FORTIFY_SOURCE=2   -fpic  -march=x86-64 -mtune=generic -O2 -pipe -fno-plt  -c foo.c -o foo.o
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:88,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:1: error: unknown type name ‘namespace’
##   613 | namespace Eigen {
##       | ^~~~~~~~~
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/src/Core/util/Macros.h:613:17: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘{’ token
##   613 | namespace Eigen {
##       |                 ^
## In file included from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Dense:1,
##                  from /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:13,
##                  from <command-line>:
## /home/m0hawk/R/x86_64-pc-linux-gnu-library/4.0/RcppEigen/include/Eigen/Core:96:10: fatal error: complex: No such file or directory
##    96 | #include <complex>
##       |          ^~~~~~~~~
## compilation terminated.
## make: *** [/usr/lib64/R/etc/Makeconf:167: foo.o] Error 1
##  Family: zero_inflated_beta 
##   Links: mu = logit; phi = identity; zi = logit 
## Formula: AUCEC5 ~ 1 + Algorithm + LOC + FixCount + (1 | Project) + (1 | language) 
##          zi ~ Algorithm + (1 | Project)
##    Data: d (Number of observations: 480) 
## Samples: 4 chains, each with iter = 5000; warmup = 1000; thin = 1;
##          total post-warmup samples = 16000
## 
## Group-Level Effects: 
## ~language (Number of levels: 8) 
##               Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)     0.33      0.15     0.11     0.71 1.00     4428     5925
## 
## ~Project (Number of levels: 32) 
##                  Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## sd(Intercept)        0.26      0.07     0.15     0.40 1.00     5668     7652
## sd(zi_Intercept)     1.73      0.85     0.24     3.70 1.00     5164     3399
## 
## Population-Level Effects: 
##                      Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS
## Intercept               -1.14      0.14    -1.44    -0.86 1.00     6309
## zi_Intercept            -7.30      1.45   -10.54    -4.91 1.00    10164
## AlgorithmBugspots       -0.40      0.07    -0.53    -0.26 1.00    23203
## LOC                      0.14      0.06     0.03     0.26 1.00    11628
## FixCount                 0.01      0.04    -0.07     0.08 1.00    18308
## zi_AlgorithmBugspots     1.83      1.09    -0.10     4.23 1.00    19995
##                      Tail_ESS
## Intercept                7691
## zi_Intercept            10260
## AlgorithmBugspots       11044
## LOC                     11920
## FixCount                12786
## zi_AlgorithmBugspots     9393
## 
## Family Specific Parameters: 
##     Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
## phi     8.93      0.45     8.06     9.83 1.00    19545    11846
## 
## Samples were drawn using sampling(NUTS). For each parameter, Bulk_ESS
## and Tail_ESS are effective sample size measures, and Rhat is the potential
## scale reduction factor on split chains (at convergence, Rhat = 1).

Comparison

loo_compare(loo(maucec51, reloo=TRUE), loo(maucec52), loo(maucec53), loo(maucec54))

##          elpd_diff se_diff
## maucec54   0.0       0.0  
## maucec53  -0.8       0.8  
## maucec52  -1.0       0.8  
## maucec51 -53.3       5.4

Bürkner, Paul-Christian. 2020. Brms: Bayesian Regression Models Using ’Stan’. https://CRAN.R-project.org/package=brms.

Gabry, Jonah, and Tristan Mahr. 2020. Bayesplot: Plotting for Bayesian Models. https://CRAN.R-project.org/package=bayesplot.