Notes

Some Project Notes

Goals

Load the data

library(R2jags)
library(knitr)
library(MASS)

# Load bu.RData
load("../data/bu.RData")

Test functions

beta.fit()

The beta.fit() function akes a vector of pmf data (either estimates of the pmf of specified quantiles) or raw samples and calculates posterior estimates of the alpha and beta parameters of a beta distribution.

For example, consider the following 10 samples from a beta distribution with alpha (shape1) = 2 and beta (shape2) = 5:

samp <- rbeta(100, shape1 = 2, shape2 = 5)

We can fit the data to a beta distribution with beta.fit():

samp.fit <- beta.fit(data = samp, show.plot = F, format = "raw")

Let’s see how well the function recovered the true parameters. We’ll plot the posterior distribution of alpha and beta values. The true values are indicated with a T, and the mean of the posteriors is shown in green.

plot(1, xlim = c(0, 8), ylim = c(0, 8), type = "n", xlab = "Alpha", ylab = "Beta", main = "Posterior distribution of Alpha and Beta Parameters")
text(2, 5, "T")
points(samp.fit$alpha, samp.fit$beta, col = gray(.5, alpha = .1), pch = 1, cex = 1.5)
points(median(samp.fit$alpha), median(samp.fit$beta), pch = 16, col = "green", cex = 2)

p.fit()

The p.fit() function takes a participant number as an argument, and runs beta.fit() on each trial in each phase. This gives us the alpha and beta values that best represent a participant’s response on each trial. Because we know the Bayesian alpha and beta values on each trial, we can compare the participant’s fitted responses to the Bayesian criterion.

For example, let’s analyze participant 1’s data.

Fitting participant 1’s data

Now, I’ll use the p.fit() function to estimate the participants’ alpha and beta values for each trial in each phase:

p1.fit <- p.fit(1, show.plot = T)

Each plot in the upper two rows shows three types of data: circles show the participant’s raw responses. Black lines show the fitted posterior beta distribution for the raw data. Finally, the red lines show the optimal Bayesian posterior. If the black lines match the circles, then the participant’s response is well-fitted by the beta distribution. If the red lines match the circles (and black lines), then the participants’ responses are well-fitted by an optimal Bayesian beta distribution.

Looking at the first row of plots (phase 1), we can see that the participant’s data are very well fit by beta distributions. For trials 1 and 2, the participant’s data also closely align with the optimal Bayesian model. However, as trials increase, the optimal Bayesian distribution becomes much narrower relative to the participant’s distribution. In other words, this participant appears to be a ‘conservative Bayesian’.

In phase 2 (the second row) the participants’ data are again well described by a beta distribution. However, the participants’ estimates are far away from the optimal Basyesian response. In fact, the participant’s estimates do not seem to change at all across trials.

Next, we focus on the means and standard deviations of the participant’s fitted data, and the optimal Bayesian response.

Participant vs. Bayesian Expectations

The bottom left plot summarizes each of the upper plots into a posterior mean for both the fitted participant response, and the optimal bayesian response. In other words, it compares the Bayesian optimal expectation of the true success rate, with the participants’ fitted expectation of the true success rate. A Bayesian optimal participant should always produce values on the main diagonal of the curve. Phase 1 data are presented in Black, while Phase 2 data are presented in Red. Looking at this plot, we see that, in Phase 1, thep participants’ expectations fall relatively close to the main diagonal – suggesting that her expectations are consistent with Bayes. However, her phase 2 responses are far off the diagonal, suggesting that her expectations in Phase 2 are much lower than that of Bayes.

Participant vs. Bayesian Uncertainty

Next we move to how much (un)certainty both participants and the optimal Bayesian reflect in their responses. To measure this, we calculate the standard deviation of the fitted and optimal Beta distributions. This statistic reflects how much certainty (or uncertainty) both the optimal Bayesian and participant have in their estimates. If the points fall on the upper-left hand of the graph, participants are conservative (underconfident) relative to Bayes, while if they fall on the bottom-right hand of the graph, they are too liberal (overconfident).

The data clearly show that this participant is highly conservative (underconfident) relative to Bayes in both Phases 1 and 2.

All participants

Means

Now I’ll plot the distribution of bayesian means and participant fitted means across participants for trials 0, 5 and 10. In Round 1, The results again show strong ordinal consistency between Bayes and participants. However, participant’s responses tend to be less extreme than those of Bayes – suggesting conservatism. In Round 2, there appears to be no relationship between participants and Bayes! I wonder if there is a problem in the data…

Standard Deviations

Now I’ll plot the distribution of bayesian standard deviations and participant fitted standard deviations. The results again show very poor consistency between Bayes and participants. Again, because participant’s standard deviations tend to be much higher than those of Bayes, people are much too conservative.

Participant level biases and conservatism

Next, I calculated each participant’s average bias relative to Bayes (that is, the average of the deviation between their expectations and Bayes across all trials), and average conservatism (average difference between the standard deviation of their posterior relative to optimal Bayes) across all trials within each round.

The expectation bias plots show that people are, on average, relatively unbiased on Round 1. In Round 1, 0.56 percent of people have an absolute bias less than 0.1. However, this still leaves several participants with substantial biases.

In Round 2, people appear to become much more biased. Here, only 0.19 percent of people have an absolute bias less than 0.1.

The conservatism plots show that virtually all participants are, on average, too conservative in both rounds 1 and 2.

Does expectation bias correlate with theta?

Next, I plot the relationship between theta and participants’ bias. The data clearly show that, in both rounds 1 and 2, people’s bias is clearly a function of the true theta value. When theta is low, people are positively biased, when there is high, people are negatively biased. Again, this points to strong conservatism.

However, conservatism does not appear to be a function of theta.

Next steps

  • How and how well can we model a participant’s fitted alpha and beta values in each trial as a function of the stimuli? In the analysis above, I just descriptively fitted their responses separately on each trial. This means there are 2 * 10 (20) parameters being fitted. A better model would have a few psychologically relevant parameters (i.e.; an updating rate) which, when combined with the actual stimuli, would produce the black curves.