Markov environments code

Generating stochastic outcomes for the game

Here’s R code for generating outcomes in a Markov environment from a transition matrix. The transition matrix tells you the probability of any outcome occuring given the outcome that occured on the previous trial.

Let’s say the transition matrix is the following

1 -> 1 = 0.10, 1 -> 2 = 0.30, 1 -> 3 = 0.60

2 -> 1 = 0.00, 2 -> 2 = 0.20, 2 -> 3 = 0.80

3 -> 1 = 0.90, 3 -> 2 = 0.10, 3 -> 3 = 0.00

Here’s how the matrix looks in R.

t.matrix <- matrix(c(.1, .3, .6, 0, .2, .8, .9, .1, 0), byrow = T, nrow = 3, ncol =3)

t.matrix

##      [,1] [,2] [,3]
## [1,]  0.1  0.3  0.6
## [2,]  0.0  0.2  0.8
## [3,]  0.9  0.1  0.0

These probabilities indicate the probability of each outcome occuring given the outcome that occured on the previous trial. For example, if a 1 occurs on trial n, then on trial n + 1, the probability that a 1 occurs is 0.10, the probability that a 2 occurs is 0.30, and the probability that a 3 occurs is 0.60. In contrast, if a 2 occurs on trial n, then on trial n + 1, the probability that a 1 occurs is 0.00, the probability that a 2 occurs is 0.20, and the probability that a 3 occurs is 0.80.

It is essential that the sum of probabilities in each row add up to 1!

Here’s how to use this matrix to generate outcomes for a game. Let’s say the game has 50 trials, we’ll start by creating an empty vector called locations with a length of 50:

locations <- rep(NA, 50)

First, we’ll determine a random outcome for the first trial

locations[1] <- sample(3, 1)

Next, we’ll loop through the rest of the locations vector and create new outcomes based on previous outcomes and the transition matrix:

for(i in 2:length(locations)) {
                               
  # Get previous outcome
    previous.outcome <- locations[i - 1]
    
  # Get outcome probabilites from t.matrix based on
  # previous outcome
  
    next.outcome.probabilities <- t.matrix[previous.outcome,]
  
  # Select an outcome based on next.outcome.probabilities
    
    next.outcome <- sample(3, size = 1, prob = next.outcome.probabilities)
    
  # Add next.outcome to locations

    locations[i] <- next.outcome
    
}

Here is the final result

locations

##  [1] 2 3 1 2 2 3 1 2 3 1 3 1 3 1 2 3 1 3 1 2 2 3 1 3 1 2 3 1 3 1 3 1 3 1 2
## [36] 3 1 3 1 1 1 2 2 3 1 2 2 3 1 2

Looks good, 3 usually is followed by 1, and 2 is never followed by 1.

Database coding scheme

The database showing the results of the game should have the following columns:

• workerid, the participant’s workerid
• gameid, the id of the game
• trial, the current trial number
• prediction.num, the participant’s prediction for that trial (using the number coding)
• outcome.num, the outcome for that trial (using number coding)
• prediction.display, the participant’s prediction using the display coding
• outcome.display, the outcome for that trial using the display coding
• points.earned, the number of points earned by the participant on that trial (1 if prediction === outcome, 0 otherwise)
• total.points, the cumulative number of points earned by the participant in the game.

workerid	gameid	trial	prediction.num	outcome.num	prediction.display	outcome.display	points.earned	total.points
aaaaa	321	1	1	1	A	A	1	1
aaaaa	321	2	2	3	B	C	0	1
aaaaa	321	3	3	3	C	C	1	2
aaaaa	321	4	3	1	C	A	0	2
aaaaa	321	5	1	1	A	A	1	3

Note, I distinguish numeric and display coding because, in some games, the numbers will be randomly assigned to different options. I’d like to keep both codings in this database so we can always connect the original outcome sequence (always defined using 1s, 2s, and 3s) to the display each participant sees (always in As, Bs, and Cs).